D.B.
Classic. Don't you love it when others get involved? My husband's ex used to put her brother, her 2nd husband, anyone else on the phone or in the room.
Your EH was childish - he needs to be a father first. If he really couldn't face you, he could have emailed you that he'd pull up out front. I think 2.4 is a little young to have to walk from the mom's front door to the dad's car by herself, but it would have been preferable to a surprise visit from Grandma. Still, advance notice was too much to ask? And how chicken is he that he's annoyed so he calls in his mother?
Kids often have trouble with transition. Secondly, your daughter was expecting Dad, not GM. Third, the woman went on a tear about you in front of her granddaughter. Obviously, she's less concerned with the child than with anything else. So of course your daughter didn't want any part of it.
I'd suggest that you contact your EH if that's permitted (phone, email, letter) and say that this was really negative for your daughter. She needs stability, as others have said. Reiterate the agreed-upon "exchange" - at your front door or whatever it is. No substitutions unless discussed in advance AND AGREED TO. If that isn't possible or if it doesn't work, then have your lawyer call his lawyer.