I would have done the exact same thing.
And quite frankly, if you HAD to be gone for the weekend/etc. (say someone died, or there was a business trip, or you were catching a flight), there not only wouldn't have been a double birthday party beforehand, but you'd have set everything up to ease into that.
My son (now 6) spends spends one weekend a month at Nana & Papa's house. (Hurray!!!) They watched him for me while I was in school, while my husband & I went on our belated honeymoon -10 days, and during one of the trips to Italy -14 days. Super super close, my son and his grandparents. And they sometimes have a bad day. It happens. As mums WE sometimes have bad days with our kids. Heck. I've had a bad week!
I wouldn't necessarily classify being DONE and wanting to go HOME as a bad day. Merely knowing EXACTLY what he wanted, coupled with being exhausted. Sigh. Never a fun combo with toddlers.
For what it's worth, I think you and your parents handled it well.
:)
R