Yeah - we have 2 kids now, ages 4 and 2, and we also have 4 families, maybe more like 5 to try to go to for holidays. It is hard and tiring, and it makes the holidays hectic and rushed for us.
Count yourself lucky, thye all live within 5 miles. We have my grandparents still living, and my mom and then my dad ( divorced) all in once town, 60 miles away from our home. Then Dh's mom, 50 miles away, her husband just passed away last weekend, after a long bout with Cancer. She has 2 adopted children which are her biological grandchildren, who are 15 and 18 and live at home with her, and probably always will due to developmental issues. Then Dh's Dad and step mom are all the way in Nashville TN, a 10 hour drive from us. Then every Thanksgiving we go to Dh's Dad's side ( all his aunts uncles and cousins - and his grandparents, before thye passed away) in West Virginia, 12-14 hour drive from home.
Only thing I can suggest is to host them all at your house.( and tell them they have to get along amicably for the Holidays for the sake of your baby, their grandchild).....or designate different weekends for different sides of the family.
When we got ot Mendota, where my Dad and Mom are, we do morning thru Lunch at one house, then nap through Dinner at another, since they are all close together it is do-able.
We do Thanksgiving every year with Dh's Dad's side in WV, so very rarely do we see them at Christmas. Easter is usually with my Mom's side....the rest of them don't do a big thing for Easter. We try to prioritize my Grandparents at Christmas, since they are in their upper 80s and will not be around forever. DH's mom and sister who live closest at 45-50 miles away, get to see us much more often, and for the last 5 years, we have had them here for a holiday meal and celebration on a different weekend than the rest.
Good luck! You have ot do what works for you.
We do try tobe home on Christmas eve and morning, so the kids get the Santa experience of finding their gifts under the tree and in the stockings, etc. They are just now getting big enough to understand/remember year to year.
You have a few years before your baby will be noticing/remembering any traditions year to year, take this time to figure out a system that works for you! Be happy they are all so close, it makes going everywhere much easier!