Yes, it's gonna be an issue. But it's also what happens to most families when they start having kids. So I would suggest you find some compromises. First, don't get so crazed with all the gifts that you are exhausted. Find some time to spend with your parents and his parents at other times. Maybe Christmas Eve at one house, and Christmas Day at your own house. Next year, Christmas Eve at the other parents' house, and Day at your house. Maybe someone can come for dessert in the afternoon, or the following day. Not sure how far apart everyone lives, but if you can find a way to include them without cutting everyone out cold turkey, it will make your lives easier. On some level, it will help to create memories for your kids too. If Christmas becomes a just-your-family event with no extended family because you are too overworked, then it's not going to make you happy in the long run either. See if you can save a few days off for just beforehand so you have some down time, and try to get your shopping and wrapping done way in advance so December isn't a track meet.
You deal with your family, and your husband deals with his family. If you tell your in-laws that you aren't going there, YOU will be the bad guy. Make it about the kids getting overstimulated in big crowds and not ever having their own traditions in their own home. Hauling all that baby gear, trying to put kids down for a nap in a strange bed, etc. etc. Make it clear that you love your parents/in-laws but need to start your own traditions just as they have developed theirs.
Try to figure out ways to give the grandparents the experience of seeing their grandchildren open gifts. Perhaps do one gift (or 2) from the grandparents on Christmas Eve, then Christmas morning becomes about stuff from Santa and stuff from the parents. Take pictures and email them, or do a short video and send it to them. At an early age, teach your children to write thank you notes (okay, not for the 5 month old, but the 2 year old can be asked for a comment or two, and then you write it verbatim to the grandparents with a little bit of crayon marks from the child!).
Find something fun to do, a tradition to start, so that the holiday isn't just, for you, about avoiding everything! Make sure you have fun things to look forward to. Some families give the kids their own ornament each year, thereby expanding their collection of memories. As the kids get older (and the 2 year old can do something small now), have them make decorations (either for the tree or the house in general). Date them with a sharpie marker in a hidden place, or in the case of a hand-drawn picture, laminate it or frame it. Bringing those things out every year helps to create warm thoughts and memories.
Good luck with this! It's a tough transition but hopefully the grandparents will remember what it was like for them when their kids got to a certain age.