My husband and I struggled with 3 families at the holidays because his parents are divorced, but we've worked out a good system that we explained to everyone. So, here's what we do - we spend Thanksgiving with one family, then Christmas day with the other family. Then, the next year, we switch. So, every other year, we spend Christmas with my family. It was hard the first year because that was my first Christmas away from my family, but we usually have a "mini" Christmas before or after the holidays with my mom, dad, and brother to have dinner and exchange gifts. My father-in-law's family prefers a quiet Christmas day, so we spend Christmas eve with them. Thankfully, my MIL and my parents get along great, so she often joins us for my family's Thanksgiving/Christmas, which helps accomplish two things at once.
My cousin's family likes a quiet Christmas at home, so they stay home to open presents and have brunch in the morning, then they join the larger family Christmas in the afternoon after lunch, which is another option you could consider.
Since it's your husband's parents who are the trouble-area here, I think you have no choice. A marriage is about compromise, so if he wants to see them for a certain amount of time, you should go with him, regardless of how uncomfortable it is for you. You should never be the wedge that pushes your husband away from his family. Imagine if the situation were reverse and he didn't like visiting your family. Would you let him keep you visiting them because he doesn't like it? Yes, Christmas is supposed to fun, but it's supposed to be fun for EVERYONE, not just for you. Sometimes we have to make sacrifices so that others can be happy. Trust me... I make a sacrifice when I go to my FIL's and I know my husband sacrifices when he goes with me to my family's house.