I'm sorry he feels that way about it.
In my marriage, we discuss these kinds of things. And if either of us wants to skip something, we discuss it, and why. Pros and cons, etc. But if I really didn't want to do something, we wouldn't do it. Not even just ME not do it, but WE wouldn't do it.
For Christmas, my SIL invited us to dinner with them 75 minutes away. Not a big deal, but we get up, open gifts, eat breakfast/have coffee, showers, then go to church. No time to play with anything or do anything with any of the gifts, an really not even time to clean up the mess from opening things. Our daughter played the organ to provide the music for the service. It was at 10:30 and it was pushing noon when we got back home afterwards, as I knew it would be. I did NOT want to jump right back into the car to drive to SIL's for a 1:00 or 2:00 meal. And I said as much.
Husband was okay with us just not going at all, if that is what I wanted. But I like my SIL and didn't mind going, but I wanted to breathe beforehand. She offered to move dinner back to 5:00, and we then agreed that 4:00 would be fine too. That's what we did. But if I had not wanted to go, we all would have just stayed home. Not because I am bossy or demanding about things like that, but because my husband and I are a team. We do what is good for our family. None of us wanted to be in a rush after leaving the joyful Christmas morning service.
So, I came home, and baked a pie and put together a side dish to take with us. And we loaded up and drove down. Had a nice time/visit. Glad we went.
We completely skipped out of town (overnight) travel to visit other family this year. We talked about it. But just didn't want to do it this year. So we didn't. Nobody is mad at us about it, either.
Maybe instead of you getting your husband's opinion of YOU sitting out one event, you pose the idea of ALL of you sitting out an event or two. It sounds like it is just too much. Some people thrive in the hustle/bustle craziness of running everywhere for everything. Some people don't. Maybe next year, discuss in advance how many activities you think you are "game" for, and then as the invitations roll in, you know what your limit is (and so does your husband).
I wouldn't make a big deal about it now. It's water under the bridge. But NEXT YEAR, I would get out in front of this, and discuss in advance that you don't want to be so busy, so you can share that knowledge with the rest of the family and they won't be expecting you at EVERY.THING. It sounds like some of your husband's siblings have already made that decision.