It sounds like babies are a bit new for him? Maybe? My husband had a learning curve when it came to kids. Now he's pretty good about it, but at first, I felt like pulling out my hair. It didnt' seem to matter how many nights I got no sleep, even on nights when he didn't have to work the next day (and he knew I was beyond exhausted), he'd just snooze right through it all. He wouldn't offer to help. Wouldn't think beyond what he was used to. I was the mom, so I got the baby...in his opinion.
Anyway, he's learned over time. It took until baby three was here before he seemed to really get it. With baby #4, he does awesome with the kids and giving me breaks when I need it (and I"m not needy or anything, just need it sometimes!).
I've had lots and lots of talks with him about it in the past. He tells me that he's more than happy to help, I just need to ask. Sometimes I expect him to offer to help (since it's SO FREAKIN' OBVIOUS I NEED HELP!) and I feel like he's purposefully not offering, which makes me feel insulted. But I've taken him up on it, and if I need help, no matter how obvious it is, I will ask him now. Most of the time he'll do it with no hesitation. And, as I mentioned, with more and more kids, he's learned A TON about stepping up and helping. He does awesome now.
It works for me to flip it around on my hubby. I'll ask him, "How would you like to always have a baby to take care of with no help from me? Instead of me trading off with you, how would you feel if I spent the weekend shopping and with friends, leaving you with the kids?" I know some guys still don't care, but it amazes me how sometimes that's all I need to do and my hubby will finally get it. He just needs it flipped around so that he can put himself in my shoes more. He's become a lot more compassionate about it.
Truth is, with your hubby, I would start saying no. I would let him know ahead of time that you are planning whatever you're planning. Make sure he gets his free time too, but plan that he helps take care of the baby too so you can get some time to yourself OR for you guys to spend time together as a family instead of him off doing his own thing.
I'm not good at keeping my thoughts to myself, so if I were irked, I'd let him know and tell him exactly why. Doesn't mean I'd be rude - just clear about it. He's thinking only of himself and not of you - NOT OKAY.