WOW-wee! No, you are not being a "B" (as you put it), you just do not want the added stress of someone potentially dumping on you, by backing you into a corner and expecting you to take on more than is already on your plate. If it hasn't been decided where they will be staying, let another relative take on that fun and offer their place. Have your husband spread the word!
You have put up a lot of obstacles as to WHY you do not want to...: have people stay overnight, don't want to watch other people's children, etc... The fact is - You should be able to say "NO" without feeling so guilty. If you don't want to have people overnight, just say "NO" . You shouldn't have to justify it so much and your husband may be disappointed but... you both have a lot on your plate, more than you had 2 years ago. It's funny because you sound soooo much more calm about having 100 people at your home but the 5 others are pushing you over the edge - so I'm guessing they may just be those relatives (and we ALL have them!) that "push your buttons"!
I hope you are having help preparing your home. It's a huge undertaking to have so many people over. I know when we had a reunion at my home, we actually recruited two friends to take over the kitchen, serving, cleaning, etc... Once the party started, my friends kicked me out of my own kitchen, told me to "let go" of the responsibility of serving food/clean-up and it was THE most amazing, fun time I ever had at my own party.
It's very generous of you to host the party. I hope you and your husband make up before the big day - you'll need each other!