First of all, don't panic. It will all work out fine. For the 2-4 year-olds, treat it just like a playdate. In the kids' bedrooms, set up toys they can all play with. Dress-up clothes, cars/trucks/Thomas train track set up if you have them, dollhouse with people, a book corner.
In my neighborhood, we have 6 families that hang out - all of us have 2-3 kids and my 5 year-old is the 2nd oldest. We never have anything special planned when they all come over...the kids just want to play. So we just make sure the toys are set up throughout the house, leaving enough space in the main areas for the adults to hang out, and giving the kids enough space to play. The kids totally entertain themselves. Honestly I think games/activities at 2-4 is asking for more chaos than organization.
In my house: in the kitchen I have a Little People farm house set up with lots of animals, and a couple of toy trucks. I have that set up right next to the fridge where I have the alphabet letters set up to play with. It amazes me how much action that little corner gets! I have a kid table I'll set up with lots of paper and crayons. That table doubles as where the kids can eat. In my son's room we set up the Thomas track and trains and the book corner. In my daughter's room there are toy bins, her dollhouse + people and little Toy Story characters, etc. that all like to be in the dollhouse, and dress-up. Downstairs I have a casio piano set up the kids can totally bang on and play music on. Also that is a bigger open space they can run around in and throw balls, etc. I have a big open futon down there too and more books.
I would put together some fun goody bags though. In the event of a child's meltdown you can always break that out. :) Also for fun snacks - freeze some of those Go-Gurt yogurts and cut them in half for each kid for a snack (assuming no dairy allergies). Also have some juice boxes handy.
If your husband or a guest suggests going outside during the party, don't panic. The parents will prevent their kids from eating the rocks. Also you might want to have a box fan and some bubbles with a huge bubble wand handy. You can dip the huge bubble wand in the bubble liquid and hold it ini front of the box fan (outside of course) - it will make tons of bubbles and I swear you can keep those kids entertained forever with that!
Good luck and most importantly, don't miss any precious moment of this big day being worried about the party. Focus on your son's birthday and the rest will be fine. :)