This has been an issue for my daughter's parties as well. If you have parties during summer vacation, especially around the 4th of July, families want to stay together and often have house guests. I would just tell them the truth. You promised your friend that you would keep the party small, so it is for invited guests only. Keep in mind that you may have some send their regrets rather than not have the entire family come, but that is their choice. The only pause I would have is the single mom. If you expect parents to stay and help watch over the kids, you have to allow her to bring all of her kids, since there is no other parent to take them during the party. However, if you can have her drop off the one invited child and return later for pick-up, the other kids can be left off the invite list.
In the future I would move his party off of the holiday weekend. It will really help with this sort of thing, even if it means having the party a week early or late.
As for people being offended, that is their choice. Unless you are purposely snubbing them, they need to understand that not everyone gets to come. Adults should have learned this lesson years ago. Different parties have different guest lists and it is up to the hosts to determine who it is, and is not, appropriate to include in any given event.
My two cents!