This is what I did for my son...I invited all family, close friends and neighbors we socialize with. Not everyone will be able to attend.
And basically with neighbors, if you invite one, you should invite all that you socialize with because they will talk to each other.
The first birthday is really for the adults to get together, but if you have adults with younger children attending, you should have some games and treat bags for them. Otherwise it's not fun for them...and it is a "kids" party.
But if it's just adults attending, then don't worry...gear the food towards adults, no kids activities needed...and you'll get great gifts for your kids...especially if they need clothes.