K.F.
We enjoy a house or yard full of kids MOST of the time. If you have ever really been "that house" on the block then it goes without saying there are times when you just have to say no.
I promise children do not take offense, they are not like adults. Just say it's family time right now but you'd love to see their sweet little faces later or tomorrow or whenever.
The only way being the "block parent" will work is if you're honest and direct. If you don't speak up and let them know when it's time to go home you'll end up resenting the fact that you have to be responsible for other people's kids when you really don't feel like it.
Enjoy being the house of choice, it is such a great thing in so many ways!