When the boys start to ignore your son, then it is time for them to leave. They are using him and he's not old enough to realize it. You just need to say something like "okay boys, I have a whole bunch to do and I need to start cleaning up. I'm going to have to ask you to go home now". When you little one complains, just send him out of the room and then explain later that you needed to get some things done. Period.
And yes, you need to send them to get permission for everything...first and foremost a phone number for each of these children that are frequenting your house. Every time they want to eat something, do something, or come in...you need to call their parent and in a chipper voice say:
Just wanted to let you know that so-and-so is at my house in case you need to get him for whatever reason.
Then if they want to eat something...call and ask if they can or send them if that's better with a note...you don't want to feed something to someone and have an allergic reaction at your house.
And yes, if they want to swim...then send them with a note to ask permission and to get a suit on.
After awhile they might start to call you first...even offer your number in case they need to call. I think it is rather irresponsible of a parent to let their children wonder aimlessly these days and go into someone's house that they don't even really know.
After a few calls and notes sent home maybe they will get the hint that you are a responsible mother and will start being a little more considerate of you and your child.