No, just, no. This is an accident waiting to happen. What happens if the girls are hurt on their way back home alone? Will the parents try to pin this on you because YOU didn't walk them back home (as if that's your duty!)? Not to mention, the guilt you would feel if something happened. You're not their free babysitting service, and the nerve that the father had to take one child and leave the other behind is just shocking...perhaps if he had asked you to please watch the baby, then MAYBE, and only MAYBE would I be able to excuse him, but still, he is the parent, it is their responsibility to find the time to be home with their kids, or find a babysitter or relative to step up and do that.
The rest of us pay an aftercare or a sitter, why can't they? Are they even compensating you for the time and effort spent in watching their two kids daily? I assume they don't even bother bringing you some dinner, some 'thank you' cookies, and heck, they probably don't even drop off the toddler with clean diapers, so I guess you have to also buy diapers for a child that isn't yours. Not to mention, I find it extremely dangerous to rely on a 6-year-old to babysit and gallivant alone around town with a baby. I would not allow a 6-year-old to wander around unsupervised in the first place, much less with a baby. They could get run over, at the very least. What if it is dark and they get scared? They could fall in a lake, become disoriented, or who knows what.
I would go over to them and introduce myself, and explain that your hands are full in caring for your own two kids. While there may be times that you have some free time for their kids to come over, this is not the case every single day. They are free to call you (or better yet, tell them you'll call) whenever you have time for the kids to come over, BUT they must either be dropped off or an adult must accompany them to your home. Same rules apply for pickup. You do not want the responsibility of them wandering off, getting lost, hurt, or whatever else can happen if the two of them are walking alone. I assume they are not next door, but a few doors, or even a block away. Based on the fence comment, perhaps they live directly behind your house, which would be about half a block.
Honestly, I think by slowly cutting off contact, you'll be better off, considering the children are manipulative (telling your son he is mean and won't be invited to their party unless they're allowed to come to your home all the time) and the parents seem to be freeloading and taking advantage of you. Don't allow it to continue, or else, it will get worse as Diane D. mentioned. If they still let the small children wander around town alone, I would call the police or CPS anonymously and report that there are two young children unattended and walking the streets, and you think the parents need to be contacted. Once the authorities light a fire under their butt, I bet they will be less inclined to be so irresponsible.