Absolutely talk to the parents. From what you say, it sounds as if there are good relationships in place between parents and kids, and I'm sure the other parents would be just as upset if this were happening with one of their kids--and you would want to do your part to help.
Let the parents know what's been happening, what you've been doing about it, how upsetting it is to you and your family, and tell them that you need their help. Emphasize how you love your group and all the kids, and you'd like to work together to fix this problem. It's a teachable moment for all the kids in the group. Even if it's a couple who are causing the problem, the other kids need to understand the impact of their just standing by and watching it happen. Their inaction is a choice to allow bullying and they're just as guilty.
Bullying comes from insecurity, so there's probably something else going on with the kids starting all of this--they need to be called out on it and asked point blank why they're feeling so insecure that they have to pick on a four year old. The other kids need to be taught how to handle these situations too, and you've actually got a bonus that this is happening with people you know and like instead of at school or somewhere where you're less familiar with all the people involved.
When addressing the parents, just remember how you might feel if the tables were turned and another parent told you how terribly your child was behaving. You might get some knee-jerk reactions that you're not prepared for, so again emphasize your disappointment in the behavior (not the kid), your love for the child, and your commitment to helping all of your kids be the best people they can be.
At home you can try role playing with your kids. Teach your son how to respond or walk away. Teach your girls how to stand up for themselves and their brother. And teach them all when they need to get an adult to help out.
Good luck!