There aren't enough details here for me to understand what sort of behavior you're talking about. I have discovered that a group of kids will nearly always run wild if allowed to do so. As Demetra C. said, the adult has to be in charge, tell them the rules, and have immediate consequences when they aren't obeyed.
I've also found that I cannot teach a group of kids much of anything in the way of good behavior. When I'm watching my grandchildren they can have one friend or possibly 2, if they've proven themselves, inside the house at any one time. Much easier to manage them that way. And.....I am able to teach them appropriate behavior by saying, "in my house, the person who breaks a toy has to pay for the toy.?
I also make a point of getting to know the parents. You stated you were afraid to do this but it's really an important step if you plan to live in the neighborhood for a long time. Your child will be wanting to play with them. You can't keep the neighbor children completely out of your lives. Yes, you can keep them out of your house. However, I prefer having children play with my daughter and now my grandchildren that I know and can see what they are doing. As your child gets older they won't be willing to stay in your yard with you there watching them.
You might be surprised at how many of the other parents feel the same way you do but handle it differently. And that they would be willing to replace a broken toy. That they want their children to behave and are unaware that they don't at your house.
Groups of children tend to get wild. They need supervision and to be taught the rules for good behavior in your house. And if you know their parents they are more likely to behave knowing that you will tell on them.