I'm totally in agreement with Lets C. As the "adult in charge" by default, you do want to set the tone and the expectations when your son is playing with other kids.
Thank him when he shares that the other kids are teasing or tempting him to break your rules. Assure him that your rules are reasonable, and you feel sad for kids whose parents don't provide that kind of loving oversight. He may appreciate that a great deal more than he lets on. Kids crave knowing what their limits are, and sometimes appreciate having a "tough" parent on which they can "blame" the rules they secretly appreciate.
It might be helpful to let your son help set some of those limits. When he's out of your sight, he will still be more likely to keep to limits he's helped establish. And when the kids are in your area, be clear that you'll end the games if you have to ride herd on them because they are constantly pushing safety rules. "Sorry, but that's life, kids. Think it over, and try again tomorrow."
Good luck, mom. It sounds to me like you're doing a good job.