First of all, "bullying" is when someone is picking on someone else who isn't fighting back - sounds like your son might have been involved in a "conflict" - whether he was at fault or not, he wasn't being a "bully". That's a completely overused term - bullying is very serious, being a kid and getting into conflicts is normal.
Anyway, that being said - speaking as a classroom teacher, I think you should go to the referring adult first if you choose to take this any further. You said yourself that there have never been any major incidents, so why jump straight to the principal? The classroom teacher wasn't even there, she/he probably has no idea it even happened. You might even want to take this opportunity to teach your son about letting things go and not making it too serious - I'm not sure how devastating this is to him; some kids don't worry too much about it. If you do talk to the recess teacher and it continues, I would ABSOLUTELY get the classroom teacher involved and then move to the principal together, with your son's teacher, if things don't change.
This is really unfortunate and I'm so sorry it happened. It's not fair when nice kids get blamed for things that weren't their fault. It's my guess that the referring adult just misinterpreted the circumstances - he/she had a lot of kids and was trying to do the best he/she could considering the chaos of recess! So, if you bring it up to this person I'm sure he/she would be happy to talk with you about this.
Good luck - I dread the day my baby gets "hurt" at school!!! I thought I was pretty empathetic to my students' parents before I had my daughter, but now I REALLY get it!