I think the whole birthday/venue/favors/entire class deal is overdone, so I commend you for doing a small get-together with an activity, and trying to scale back.
I don't think you have to provide a favor at all, but if the 3 kids know that it's your son's birthday and will be bringing a gift, and if everyone knows there's a reason why you are paying for the gathering, I think there might be an expectation that favors are common. However, I think this is a good age to start teaching kids that they don't get "paid" to come to a party, that they should be gracious guests and gracious hosts, which means being content with and grateful for what is offered.
If no outside food is allowed, what will you do? Skip it, or provide food from the venue? Are you sticking to a budget (no problem if you are) or trying to send a message that we don't need favors (no problem if you are). If you would skip the food entirely, and let the whole party be the play activities, then I agree with the suggestion below that you could consider sending them home with something of your own hands (cupcake, brownie, etc.) with a simple "thanks for coming" tag. But I reiterate that it's not necessary. When teens and adults go to parties of any kind, they don't get favors. It's okay to make the party into the sole event, and perhaps even set a precedent for other grateful parents!
We always did small, at-home parties with games (not hired entertainment or bounce houses) and usually the activity was something like a scavenger hunt for stickers, treats and candies, which became the "favors" or "goodie bags." By 8 or 9 when my son wanted to take a few kids bowling or to the movie, we didn't do take-home favors. To my knowledge, no one ever said they wouldn't ever come back because we were too cheap! And if they had, we wouldn't have considered it a great loss.