Does he not gravitate towards any particular kids because he doesn't like ANY of them, or because he likes ALL of them?
If it's that he doesn't care for or about any of the kids then I think a family-only birthday party is just fine- make it special with all the regular party stuff, same as you would for a bunch of kids: cake, balloons, party favors, fancy paper plates with his favorite character etc on them, games.
If it's that he doesn't single out any specific kids because he is comfortable with everyone, then a kid party would probably be just as fun for him. In that case, ask a few kids from his class. You could tell the teacher you are planning to invite a few kids from the class, but don't know which ones to invite, and ask for suggestions. Or you could ask your son which kids he would most like to invite. (Just remember to send the invites through the mail or hand deliver- not have him hand them out in class if everyone is not invited...)