First -- a birthday party is not the time to make friends -- so I wouldn't invite people with that intention. I'd stick with the friends he feels close to
I agree, people will feel strange that they only were invited to one part of the festivities. And to come for just a one hour party, may seem like a lot of work for people.
It's challenging == but either choose another venue or just choose kids he's close with . It's always a problem when some kids are invited and some aren't -- but I don't think it makes it better by inviting some kids to part of the event and some kids to the other part of the event.
There will always be less kids than you think show up, but if the restriction is 25, and you have a class of 45 -- I wouldn't take the chance (as some have suggested), of just inviting everyone.
Let him build more friendships next year when he's in the mainstreamed class 3rd grade class. But I wouldn't make this event about that goal