No.
Many of my daughter's classmates/peers are small. There is never any problem. I see them and know these kids, and they are very well adjusted and happy and socialize just fine and integrate just fine. Their size, is NEVER an issue. These are boys and girls that I know... of all ethnicities and cultures.
My daughter on the other hand is tall for her age... as well as my son. My son is only 3.5, but EVERYONE thinks he is in Kindergarten and he is bigger than most Kinder kids. So then they judge him based on "size" thinking he "should" act older than he is. Misplaced & mistaken expectations. So there is that perception too.
And, as for my daughter... it is the girls in her grade level, that are bigger/taller than many of the boys.
Do not focus on your child's size. If you do, he will then get hung up on it too. Guaranteed. And then he will have issues because of it. So do NOT displace your apprehensions about his size, onto him. Just always focus on what he can do, what he is good at, how proud you are of him.. and foster his INNER sense of self.... NOT if being contingent on his "appearance." NOT EVER. A child, if raised without focusing on superficial appearances or compared to others... he/she will be "blind" to those superficial affectations as well. And then, they will have an inner sense of self... that is FAR more solid in stature.
All the best,
Susan