Does the NICU at the hospital where your son was born have a follow-up program? My oldest was a 32 weeker too and Bryn Mawr does follow up with the babies. Our ped at the time blew up off when I brought up concerns about delays in gross motor development. When I took her back to the NICU they evaluated her and suggested early intervention.
My daughter received occupational therapy and made great progress. We started her at 7 months because it took about a month to get everything in place. At eight months, she was not sitting, rolling or crawling but by 13 months she was walking. She is now 8 years old and still struggles with her fine and gross motor development but is doing great in other areas.
I wouldn't worry about what other people think. In the end, your son is most important and you are his best advocate. I had some people in my family very critical of my when my daughter was little because they thought I was "pushing" her too. Bottom line, I didn't want to look back in the future and say, "If only I had done more." One thing I have learned is that if you think people in your family are not going to support your decision, then be selective about what you tell them about your son.
Good luck,
Barbara