As a former Child Developmental Specialist, I can tell you that your pediatrician TOLD YOU NOT TO WORRY but also told you to work on those things. There is a test called a developmental assessment. If a child is behind in three or more different areas of development, they are considered developmentally delayed. At his age, this is something that he will probably catch up with, especially if he has been sick, but if he does not catch up in all 3 areas by 12 months, he will still be considered delayed. Delays are common and all kids develop differently, but when 3 global areas, like I said, are behind, it is something to be aware of and work on.
Lots of preemies are DD, and then they catch up, or kids that were sick. But sometimes, they don't catch up without therapy. Or therapy reveals that there is a medical issue going on. Please don't be offended or defensive. She is not saying anything bad about your kid or you (I know it is hard to not feel that way!)
Plenty of kids that I worked with caught up with their peers and were just fine. At about the age of 5, if a child were still behind in 3 or more areas, that usually means they have a more serious condition.