He might have received recessive genes from both of you to be small. He might just be one of those kids that don't grow until they're adults.
I had a friend in high school that was about 5' at graduation. He's at 6' now. He grew in college.
My grandson was nearly 5' in 3rd grade. He's been 6' since middle school.
Some kids grow early, some grow right where you think they should be, and some don't.
I was a nanny for a family for several years. Both parents were larger people. Mom was larger and short, dad was larger and tall. Each of their kids were hefty kids except one girl, who was absolutely normal. But mom took her to doctors and worried a lot about her looking like a refugee camp detainee. I sewed girly dresses for the younger girls and this one, only one child, was exactly where kids her age were supposed to be.
Perception is everything but when a doc says there isn't anything to worry about you need to take a breath and consider that you might have a skewed point of view.
I costume a LOT of kids and I can tell you there are a LOT more smallish kids than fat obviously overweight kids. It's not that big of a deal if the doc is okay with where they are. IF the doc is worried then you need to listen to them to see what can be done.