My oldest is super tall and super thin. The doctor was telling us to try a bunch of things, but we were already trying those. At four years, his percentile numbers are the same as when he was a little bundle. So we basically don't worry about it now, as adjusting to his feeding has become second nature.
We buy full-fat yogurt always. Most are lowfat or fat-free. We had to search to find the full-fat one, and it was delicious! We also use olive oil rather generously, use whole milk always, and try to find the fattiest food we can. (I mean healthy fats, not "loaded with sugar" fats.)
We eat a lot of potatoes, pasta, and whole grain breads, so we know he is getting full. He just burns it all off!
If your pediatrician isn't worried, and your son looks happy and healthy (although not as chubby as the baby down the block), you need not worry. It doesn't mean you won't! But you don't need to.