The KEY words in your post are these:
"I was very skinny as a child and so was my husband."
You almost buried them at the end, but there they are -- your clue to what's probably going on.
Please, please do not let your own feelings that "I'd can't stand to see him so skinny" influence you to the point you drive yourself, him and your husband around the bend.
You don't say if the doctor recommended the nutritionist because the doctor now is concerned about your son's development (though you say earlier that the doctor was not concerned -- which is it?). It's fine to see the nutritionist but also be aware:
Your son possibly has a very, very high metabolism that means he burns things very quickly. Has the doctor ordered any metabolic tests on him? Has the doctor done blood tests or other testing to ensure that he is actually absorbing nutrients when he eats? He should.
There are conditions (celiac disease is one) where a person can eat and not absorb nutrients and end up very thin and undernourished. Or it could be metabolism. But if the doctor is now concerned, the doctor should be ordering tests rather than just bringing in a nutritionist. The nutritionist won't help a whole lot if your son doesn't absorb fat or nutrients from his food.
BUT I want to add: This is not terribly common, and a fast metabolism and tendency to be small and thin are genetically very likely from what you say. If you focus like crazy on his size (assuming he is otherwise healthy, no conditions, absorbing nutrition, etc.) and the issue is simply that he is small and is always going to BE small due to genetics -- you will all lose your minds with doctor visits and poking and prodding. Poke and prod enough to be sure he's OK and to do due diligence, but if he is OK, let it be.
Here's why I offer this: Our friends' son was tiny. Really short and thin as a baby and toddler. And they went to specialist after specialist, and some doctors said nothing's wrong and others scared the heck out of them with "It's such a mystery, we're sending you to yet another program" talk. They pretty much ended up with ulcers from the stress of fearing their son was "failing to thrive." They were stuck by one doctor into a program they learned was actually aimed at teens with eating disorders (their son was a TODDLER at the time--they quickly told that doctor to take a hike). Then finally one doctor actually looked at mom and dad and said at long last: "You are both very short. And dad is short and VERY, very slim (dad could eat pure fat forever and never gain). This is not some condition or problem, this is genetics. Your son is developing fine relative to his own previous benchmarks. Go home."
Tons of doctors, time, and especially stress, to find out that no doctor had really just looked and and questioned mom and dad when treating the son.
I am not saying your son is dandy and you should do nothing. Do get this investigated. But always keep in your mind that your son IS you and your husband, and if you were both "very skinny" children as you note -- that is showing up in him too. You may reach a point where you need to accept that he is small and put aside your own emotional reaction to his size and desire to see him be larger.