I've had a similar problem with my husband comparing our oldest two children. Our oldest (a boy) is quite advanced. He was reading some easy books by the time he turned 4, and by the time he started kindergarten, he was reading at an upper 2nd grade level. He could count to 100 before he started kinder (he was so proud that he counted to 100 before he was even 5--though he turned 5 the next day!) He could also do some simple addition and subtraction, and his ability to reason sometimes astounded me. (And by the way, for all of that, he didn't potty-train until about a month after he turned 3.)
Our second (a girl) was born when our oldest was almost 2 1/2, and my husband has often compared their development. I have told him numerous times that it isn't fair to compare them when we simply haven't been able to devote the same kind of attention to her that we devoted to him; it's just not possible with multiple children. The thing is, she's quite smart, too, and she's now reading quite well and won't start kindergarten until next fall.
But we had one moment, back when my son was in kindergarten, where she was completely demoralized by something her father said to her, and I think it was finally a wake-up call that he needs to let up on the comparisons. She was 3 1/2 or so at the time and was 'reading' a book; it was a very repetitive book about going on a walk on a farm with basically the same thing on each page, but a different animal and its sounds. She was reading it saying, 'I went on a walk. I met a pig. The pig says oink-oink.' Well, my son sat across from her and kept saying, 'No, the pig says oink, not oink-oink.' I kept telling him to leave her alone; she was doing just fine on her own without his help, but he just wouldn't let up. Finally, my husband steps in and says, 'It's okay, son, we all know she's not as smart as you.' I jumped all over him (I was PISSED!), and he defended himself by saying that he meant it 'relatively,' but I maintained that children don't understand 'relativity' and all our daughter heard was that daddy doesn't think she's as smart as her brother. I was talking to her that night, telling her how smart she is, and she said in a pitiful little voice, 'No, I'm not. I just write my name in capitals.' (Another thing my son had chastized her for.) I told her her brother used to do the same thing before he went to kindergarten and it didn't mean in anyway that she's not as smart as he is.
It was really at that moment that I decided to push the reading a little harder with her than I did with my son--and by pushing it I mean I did reading lessons with her without forcing her to do them when she wasn't interested (the book Teach Your Child to Read in 100 Easy Lessons is fantastic, by the way; we've only finished 60 lessons or so (that's how much I've 'pushed' it!), and she's reading really, really well). Anyway, I felt like learning to read would give her a boost in confidence, and I think it has. She's still really self-conscious, however, about her dad hearing her read. I think it's part of her shyness and lack of confidence that she doesn't like for anyone but me to hear her read, and she doesn't like answering people's questions because she's afraid of getting the wrong answer. I tend to think that a lot of this is just her personality (I was also really insecure and lacking confidence as a child), but I sometimes wonder how much this stems from her brother overshadowing her and her father's expectations that she live up to his (her brother's) standards, but at least my husband has ceased to compare them quite so much, and never in her hearing again.
I hope you are able to make your husband understand how much this could hurt your son's feelings if he overhears him say something and destroy the confidence we all want to see our children have.