Colds will always happen and I agree that it's not fair to send him away for a family holiday gathering. I think I would tell the parents of the newborns though so they can make their own decisions. When my son was born I was nervous about him being around people anyway, so I would have appreciated the information so I could plan accordingly. At least that way they can do take extra precautions (extra wipes used on things, extra careful with who's around them, extra handwashing, etc.) if they do decide to attend. Also, you might want to talk to your son about staying away from the babies to do his part for keeping them well too.
As for the other children, it sounds like they're older and probably more immune to stuff. I would probably still reinforce to your son to wash hands frequently and make sure he covers his mouth (with his elbow) when coughing and all that kind of stuff.
Good luck with your gathering! Hope everything goes well!