Are you hurting your child healthwise? I don't think so. Knowing what my kids and my nieces/nephews have experienced, they are destined to get sick whether you start them in a group setting at 2, 3, 4, 5. It doesn't matter. I, too, have heard that them getting these illnesses now will often make them more resistent to getting them later on in life.
Will his immunity build up by the next cold/flu season? Most likely not. Again, what I have seen is constant or near constant colds, sniffles, etc for a few years etc as long as they are in a group setting. I believe it takes years to build up immunity.
If you aren't comfortable with this situation you could find some alternatives that don't meet as often and may have fewer kids: playdates with friends or join a playgroup, library programs, etc. Good luck to you. Hope your little one feels better.