I agree with Leigh and Dawn. While you do not want just ANYONE to touch your baby, you don't want to keep him in a bubble either. So many children get sick nowadays just from going outside because people have been trying to keep everything around them disinfected. I had a problem with both my boys, I guess they were too cute. My first one spent the first three years here in the States, where people looked at him, some times held him and were on their way. I spent five months in Mexico with my second one, when he was 4 months old. Down there, just from family, he was being held by about 20 people on any given day, and I couldn't tell you when they washed their hands last. The result: my younger one rarely gets sick, and while my older one is still pretty healthy, he has more allergies than the little one.
Anyway, I'd recommend you play it by ear. Let some people touch him, but you decide who, and to what extent. When my older son was little, I held him next to me and held his hand, and kind of waved it, keeping him at a distance when people looked at him, saying "Say hi, baby!" and if they wanted to touch him, even hold him, I would hug him tighter and say "no, that's ok, I'd like to hold him right now." And really, you don't need to care about other people's feelings, period.
Good luck, and congratulations on having such a cute baby!