Oh, J., I have so been there (with the big baby comments). My son was (and is) extremely tall (he's 9 and I'm almost looking eye-to-eye with him, and he's always the tallest boy in class, sports, etc.). As a baby, he was long and chubby -- but his height and weight were proportionate, etc. At 12 weeks, he was in 12 mo. clothing...
Anyway, I got all kinds of comments on his size. I found older men were the kindest (things like, that's a future linebacker you have on your hands); elderly women meant well but came across as judgmental (like, what are you feeding that child? And when I'd answer breast milk, they'd say, that must be some super-powered milk!) ... but it was mother-aged women in their 20's - 40's who were the worst. Everywhere we went, I heard about how fat he was, how chubby he was, how much do I feed him, are you worried about his size, etc.
It got to the point that I was tired of people saying foolish things and being rude. So at the post office one day, when the woman behind the counter was going off on a tirade about my son's size, I told her that I would think she'd be more sensitive to making fat comments about people given her own obvious weight issues.
That stopped it pretty quick.
Generally, I just smiled at people and said something about how blessed we are that he's healthy, but when people went over the top, I felt no need to hold back. Might not be a solution you care to follow, but it worked for me (and it felt SO much better to not stand there and bite my tongue while someone was going on and on....)
As for touching, grab their wrist and ask them to please not touch your child. They have no right to do so. It's like when strangers would touch my belly when I was pregnant -- why on earth do people think they can do that?? It doesn't stop in infancy, either; people keep thinking they can pat my daughters head, tickle her chin or belly (she's 5); usually it's the baggers at the grocery store or older folks, but it still makes me crazy. I usually say something about not liking people to touch her (and they usually look at me like I'm whacked) -- but it scares her a little, and I think it's completely inappropriate.
Bottom line is do what makes sense/feels right to you...