J.G.
When you are alone with your son: Ask him quite calmly, without sounding like you are mad or upset... tell him you've been noticing what he does and ask if he realizes it. If he says "no." Ask him if he can try to be aware.. and maybe do it less.. or be more discreet. If he DOES realize it... maybe he'll offer an explanation (does it itch? or does it just plain bother him or what?)
I have a three year old that will do this frequently... he is ultra-sensitive which only makes a normally sensitive area well.. doubly sensitive.
If it sounds like it could be a medical issue.. seek out his doctor - but don't make too big a deal out of it if it is a heightened sensitivity issue.. just ask him plainly to be more careful when and where he does it.