D.E.
Some people just freak kids out. If your neighbor has never been alone w/ your son, then Id say there's nothing to worry about. If he has that is a different story.
My BIL who is one of the nicest guys you'll ever meet, unfortunately caused most of the babies in our family to bawl their eyes out every time he came in to the room. Once they hit the toddler-ish phase they all love him. We think it was his voice, because we KNOW he'd never done anything to them.
But maybe your child senses something about your neighbor. It's hard to tell if your child isn't communicating well.