No, you are not uptight. A BB Gun should be a supervised hobby. And it should be done at a range - not in a drive way - the BBs can stray.
Gun safety and safe handling is something that EVERYONE should know. Guns MUST be respected. It is a TOOL not a toy.
You can contact your Local NRA and get lessons on proper handling of fire arms - education is knowledge. Do NOT be afraid of guns. Not saying you are - but education goes a LONG way in safety.
If you are not comfortable with your son handling a BB Gun - I would ensure that he was taught the proper handling of ANY fire arm...a BB Gun can still be dangerous in the wrong hands.
Talk to the neighbors. Find out what their stance is on gun safety and your concerns that their son is using the BB Gun in an inappropriate place...if they balk - then obviously they are ignorant of safety.
You can teach your son gun safety. If he goes to their home - and guns are not in a locked safe - he really should leave the home - this is assuming of course that you already spoke to them about your concerns and they balked.