Hi M.,
This is not a silly concern. I think you're a compassionate mom who wants her son to be a strong man when he matures. Maybe you could get him a Big Brother from the Big Brother/Sister organization. They are adults who spend one on one time with the kids. Sports is also a good avenue but you have to make sure the coaches are the types of role models you want for your son--don't put him in something simply because the person is a man. You can also look for male babysitters.
My husband was raised by his mother and has a sister. Surrounded by women!! But he is the most gentle, loving man while being a secure, confident man who will fiercely protect those he loves. Don't sell yourself short--a strong, confident woman can raise a strong, confident man.
You're right in working on this issue early. Let us know how it goes!
D.