S.E.
I have never lived in a neighborhood that had kids exactly the same age as my own. They have always played with kids older and younger than themselves. I do let them roam the area (one block around the house with a playground) with reasonable time limits. But we live in a very safe area, I know of 5 police officers that live w/in a 3 block radius, and the rest is about a quarter military (with families) and very attentive mothers, so very little can go on.
The thing is, you are going to have to start giving your son some freedom, or he going to start taking it and you won't know what it is that he is doing. Age difference doesn't bother me so much, you can have good and bad kids at any age. It is better to know the kids your son is playing with and giving some trust.
Second, listen to your husband about what it is to be a boy. You don't have to agree with him, but he will have a better perspective than you do about how a boy feels and what trouble he is most likely to get into. Girls and boys are equal, but they do think differently. Then explain what you feel is best and why. The best solutions are usually found somewhere in the middle.