For now, I agree with your husband. Too much interference will help to "define" your child as a wimp. For as painful as it is, it's important to sit back and allow him to fend for himself. Now, if these "bullies" are tormenting other kids, then you and other parents have a right to complain. So you should keep an eye on their behavior; at least monitor whether it's getting worse.
In regard to the use of "weirdo," is there something he's done recently to draw attention to himself - good or bad? Sometimes being interested in other things, for example, (liking art over sports) makes a child appear to be an outsider. If that's the case, talk to your son and be sure to foster his interests, despite what others may think.
If he's exhibiting behavior that's odd, for lack of a better word, discuss ways he can change. Has the teacher noticed anything in regard to your son?
If he does indeed like certain activities, enroll him in one over the summer. He's likely to make other friends - and quite possibly friends from his own school. Plus, it gives him a chance to shine in something, which is a good ego booster.
Kids outgrow each other. Hopefully, your son will see that he has little in common with his old peers and will move on to others who appreciate him.