Actually I agree with you, this is a REALLY good idea to get them involved young. I grew up in the Martial Arts, and later taught young children, and loved every minute of it.
If your husband is worried about agression, he does have a concern there, but I would recommend looking into the more "Defensive" martial arts, like Aikido (if you're in the greater Seattle/Redmond Area, look into aikido kids) I grew up in "Jukido" which was a combination of Aikido and Judo. (Jukido - The way of gentle flowing power) And let me tell you, this is GREAT stuff for kids. Because it requires an agressive action AGAINST them, before they can do harm. it's really hard to just go up and throw someone. :)
Judo/Aikido are very "Defensive" types of Martial arts, because it teaches a defensive posture, how to fall properly without getting hurt (which is good for all kids). and like what I learned growing up, is that "Size doesn't matter", and it allows someone smaller to utilize the motion and agression of someone bigger against them. (The bigger they are, the harder they fall).
Instead of Blocking a Punch (physical connection) Aikido and Judo avoids and redirects that motion -- I.E. the "Bad guy" is doing all the work. :) Generally speaking Punches and Kicks aren't taught, but things more intuned to kids, (what hurts the other, but not themselves) like wrist locks, arm bars and such. Nothing quite like watching a 6 year old boy move a 32 year old adult around like a puppet. :)
And on the self Esteem side, I find these types of Martial Arts amazing for kids, because they become confident knowing that should someone try to hurt them, they will be just fine, and can deal with it. That they can fall down or get knocked down without breaking something. and a child will "Walk taller and carry" themselves with confidence, which usually scares away most bullies and bad-guy types anyway.
The art "Jukido" that I grew up in, (I got my black belt at twelve) also had a complimentary Karate program but you couldn't learn those techniques until you were 12. Because it does require control, and learning how to punch and Kick can be agressive. I didn't understand the limitations at the time, but when I started teaching (To put myself through college) I figured it out. :)
If you want, I could always suggest some good schools or instructors in the area.