I started my daughter in swim lessons at 3 months. She is now 12 months and loves going to swim class. It is very comforting knowing that just in case she were to fall into the water, that she can swim. Most swim schools actually recommend parents to start their children in swim school before they are 6 months old. Babies before 6 months are natural swimmers and more likely to adjust better to the water since they still remember being in the womb and hence find the water comforting. After 6 months, they say that your child is less likely to adapt as well and as quickly to swimming since they aren't likely to remember being in the womb. Swim schools especially say that if you child shows any sort of fear towards water before 6 months to definitely start them in swim lessons because it is not natural for babies, at that age, to be afraid of water.
I am a big advocate of starting kids early as I have found it to really help my daughter. I, as your husband was, started swim classes at 4 and I hard a very hard time getting adjusted to the water. Putting my head under water terrified me! As for my daughter, she is swimming under water and kicking for about 5ft or so all by herself and she is very confident with her ability.
I'd recommend researching swim schools in your area. There are only a select few that provide "infant swim" classes. I would then go and watch one or two of the classes to observe the style of the class and how the teacher interacts with you and your child. (I went to a few different schools before I found the program that I felt was best for my baby.) Taking your husband to watch a class may help open the door to agreeing that earlier is better and that many parents do start their kids in swim school long before that age. (In fact, most swim schools start at 6 weeks!) Hopefully this helps!