K.F.
Your son is not too young for swimming. You may have a problem finding a karate class for him because those classes are instructed by people who may need to see the maturity (ability to follow instruction) of your child and may also have a minimum age limit. When the children in my house were small the rules of the dojo where they trained stated they needed to be at least 6 years old.
Make some phone calls and schedule your appointments.