I can tell you as a mother of a special needs child that working on their weaknesses is not a bad idea. Just because he is doing well in other areas, that does not mean his speech cannot be improved by speech therapy. My older son has been in speech therapy since he was around 20 months (he's now 4 1/2) and it was recommended for him around 17 months. The process can take some time so your best bet is to have him evaluated. That's great that he's communicating with you! If he catches up soon with his talking then you won't have to worry about his socialization as much when he starts school. You will be promoting his social-emotional growth by doing all you can to help his expressive speech along.
I guess I'm not sure why anyone would not want to get help for their child in this type of situation. My son is also in a research study for brain-injured children that looks at language development. In short, they compared brain-injured children who are receiving speech therapy (and have delays) with typically developing children (control group) who do not receive speech therapy. Preliminary results indicate that the brain injured children receiving therapy had in many cases surpassed the control group.
Speech therapy can only help your son. What's really the dilemma? It does not mean you are a bad parent or he is lacking in some way. Their brains are most plastic now so that is when they can benefit the most from intervention. My 10 yr old nephew did not start getting speech therapy until he was 4 years old and he still has major issues that will probably last through adulthood. Why take a chance?