Some kids are just higher maintenance. Two of my three were like that as toddlers (one of them still is at 8). The best bet is to avoid issues as much as possible. If he can't sit still at the library story time, don't worry about taking him (you're not a 'bad mom' for not going to story time!) Try to get him into more groups with older children, since he is fine with them. And watch him carefully when he's in situations where there could be problems and remove him from that situation at the first sign (before it becomes a problem).
As far as other parents, you're going to have to develop a thick skin. I've had other parents, teachers, and church kids' group leaders ask me about my son and/or give unsolicited advice. I've been told he might have everything from aspergers to a seizure disorder to ODD (and none of it by a doctor), and that I should be more strict ("I would never let my kids ... ") etc etc etc. I just decided, I know my son better than they do. And I know all the wonderful things about him to. Try to focus on the good things, and help him work through the challenges. You're taking steps to screen him for issues, and you said that people who know him have seen improvements, so you're going in the right direction. It may not be over quick and easy, you're going to have to be patient and work with him a lot. But you also know how wonderful he is and its worth dealing with the challenges to see his big smiles and hugs and kisses.