His response worries me. He is not a piece of china, but is the father mean emotionally to your son or nice to him? Can you change your visitation? Have it appealed or something? So that you have to be there. My papers specifically say if she goes to her dad's I am to be with her the whole time. I would try my damndest to change it... he doesn't sound like a very good father. He has neglected to see him, put a notarized statement of all those visitation he missed. I agree it's important but I always agree the child's safety is first so I would try to change it that I have to be with him type of thing of that his dad has to go to him.
Has he paid child support? If not, that is grounds for abandonment and losing his rights I believe.