Ok, you do not have to let this man rule your family. You need to DOCUMENT DOCUMENT DOCUMENT everything that shows your ex is a negative influence on your son. Write down anything your son says, especially about his father hitting him or saying or doing anything inappropriate. You need to write down anytime he is late coming to get him or drop him off, anything that is negative.
You've said you are using a public aid lawyer. Talk to him about getting Child Services involved- I know that is scary, but tell them that you feel very very strongly that your son is NOT SAFE with his bio dad and he is being put at risk when he is alone with him. Get them to investigate. They will check you out too- but if you are taking care of your son and flying straight, you will have nothing to worry about there.
Put it to a caseworker as you wrote it up above- that there are no marks, you don't want to reinforce, etc- show that all of your actions are stemming from concern for your son's well-being, and it will go in your favor.
Don't play nice. Your ex won't- doesn't sound like he knows how to. Your job right now is to take care of your son. If your ex has the judge convinced he is some kind of 'father's rights' case no matter how bad a father he actually is, that can be hard. But there are ways around it. Talk to your lawyer some more and go speak to Child Services for other ideas. They deal with these kinds of things ALL the time and can give you specific advice and other avenues to explore.
You are strong, mama!~ Be strong for your son and your new life and you will get there. Don't let a bad man keep you down and boss you around! Good luck!