A.L.
You need to find someone else (non Family) to speak to. Any physical contact like this between either of you is not healthy for you or your child. This is an abusive relationship and you need to take care of yourself for your child's sake.
How would you react if he tried to break your child's arm, that may be coming if you don't do anything about it now. Please please for the sake of your child if not for yourself seek outside help.
Please call Call the National Domestic Violence Hotline at 1-800-787-3224 they can help guide you.