My first reaction is that, "believe" in divorce or not, I am sure God would not want you staying in a relationship that you get "thrown" across the room. However, that was not your question, just my opinion.
If you are interested in having this work, you both need to go to counseling. And, really I think a licensed therapist is what you need, over a pastor or other church employee. That way you can both get out how you feel in a safe environment and he can really hear how you feel about his constant allegations. Therapist's have an amazing way of making couples hear each other when you cannot do it on your own.
Regardless of what you decide, please don't allow yourself to be treated that way again.