P.C.
If you can walk away without any feelings of anger, betrayal, etc., then, you know you're ready. You truly have to get to the point to where nothing he can say or do bothers you. Second, ask yourself would you want your daughter in this type of situation as an adult? Would you want her to live like you do? Whatever she sees at home, not matter how "dysfunctional" will be her normal and she'll accept it as such in her relationships.
I was in your position with a 20 year old step daughter. I think her goal was to stir up trouble, and did she! He, too, refused counseling, etc. I went to counseling by myself. Of course, then, I was the crazy one. Finally, after six years, I was able to walk away, not caring if he won the lottery or slept on the street. I tried everything and did my best, but I couldn't do it all myself.
Get yourself some counseling for your own piece of mind. It may help you find a way to stay, cope, and improve your home situation. If it doesn't, then, you'll know you will have tried your best and can walk away with not regrets or "what ifs". Good Luck