Money and visitation are two seperate issues... Failure to pay will never affect a parent's right to see their kids. Even if you had a formal agreement u couldn't keep your kids from him for not paying.
In regard to custody... just hearing your story it seems clear that you would get primary custody no problem.
But here are is something to consider...regardless of your thoughts of him, your kids still love him whether they say so or not. If you go to court to keep him away your kids wiill feel guilty for loving him. Of course you want to protect them, but they will also want him in their lives even if he is/has been a jerk. If he eventually wants to see them maybe you should leave that option open... not for his sake... but because your kids deserve the opportunity to be with their dad. Get a formal agreement for custody but don't try to take away his parental rights entirely.
In some ways I wish my ex would just "go away" or that I could shield my kids from the pain he's caused... but they love him and so I have to just accept that.