V., I can relate. I was a single mother of two and my ex-husband saw them just a few times before he moved to Flordia 6 yrs ago. He has not bothered to call, write or even visit, and they are fine. My oldest daughter has even stopped talking about him.
He would call me just to start a fight, knowing full well my oldest daughter would be around. I just stop answering the phone, he was very verbally abusive. I feel my kids did not need to hear it or be around it. I felt I was better on my own and I survived very well.
You need to realize that all he is doing right now is hurting those kids. Besides the fact that he is showing him immaturity by telling a 3 yr old. So what happens the first the son sticks up for you to him. My suggestion would be to stop letting him come over, and yes it will be hard the first time, but I would rather be safe than sorry. Then once you go for support, you can tell the Friend of the Court you want surpervised visitation because of the way he is acting.
Then as for him signing off he can not sign off until you are remarried. My ex-husband went as far as to tell the court the my second daughter was not his, well needless to say he would not even take a blood test.
You sound like you have your head on your shoulders, working and going to school.
You and the kids will be okay.