A friend of mine recently went through this and he did end up having to go to court. Her son was the name on the lease and she was living there without providing any assistance financial or otherwise. He told her she needed to leave and she refused. She had her son committed to a mental institution! After three days of observation the DR released him and he talked to a judge who said he would have to take her to court to get her out of his place!! He did, along with his brothers to testify that their mother was nuts. She was served formal eviction papers and she did leave.
Hopefully you won't have to go to that extreme. I would just tell him that right now you need to focus on you and the kids and ask him to please move back to his place. You could also talk to your landlord and ask them- they'd know. Definitely explain that he has his own residence, because they don't really like people living there 24/7 who are not on the lease, but tell them you are concerned because you've begun to receive mail for him at your address. He spends the night there sometimes and you want to break up with him but it looks like he's wanting to reside there against your will. See what they say.
S., mom to four, bedrest and preterm labor with #1-3.