You need to lay down specific rules regarding him visiting her.
He can legally do what he's doing because you didn't get specific in the papers.
My daughter shared custody of her son 1 week with dad and 1 week with mom.
Then when visitation started up he'd start an argument with her at his house then have her arrested for trespassing. Domestic issue, to make it where she couldn't come see her son when he was with his dad.
During her visits he'd get off work at 2:30 am and drop by. If she didn't let him in to visit the kiddo he'd call the police, show them the papers, and they'd tell her she had to give him access to his son. So I think you should have made your visitation paperwork much more regimented. As it is now he can come any time he wants just as you can pop in at his house at any time. You need to fix this.
When it was her time, when she had kiddo, her apartment got broken in to and all the diapers, formula and food she had was mysteriously gone. When the police showed up for a welfare check, they'd had a report she had her son and no formula or diapers for him.
He also damaged her low income apartment and stole her bedding items. So she was sleeping on a bar mattress...
So the police would come and she'd have to go buy a bunch of stuff for her week.
I finally told her to keep her stuff at my house and I'd bring it over while she was going to get her son. Sure enough when we went to her house her food and stuff was gone again. I waited until she left then took the boxes of stuff from my car. When the police officer showed up later the kitchen was full and the beds had sheets and blankets as needed. He eventually stopped doing this.