Hi there,
I'm a single mom and I was in the EXACT same situation as you.
I was all about "I want to be nice to the dad and a kid should have 2 parents, even though he doesn't pay" thing...EVEN though I had these HUGE misgivings, just like you.
It went on for 5 years, and even though the father was not physically abusive, he was emotional abusive in other ways that makes one cringe to hear. By the time she was 5 yrs old and ready to go into kindergarten, I was a nervous wreck and so was she, BUT WE DIDN'T EVEN REALIZE IT.
At that time, he abducted her. That was my turning point. When I got her back, 48 hours later, I refused to let him see her unless he stopped doing (the various sicko things he was doing), and [he would have to] agree to my various requests.
Note: He also had refused to sign the birth certificate and NEVER made a single payment (except to pitch in for medical, as his reasoning was that parents who share equally shouldn't have to pitch in -- ie, he'll pay for her stuff when HE has her and vice versa). Needless to say, I was bearing the brunt of the whole kit and kaboodle
Needless to say, he had me HIGHLY confused, at the time. When I finally put my foot down, he couldn't take my assertiveness (compared to my passiveness prior) and dropped ALL interest in my daughter like a ROCKET. He disappeared and we never heard from him again.
Lastly, 2 WEEKS after I "stopped" being a complete idiot, my daughter & I RELAXED so much, we actually blinked our eyes in surprise. I HAD NO IDEA how stressed I was, and how WONDERFUL it was not having to WORRY about a seriously mentally-strained persone in our..."field".
To this day, I deeply regret that time I left her with him, as some awful things were done. IT'S ALL MY FAULT. Don't make the same mistake I did!
Best,
C., RN