You definitely deserve better.
Part of your difficulty as I see it is you are giving him way too many choices in your life. There is no way I would let any man float between me and another woman. If my man was with another woman I am quick to cut the strings and let him be with her. There is no floating or swinging back and forth. My life is too precious for all that high risk behavior.
I didn't even give a guy a chance if there was any residue of an ex on him or around him. I'm more of the there can be only 1 camp and I'm that only 1. I was single for a very long time (18 years) before marrying my husband. Well worth the wait.
You should go to court to establish child support, custody and visitation but be prepared. Your post partum emotions won't play out in court and if you take your emotional self into the courtroom you run the risk of loosing custody at the worst or sharing custody at the least with a liberal visitation.
You won't be permitted to dictate who your daughter's father allows or permits around your child without some concrete evidence that she posses a threat to the well being of your child.
On the other side, I was the girlfriend to a guy who had a daughter while we were together. His baby's momma got pregnant before I came on the scene. He wanted to be a father to his daughter and I didn't say or do anything to prevent that from happening. While his daughter was about 3-6 months old I found myself babysitting her. I had no children at the time. It wasn't really a great experience for me. Me and the guy broke up when his daughter was about 4 years old but not because of her or her mom. He was just aweful to me and I finally had enough and was determined to get just that for myself. Fast forward to today, he actually is married to his daughter's mother and they have been married for about 2 years but it was to keep her on his job's insurance. They live in separate places and have separate lives. It's really weird but they are a family and even vacation together. He has always been a great father, just a lousy boyfriend to me.
Her father may not be comfortable with taking care of a baby which is normal for many men. He just feels more confident with a woman present.
Again you can't stop him from making the choices he wants to with his life. As it affects your daughter, you will need evidence of poor parenting which may be addressed by social services or through the court system.