I am currently getting a divorce and have a 4 yr old daughter ... from what I understand, unless you can be proven to be harmful to your child (abusive, alcoholic, etc.) there is no way the baby will be taken from you. NOW, if the father decides he wants visitation, etc., he most likely will get it unless he too can be proven to be a danger to your child. You mentioned there are two possible fathers ... do they both want custody? The good news is that if the guy who says he wants to take the baby away from you IS the father ... by starting this whole thing up, and wanting to be a father, he will be responsible for child support. I know I am rambling, just trying to fit it all in :)
Don't worry so much about what he can do, as I mentioned, unless he can prove you are unfit he won't get his way - at least fully ... plus once he sees how much work babies are he may decide he wants no part of it (good for you not so much for your child)
Whatever happens, remember that the father is part of your baby and regardless of how you feel about him, your child should feel safe and happy having a relationship with him should he step up to the plate.
Hope this helps - now enjoy your pregnancy!