Hello Dear,
It will depend on the outcome of the case; generally if the judge allows him to see the child, let say every other week-end, then he might order you to move back, because the law will not allow you to prevent the other parent from seeing his/her child, especially if he is of no danger to that child. Start preparing yourself for that possibility, both parents deserve to have access to their children because the judge always looks at this as being for the child best interest; it is not about you or him, it is about the child and she deserves both of you.
Be brave and let your daughter benefits from the love of her parents. I grew up without my father and believe, not 40+, I did not like my Mom decision to not let him be involved in life, I searched when I became adult, I come not go back to those years, but I wished we had those special moments; she is a girl and his dad attention is very determining for a lot of issues later.
Do the right thing, let the emotion aside, il will pay later and you aill never regret it, if he is not of a danger for the kid.
Vava