I never heard that babies think you're never coming back! If that were true, no one would ever have a babysitter or a daycare provider, a job or a social life. Sometimes our kids go ballistic when we leave, so we feel terrible. Sometimes our kids do just great while we're gone and don't miss us, so we feel terrible! No matter what, we heap a certain amount of guilt on ourselves. We often forget how often we can't seem to please our kids when we are with them - yet when we leave, we worry about how they will manage. Your situation is worse because you were abandoned, but it's important not to project that on your own child.
You're actually doing her a favor. You are teaching her that she has more than one loving and caring parent who is more than capable of nurturing her and putting her to bed. Dads are not babysitters - they are parents. This builds her sense of security and creates a special bond with her dad - this takes away NOTHING from her relationship with you. Down the road, it will hopefully help her choose a better husband for herself! Meantime, you are showing her that going to school is important and that mommies are people too!
These are not stupid questions - they are normal emotions. But you will do everyone a favor if you try to overcome your fears! It will be fine! My son's preschool teachers taught us a little song "Your mommy comes back, your mommy comes back, she always comes back to get you. Your mommy comes back, your mommy comes back, she never will forget you." But even so that was when a parent left kids at preschool and not with the other parent.
Hang in there! The first few nights will be harder than the second week!