The other posters are right - joint custody means joint decision making ONLY. Visitation is a separate matter entirely. It is very unlikely that any judge will grant your daughter's father's requests in your situation. All of your concerns are reasonable and valid. You are in a very strong position. Do not let his unreasonable requests throw you or scare you.
That being said, the courts do feel (and I am hearing that you do, too) that a child truly benefits from having a strong relationship with both parents. At this point, it sounds like the relationship your daughter has with her father is minimal. So, it needs to be built up in a way that fosters positive regard for her dad, and also does not throw her into chaos. Make the suggestion that he begin visiting her to build that relationship. Take the tone that you absolutely think your daughter needs her father in her life. Make every effort to foster contact through the phone, pictures, letters, videos, etc. Demonstrate your willingness to develop a good co-parenting relationship with your daughter's father.
However, your daughter's well-being must be the priority topic in all of this. Document everything in regards to her condition, and how well she is doing in her life with you. Suggest reasonable alternatives, and demonstrate flexibility and positive regard.
Finally, I would suggest you talk with a lawyer. If money is tight, consider consulting with a lawyer and paying for that time on an hourly basis, instead of paying a retainer. The legal arena is confusing, and if you do not have all of the information, you could agree to allow something that a judge would have NEVER allowed. Having the information before the mediation would be very beneficial for you.
It is important to choose the right kind of lawyer. Find someone who is highly respected, and who has a preference for settling via mediation. DO NOT consult a "shark" lawyer. These are the lawyers who will try and scare you with worst-case scenarios, and take the most contentious tone possible. The other professionals involved, such as the mediator and the judge, know that there are good lawyers and lawyers that like to stir the pot of trouble. It speaks very well of you if you choose a lawyer who has integrity and good mediation skills. It will also save you a ton of money, because these lawyers have no interest in creating the kind of drama that will take years in court to settle.
And one final point: DO NOT allow your child to travel to her dad's unless you have a court order that says you have to, and that it is very clear when and how she will be returned to you. This protection is *essential*. I am not trying to scare you, but I personally would be very concerned about allowing my child to travel out of state with someone who has shown such little interest in his child. He sounds like he is out of touch with her needs, and focused on what he wants. I would specifically be concerned that he would not return her to you. This is the primary reason I think you would benefit from talking with a lawyer. They can advise how much of a risk this would truly be in your situation, and help you put reasonable protections in place. Once visitation is granted, it is extremely hard to get this reversed. So, do all you can to make sure out-of-state visitation is not granted prematurely.
Stay strong, mamacita!