I would speak to the youth group leader and the pastor together, privately and immediately, and if they do not see the issue here, and if they insist this man must be made welcome in that particular youth group/class -- well, I would either take it beyond them to the church board (if there is such a thing in your church) or I would with great sadness worship elsewhere.
The only other thing I can think of is to enlist the help of other parents of girls in your daughter's age group. If several of you, and not just you alone, go to the class teacher, pastor or board about this, it may have more effect. Go in a loving and Christian spirit, noting that this man has as much right as anyone to be in a church family, but also noting that for his own protection as well as the children's protection, the church and teacher have to treat him as the adult he physically is and give him a more appropriate role than one in a youth group.
It's surprising that someone in authority in the church does not already realize the potential for the many problems here (as others note in so many ways, so I won't go into it here) as well as the potential for the church itself to be held liable if something awful happens on church property. Unfortunately, the church may only respond to this issue if you bring up that very aspect -- legal liability falling on the church if this man crosses boundaries while on their property. And to me, some moral liability falls on the church if anything happens anywhere else with a child he met through church.
"He'd never do that, he's an innocent, basically a child mentally, and we need to be kind," is what they may say, but they need to be educated by someone like Gamma G. below -- someone with real-life experience, who knows that at 30 years old, this is a grown man with a grown man's sexual needs and interests -- but possibly without the social abilities and "brakes" to know what is appropriate.
You may be told that you're prejudiced or that you're assuming he's a bad person, etc. You need to either get the people in charge educated about what is appropriate for a grown man in his situation (and it's not being with middle schoolers), or you need to move your family. Sad all around.
Please post again and update us on what happens if you decide to speak up about this situation. I would really like to know what happens here.