Even though you don't like the guy, you're right to not forbid her to see him. My daughter went through an on agaon-off again relationship with a boy who was not good for her. Eventually, she figured out that he wasn't going to change, and that he was, in fact, getting worse, and ended it for good. She's now engaged to a young man that we all think is wonderful.
I understand not letting her go to his house because there's no supervision, but how much supervision is there if the two fo them go out? Fact is, the safest place for them to be is at your house. If she broke up wih him and with another boy over pressure to have sex, then I wouldn't worry toomuch about her changing her mind. It sounds like her mind is made up. You've taught her well, now she's applying what you've taught her.
As for whether other people might think you approve of the relationship if he comes to your house, I say TO HELL WITH WHAT THEY THINK. You do what is best for your daughter, and if other people form wrong opinions, that's their problem, not yours. Besides, you know what they say about opinions...