Hello! =)
I didn't read anyone else's responses, so I am sorry if I am repeating what anyone else said.
If this were ME, here is what I would do. I would say "no thank you" to any play dates that require either one of you Moms to be the only caretaker. Don't let her come over without her Mom and don't leave your daughter at her house if you can't stay. It's a very simple thing to do. I am friends with all of the mothers of my kid's friends. In fact, my kids only get to spend time with their friends because I became friends with the parents first. My neighbor will come over for the afternoon and bring her kids. My neighbor and I will hang out and the kids will play. This way, everyone is responsible for their own child. We don't have an issue with the kids hanging out at either house without the other parent staying, but quite honestly, if my kids are going to play with their friends, I want to "play" with mine, too. =) Neither one of us Moms want to be the babysitter (though again, it wouldn't be an issue, and we do babysit for each other when needed). We just take the time to socialize with each other while making sure that our kids are playing and learning to do it in a healthy way. Now some Mom's are not into that or can't do that because they have things to do. For my neighbor and I, we just don't get together if we have other things to take care of that day. Our kids (all together) are 1, 2, 5, 7, 8, and 10. Some need more supervision than others. The older ones can all play together nicely. The point is that if someone is doing something they shouldn't, it comes out really quickly and the correct parent can address the issue. Solves a lot of problems! =) Also, by spending time with the mother, as friends, we know each others parenting style and expectations. So when I leave my kids with her, she already knows what I would or would not want them to do.
As far as the going to the man's house, I would have lost it quite honestly. Just because he is their "good family friend" doesn't mean that he is yours. The fact that they went over there without her knowing is a huge issue. I am the type of parent that doesn't even leave my kids in the front yard when I run back in to get the sunscreen. This isn't because I am sheltering them or because I am paranoid. I have just made a very deliberate decision to NEVER be the reason why my children have a horrible or tragic experience at the hands of someone dangerous. If someone snatched them up, it would be my fault and not theirs. While rare, it happens and I will not be the adult who puts them in that position.This goes for friends houses, too. My rules don't change but should they find themselves in a situation that they know I wouldn't put them in, I have prepared them as best as I can for it. My kids know how to tell an adult, even one they don't know very well, "I know that my Mom would not let me do that." or "can we call my Mom and ask her if that is ok?" I believe that four and five year olds should not be left unattended in a backyard, if there is a way out, even if the phone rings. A phone takes a few seconds to answer and if it's a cordless phone it then goes outside after being answered. To be able to make it all the way next door or down the street without being noticed does not make sense if someone is just answering a phone. That happening has nothing to do with the child, but the parent who is supposed to be watching both children. I say, do not compromise on that one. If this had happened to my 4 1/2 year old, that would be the end of all visits (unless I am there) just based on that.
I don't think there is any reason to say anything to the mother since it sounds like she pretty much knows. I would just try to make this a friendship that keeps your daughter's best interest before anything else. I am sure this is a situation that you wish you weren't in, but I say make the best of it. If you want to save the friendship and keep it a mother/daughter play date, at least you will know that your daughter is safe and a 5 1/2 year old won't have as many opportunities to be so sneaky. Hope it all works out!! Hugs!