Good question.
I always took my son to the restroom with me, even at 5. Even at 6 or 7.
Now, it depends on the circumstance, but I'm a single mom. If I took my kids to the fair or some other event where there were tons of people, I wouldn't dare let my son in a bathroom with men unaccompanied.
And, I never got any weird looks either.
I think you should never worry about the look on a woman's face. Honest to God, my son and I were in a store just before Easter and he mentioned to me that this older lady kept giving him filthy looks. I didn't see it, but sure enough, when we got in line....he nudged me. There she was. She looked madder than a hornet.
I just told my son, "Honey, I can bet you that's the same face she woke up with when she got out of bed this morning and it's probably the same face she'll wake up with tomorrow."
I let my son go into a male bathroom, but it was usually after I watched and made sure that there was no one else in there. And, I've let a stranger watch after him, which I know sounds weird....but I could tell the guy had his own boys in tow. I stood right outside the door and listened.....
I can't tell you how many times I've been asked by a dad to take their daughter into the ladies room to potty because he didn't want her in the men's room and ladies might freak if he went into the women's restroom.
At 5, they might not so much need the holding hands thing, but when it comes to public restrooms, we have to be careful and we have to let them earn our trust too as far as not wadding an entire roll of TP into the toilet and flooding it or something.
A 5 year old in a women's restroom is not that strange in my opinion. Like I said, if it's a case where you might get separated or not know what's going on, it's best to keep a little kid close. Not all people are freaks or anything, but better safe than sorry.
Again, just my opinion.