My gut instinct is to say age 6 or 7 might be old enough. (my son is only 3, so we'll see.) But I really don't think 4 is old enough to go it alone in a public place.
As a woman, I think age 8 is too old to be going into the ladies room with mom (Unless you are in a really questionable place and need to be there for safety). I feel weird when I see a boy who's 8 or older in the ladies room.
I have to agree also with the other posters who said it depends on where you are. If my son could take care of everything himself for #2, which is when he wants privacy, then I would let him go alone at church for instance even at 3 yrs old. But say I was in Las Vegas- depending on what part of Vegas we were in, and what time of day/night it was, he'd have to be like TEN before I'd let him go alone, and I'd STILL stand right by the door. (not that I'd take a 10 yr old to Vegas...)
Bottom line, do what you feel comfortable with.