Yes, it's actually past time. I don't believe in freaking out if someone accidentally sees someone else, but at this age he needs to be learning about privacy and knocking on doors. He will be in school soon, and he needs to have a sense of propriety when it comes to rest rooms and so on. He also needs to have a sense that his body is private and not meant for public display - that's a safety issue and a question of knowing boundaries. Just because a parent sees a child naked doesn't mean that the child should see the parent that way.
I think explaining bodies to kids is a good thing - they should know about anatomy and the real names for body parts. But I think you do that with books and not with them inspecting or getting a glimpse of anyone's pubic hair or breasts. If you still take him in to a public ladies room with you, I get that - it's a safety issue. If you feel you must take him into the family restroom or the handicapped stall, he may get an inadvertent look at your rear end, but it doesn't mean that you whip your panties off and let him inspect you while you pee - see the difference? If he can use a separate stall himself while you are outside, great.
The thing is, you don't want him discussing other women's (or men's) body parts with them and comparing them with what he sees at home. I'm not a prude by any means but I think you need to give your child the skills to navigate the real world - and seeing other people naked is not part of that. He can't be going to Grandma's or to play dates and thinking it's okay to walk into someone's bedroom or bathroom because "it's no big deal and I see my mommy all the time." And other kids need to learn from their own parents - and later on in health/biology class - and not from their friends.
I think you'll be glad if you start this implementation of boundaries immediately.