In my personal opinion it is a thousand times better to teach rather than remove.
I grew up in a house similar to yours. My home growing up was extremely conservative: no alcohol, no tank tops, no two pieces, no dancing, no secular music, no movies unless they were "appropriate"... you get the idea.
I developed a lot of anger and resentment. I realized the truth and reality of things - began thinking with my own head and not just blindly obeying... and I realized how much I profoundly disagreed with what I had been taught. I moved out when I was still in high school... I graduated on my own from a public school instead of the private Christian school I had been going to. I partied like crazy - drinking, smoking, sex, drugs, late nights and early mornings - its amazing I graduated high school and didn't wind up dead.
I wasn't taught caution.
I wasn't taught reality at all - I was simply told not to.
I drank too much on many occasions and had to go to the hospital. I slept with guys I didn't know because no one taught me not to. No one told me how to be careful if I had had one too many.. or how to tell when I needed to stop. No one told me not to drive after I had had a glass or two of wine, no one taught me not to mix certain drinks with certain foods to avoid getting sick... Nobody told me that pot was *way* more common than anyone wanted to admit. Nobody explained why I should use condoms... I was simply told "Don't have sex until you're married, and don't drink because its a sin."..... "Because the Bible says so." Sorry ....but they're going to need a lot more than that to combat peer pressure, physical urges, and plain ole' curiosity.
Its kind of absurd.
Anyway - in our house, we drink. We keep whatever we want in the house. When my boys ask me why can't they drink that, I answer them truthfully, "Because this is an alcoholic drink and you can't have any until you're old enough."
Often the conversation goes like this,
"Mom can I have this?"
"Nope, sorry babe. We still have some juice though, or there's soda in the fridge."
"But I want this."
"You can't drink that because its wine and it is only for people who are 21 or older."
"Why?"
"Because when you're younger than that its against the law to drink it... just like its still against the law for you to drive a car right now. When you're 16 you can learn to drive, when you're 21 you can learn to drink other things... but for now, you have to choose water, juice, soda or milk."
....The car thing really helps. My boys loooove cars. All cars, motorcycles, trucks, dirt bikes, etc... They want to drive NOW.
Tell the truth. He can't drink because he's too young. When he's big enough, you'll let him choose what he wants to drink, and you'll help him learn what's good and bad about it and how to make intelligent choices.
Otherwise you're only doing one of two things - insuring rebellion and a hard partying kid like I was.. or you're insuring a sheltered lifestyle that will eventually learn how things really are... and be quite aggravated and feel very deceived that you didn't just tell them how it really is.