First of all, I'm so sorry you are going through this.
Secondly, don't apologize for saying what you said during your son's early years. There's nothing wrong with what you did. It's perfectly normal that, as he gets older, he's going to have more questions and need more information.
A lot of how you answer this question depends on your religious beliefs, which are so personal that it's hard to suggest answers to you. I think the links that B provided might give you a start. If you follow a particular religious tradition, you might speak with a member of the clergy of that religion or denomination. Ministers, priests, rabbis, and all other spiritual leaves deal with questions of death all the time, and they have experience in dealing with children who have lost someone.
Depending on your beliefs, it may be that your son is old enough to start to grasp the concept of there being different parts of people. There's the part you can see (the body) and the part you can't see (it may be too hard to explain "soul" but maybe he can understand the rest: the thoughts, the personality, the emotions). You can see the person's smile but you can't see their happiness. You can see them hug someone but you can't see love. Maybe it would help to explain that there were different aspects to his father - the body, and the intangibles. Perhaps you can start to explain that there is no body for Dad to occupy so there is no way for him to come back, because what is left is the thoughts and the love.
It may also help to start showing him that there are many types of families - not all families have a dad, not all have a mom. Some don't have either - the grandparents raise the kids. IF you are comfortable, you can tell him about friends of his who have 2 mommies or 2 daddies. There are adoptive families with no bio conniption t the kids. There are divorces and deaths.
The main thing is to find a series of explanations that you can share over the years as he gets older and has a greater capacity to understand as well as a greater need for more sophisticated answers. Make sure that these answers meet your beliefs and aren't just answers that you think will satisfy him.
It's okay to get some counseling around this, either for yourself or for your son. Good luck.