I like Bvic and Mamazita's reaponses about "we don't have a daddy in our family." And focusing on what he has. It's easy for kids to understand without being a huge deal.
I'm a single mother of an adopted child with no father listed on her original birth certificate or the one from the adoption. She literally is as close to father-less as it gets. She started asking at about 3 when they celebrated Father's Day at school, and it was also when other kids started saying "you don't have a daddy.". When she'd ask, I'd say, "well, tthat's true. God makes all different kinds of families, and he didn't give you a daddy or me a husband. Instead, he gave you a Meme. An uncle, Cousin Todd..." And we'd go through the full litany right down to our dog and the dogs of our extended family! We have a very, very small family. Sometimes she'll ask "why?," but the standard response is, "I don't know how God decides who will be a family. But, he made sure that you have a family where we have lots of love to share. He gave you a mommy, Meme, uncle..." And go through the litany. Someday, the story may be more complex, but it works for the age.