A.V.
Absolutely. I do little trips for my birthday and we've been there and to Monticello and brought DD both times. DD was not quite 3 when we went to Mount Vernon. That also has a museum that shows George Washington as a young man through his old age. Heck, you could spend all day and still not see it all. Just put someone in line for the tour and let the kids run and trade off. We let DD run around while we waited in line. Also, go check out the pole barn and farm. Watch for loose sheep. :)
It rained the day we went and we spent some of the raining time in the barn talking to the very interesting reinactors working there.