There is a really nice Children's museum in Portsmouth, which is probably half hour from VA beach. Also in Norfolk, there is currently a really awesome Pirates exhibit at the Nauticus. It is a temporary thing, so you may want to research first to find out whether it will still be open when you are there, but my sisters and I all thought it was great, and if our children were older (toddlers right now) we would have loved to have taken them there.
My sister lives in Norfolk, and the drive to VA beach took about 20-30 minutes during prime beach weather (August) so probably would be on the shorter end of that travel time. Water will be cold in April...
Williamsburg was a lot of fun, and it's pretty much at your own pace, so that helps with kids. You can tour buildings you want to and skip the ones you don't. It's very open there, so if your kids like to run around and need space, there is a lot of it in Williamsburg.