We've only taken one really big vacation as a family of four and it was a Disney Cruise. It was AWESOME! So awesome in fact that I just booked another one for next summer. : )
This summer, we are spending a week in Boston. We'll be visiting family, but also hitting up some of the museums and tourist spots. A couple of other friends have done Boston recently as well and both said their kids had a blast (our oldest kids are all 6). There are a lot of different things to see and do, plus public transportation is pretty easy.
Southern California (where I live) also has a ton to offer. You kind of need to choose between San Diego and LA/Orange County, but you could do a few nights in one and then switch hotels and go to the other. San Diego has Sea World, the San Diego Zoo, the Wild Animal Park and some great beaches. There's also an old aircraft carrier down there that I think does tours. Skip Legoland - only your 6 year old would like it.
Orange County has Disneyland and Knott's Berry Farm. We have great beaches, malls if you like to shop, and more movie theaters than I can count. Laguna Beach has amazing art programs in the summer - the Sawdust Festival, the Festival of the Arts, and Pageant of the Masters. Pageant is an INCREDIBLE show where they create living art pieces and I would say don't come to OC and skip it. Plus, Laguna Beach has a great beach, lots of restaurants, and fun/funky shops.
Then, of course, LA has a ton of stuff to do. Santa Monica with the famous pier and the nearby Venice boardwalk, Hollywood, Beverly Hills, Griffith Observatory, good museums, and tons of great restaurants and shopping. And so much more.
So, if a Disney cruise is out of the question for this summer (or any cruise really), I would say come to So Cal and you'll have a blast. It's not nearly as hot and muggy as the east coast (or TX).