C.M.
I went from a Sequoia to a Highlander Hybrid, and I recently traded the Highlander in for a Cadillac SRX. I origianlly changed because I did not like the new Sequoias. My kids were almost 1 and 3 at the time I changed. The Highlander was big enough to accomodate the diaper bags, double strollers etc; and it had the fold down 3rd row seat, if I needed to take someone else along with us.
I love the Cadillac, it is just slighlty smaller than the Highlander, but it would not have worked for me if I still needed the double stroller all the time (kids are 3 and 5 now). It is large enough for my daily life, trips to Costco, etc. i have a Mastiff (190 lbs) that fits in the back. I did not go with another Toyota (or Lexus) product, because I no longer had confidence in the brand. My Highlander was recalled too many times. I guess that's another post :)
I drive a lot: 36,000 - 40,000 miles/year. I did not feel like I needed (or wanted) to drive such a large vehicle every day. We do have a large conversion van, that we have for road trips, that I can drive if I need more space for the day. I would probably feel differently if I didn't have that option.
I was worried about the space issue with my last trade. The dealer let me put the carseats in the car, along with all my junk, and drive around for a day. Maybe that would be a good option for you?