Do what's right for you! Downsizing can save a lot of money and give you "fresh" space. Most townhouses have either a loft or a lower level that can be outfitted as a den or office. Put in a really good sleeper sofa or a daybed that doubles as a couch in an office, and make it a trundle so that 2 girls can sleep there during breaks. Same with a 2nd bedroom - friends of mine used a trundle so that a twin bed was available for a single visitor, but it quickly converted to a king when a 2nd twin was raised into place.
When they come on break or when relatives come for holidays, they don't have all the stuff that "live-in" kids have, and most of the time together is in the public rooms of the house. Kids that have been away at school don't mind doubling up during visits home.
Knowing they'll be living in cramped quarters helps motivate the college graduates to find a job, an apartment, and roommates. With what you save on the mortgage and other expenses, you could give each graduating child a security deposit and first/last month's rent!
When our son graduated, he had a job lined up. He wanted to stay here for a while to save up money, and he did - but he paid rent, which was a good start in letting him realize that his salary wasn't all for fun! He took about 6 months to find an apartment with a friend, and he happily moved out. He really wasn't around all that much when he lived with us - up early, worked late, went out for dinner with friends, went running, hit the gym, and all that. He really just needed a bed and a washing machine, plus access to the refrigerator for breakfast. If you have 2 going to grad school, you'll have 1 more to deal with in a few years, and the youngest won't be out of college for 5.5 years. That's a long time to hold on to a house that's too big and too expensive to heat and maintain.
Resist the urge to store all kinds of kitchen gadgets and towels and so on - it's expensive to move and to store, and the kids won't necessarily use them. It's much cheaper to have a bunch of garage sales, donate them to agencies helping people who transition to new apartments (formerly homeless families, battered women starting over, new immigrants), or sell the better stuff on eBay or Craig's List.
There are a lot of people going into the business of advising those who are downsizing - they help sort, prioritize, and advise. It's usually worth the fees to save on moving costs and storage costs, and you get back a lot of that if you sell things wisely.
Spend the next year working with a downsizing expert while your daughter finishes high school. That will give her the stability of staying where she is through the end of her school days.
You might consider moving to a townhouse that's not far from where you are now, so the returning kids (2 younger ones) can connect with their friends when they are home on college breaks. But don't move to a place you don't love. If the kids are within easy driving distance of friends, it doesn't matter if you're actually in the same town or not. But remember that many high school kids lose touch with each other, and become better friends with their college friends. So they don't always have a social group to connect with after a few years away.
Remember that townhouses look great in brochures and when you tour the pristine models. Make sure you are okay living in closer quarters with neighbors. You and your husband might take a brief vacation in one (time share or other set-up) to see how you relate to the smaller space. I'm sure there isn't a ton of money left over with 3 kids in college but it might be a decent investment of your time. You don't want to regret your decision.
Overall, remember that your kids will not think twice about leaving you and moving away for a good opportunity. And that's as it should be! I don't think you need to feel too much of an obligation to stay in a big empty house just in case they don't get jobs! I understand your feelings - I had them myself. We are lucky our son came back to this same area for work - he now lives about 40 minutes away and we're grateful for that.