I love it!!! Our walk is about the same as yours. It's way better and quicker than driving! We've been doing our walk EVERYDAY since my son was 5 in K and my daughter was 3. Now my son is 8 in 3rd grade and daughter is 6 and in 1st grade - so we're on our 4th year! Now that it's cold out, I don't even have to get out of my pj's in the morning - I just put on snowpants and coat over them!!! The only problem that I see you might have is your youngest being out in the cold, but if you oldest is not starting school till next year, then he/she should be fine by then. If the sidewalks are too snowy to use a stroller or a wagon, then pull a sled! Super fun! Easter bunny brought my kids umbrellas a few years back to use when it's rainy - like I said we walk EVERYDAY, unless there is lightning.
Also, I feel much more connected with their school. I see the teachers everyday. Not just their personnel teachers either - I see art teacher, music teachers, helper teachers, play ground teachers - I literally know almost everyone that works there. The other day, my son forgot his homework and it was no big deal to run it over to him. When the kids have "extras" at school, like plays etc, it is super convenient to live so close. Even if we do drive for these extra activities, it's so nice that it's so close and we're home in 2mins!!
So yes, you're crazy for 2nd guessing this :) Not to mention the fact that you can take the kids over to play at the playground all summer long! And sometimes schools have special summer stuff going on like Library Wednesdays etc.........Living close is a great way to keep involved!