I don't think you're nuts AT ALL!! That's usually the route I've gone the last couple of years. My son will be four this year. I have had his party in the afternoon, after lunch time, so people could just eat lunch on their own, and then come over. It also helps me get him a nap before they get here. (that helps keeps things fun for everybody, a rested child is a much happier child around here!! :) )
You don't "have to" do anything that doesn't work for you. Don't feel badly about it if you end up at some other party and they've done something different than you. You don't have to compare what you did. (we all do it, though, don't we?
When I have needed help, I have even asked someone else in my family to bring over the veggie tray. They don't mind. I think one year I threw some appetizer dish into the crock pot, but most of the time it's just been cake, really. Besides, my son is mostly interested in that anyway. We don't really do sweets regularly around here, so the cake is the biggest treat of all. As long as that happens, it's a very special day. :) Well, you know, in addition to the attention, and presents!!
That's another thing, don't put yourself so far into party prep and everything that you can't have fun and spend time with your child at the party. Why be worrying all about the food when you could be enjoying your child?? Get more pictures, more hugs, and see more smiles!!!