What a shame. I don't think you're freaking out - I think your original idea sounded perfect with just the 3 of you. That's how I'd do it again with our first child. It's a really important milestone for your family, and I think you should do it however you see best.
In all honesty, a first birthday party is a photo opp and WAY more for the adults than the child. So, to keep the peace, if you're comfortable with changing the date/location, I'd recommend it and still keep your original plans for the 3 of you.
We've had to bite our tongues several times with both sides of our extended families because their opinions on how we do things were so different than ours, but to keep the peace you have to pick your battles.
With my personality, I'd honestly not bend, but that's me personally. I can be very stubborn when pushed in a direction I don't want to go, but I also know it's not always the best approach.
Good luck and congratulations on surviving the first year! The second year is SO much fun.