I know exactly how scared you were. A little over a month ago my five yr. old son was very sick and I simply thought that he had caught the bad flu that was going around. He started to get worse on a Friday and by Sunday he wasn't himself. I called our pediatrician on call, explained the symptoms and was told that he probably just had the flu. However, he was very lethargic, wouldn't eat etc. And against my better judgement, I waited until Monday for him to be seen. So I waited until Monday morning and when our pediatrician walked into the exam room he knew right away something was wrong. The next hour was a blur, they called an ambulance to the doctors office, and we were wisked down to Childrens.
As a mother, I've never felt worse in my life feeling that I allowed my son's health to deteriorate. I was reassured by our pediatrician that the signs are hard to read and that he was very upset with his partner (who I spoke to that Sunday) for not encouraging us to go to an Urgent Care. Seven hours later we were allowed to take him home but we were told that he needed to stay home from school because it was contagious. I also have a two year old and luckily he did not get it but we also tried to keep him away from his brother as much as possible. Although we started to see our son improve within a few days, we kept him out of school for two weeks just to give his little body time to heal.
When we went back for a follow-up, the pediatrician said that the report he received from the ER stated that had I waited another hour for him to be seen, his lung would've collapsed at home! I swore to myself that I would never be concerned again about what doctors or others thought about me, my first concern should be my children. I simply didn't want to be someone who called the doctor everytime my kids got the sniffles but that experience taught me that anything can happen so very quickly.
Although I'm sure that your son is probably showing improvement already, I would recommend postponing his party. Although he may be acting better, I've learned that pneumonia takes time to recover from and children can be very fragile. I'm sure the other parents will appreciate that you're looking out for everyone and it's better to err on the side of caution.
I hope that he has a wonderful birthday regardless and that he recovers quickly!