K.I.
My mom gave me some advice when I first got married, I did not take it b/c I didn't need it (not sure I agreed with it, still not sure I do) but I think for you, it might come in handy...so I will pass it along to you:
My mom told me to start saving money on the side, without The Man knowing. Every payday start putting aside (in cash) whatever you think that you can. $20-$100 whatever. That way you will always have an emergency fund...just in case. Maybe this will work for you and give you some piece of mind? In the end, if you don't need it you could always bust out with the huge lump sum and have a wickedly awesome 50th Wedding anniversary present/trip or something? IDK?
My Mom said her Mom gave her this advice too. My Grandmother apparently did this, different generation different mindset, I guess?
If you have talked to him and he still doesn't get it, I am not sure what else you can do? Have you considered separate accounts? What are his thoughts on this? Have you tried each having a set monthly allowance? I do not think it is fair AT ALL that he spends willy-nilly while you are the only one saving and skimping!
~I am gonna be blunt and honest here...sorry in advance if this upsets you: Reading what you wrote about him accusing you of 'wasting money' on the kids straight pisses me off after reading he needs $90 golf shirts and massages?!?! What the Heck!!!??? Your husband sounds like a selfish man...and I am not sure there is any way to get your point across with someone who is THAT selfish. He might never 'get it'!
So...IMHO, if I were you I would be taking my Mom/Grandmother's advice...like ASAP!