A.L.
My son is 5 and still does this. I believe its a texture thing. To combat this problem, we just ask him to take a no thank you bite of anything new, and have a napkin or paper towel ready for him to spit it out into so the vomiting doesn't start. It started working when he was almost 4.
Last week at Disney my son did this when he tasted a bite of cheesecake, and we just got a napkin and he took care of it and washed it down with a drink. He totally wanted to gag, but was embarrassed that people would see him. I think it will take a little time and maturity for him to stop. Good luck!