Hi J.,
I'm sorry you are going through this right now. My son was throwing up a lot from a stomach bug when he was 11 month, and I know it really hurts to watch them, especially at such a young age of 4 month! :(
I would suggest to hold off on all solids until 6 months, except for cereal. It is highly recommended by most pediatricians to wait until 6 months to even start cereal, but I know a lot of moms start at 4 months, and if there is no history of allergies in your family, I think that's ok. But in all honesty, even if your doctor said its ok to start, what is the hurry? Your little guy will get to sample all kinds of foods soon, don't push it, especially with the way he is reacting to it.
My biggest advise to you though is to call your doctor if you haven't already, and describe what is happening. It doesn't really sound like allergies, but it could be. It could also be that his stomach is just not mature enough to handle all the variety at this point. Just ask your doctor and see what they say.
Best of luck.
-Z