Hi N.,
I think you should keep trying to feed him breastmilk when you're with him. My boy's almost 10 months now and he's also been allergic to formula. He had a massive breakout after 2 sips when he was about 3 months old so he's been exclusively on breastmilk. We started solids at 4 months with the pediatrician's recommendation but he's very happy nursing as well. We've been nursing 4 times a day with 3 meals a day of solids. This week I've brought him down to 3 nursings/day. But I noticed that if I increased a feeding for 2-3 days, my milk supply would go up. You do have to take your vitamins and drink fluids to keep up(at least from my experience).
My boy's been allergic to yoghurts and cheeses as well but in the last week, he's been eating yoghurt just fine without any breakouts. I've been told by another mom that they grow out of it as they approach their first birthday. Maybe your boy will as well.
Good luck!