Breastmilk is always best! But, the mother's diet is also important to ensure this.
You don't mention your son's special need, but he can be negatively impacted by foods in your diet (especially dairy, wheat and soy) if he has a sensitive system and neurological issues. For example, a GF/CF diet is very helpful for kids on the Autism spectrum. Also, many Down's kids have celiac (gluten intolerance) and it's not uncommon for CF kids. And, actually, I don't think I'm aware of a single problem that a GF/CF diet wouldn't be helpful (i.e., any neurological issue is often helped by this diet from everything from the most severe Autism to things like dyslexia).
The gluten molecules/antigens can be passed in breastmilk - I discussed this with a top pediatric GI doctor (both my sons are celiac and have dairy allergies). Very few (if any) conventional doctors are at all knowledgeable about the impact of diet on health issues - they get one course in nutrition, which actually doesn't deal with negative health impacts, but rather is the ABCs. And, unless the doctor is proactive in learning more in this area, they never well. All their required continuing education courses are funded by big Pharma and are almost entirely focused on drug solutions for symptoms versus looking at how nutrition and supplements can really help. And, just for clarication, I do absolutely believe that there is a place for Rx drugs, I just believe that basic nutritional solutions should first be considered.
I would trust your instincts. You are still providing your son lots of immune support and important nutrients for the brain by breastfeeding. I breastfed my sons for 2.5-3.5 years and in hindsight that dramatically reduced the severity of their celiac disease. You do need to make sure that you're eating a good diet (very little sugar and processed foods and organic as much as possible), with lots of healthy fats, B vitamins, iron, etc. And, it might still be useful to give your son some supplements and make sure they've tested his ferritin, B12, folate and minerals like calcium and magnesium. The latter is also really, really important for the nervous system. B12 is very important as well.
So, my recommendation (based on dealing with my sons who we later learned had celiac) is to continue breastfeeding and the babyfood. Mom's almost always know what's best and the best doctors agree.