K.R.
It's very important when talking about allergies, milk in particular, to distinguish between an allergy and a sensitivity. Many people, even adults, have a sensitivity to dairy; their bodies have trouble digesting it. In that case, your son would probably have an upset stomach, may be gassy or have trouble pooping or have diarrhea.
An allergy is systemic; it affects more than one system in the body. That's why you get swelling, rashes, etc. And some allergies can, I understand, become more severe with repeated exposure. As your body reacts again and again to the perceived threat, it will respond more strongly. My husband had penicillin for the first time as a child and developed a rash; he was diagnosed as having an allergy. Some years later, he was at camp and needed antibiotics. They didn't believe him that he was allergic, and he had a life-threatening reaction. If he were to take it again, the results could be fatal.
If he's on the formula now and not having any symptoms, I wouldn't worry; it sounds like he's not having any symptoms of an allergy. But if you want to be sure, an allergist is the place to go. They can tell you for sure whether or not your son has an allergy.