M.H.
Ditto Diane B. I'll add that if he has thrush in his mouth then his entire little body is eaten up with it. Yeast grows from the inside out so if you can see it there's a lot. You have to get it under control from the inside because it weakens the body and can cause susceptibility to other things.
Remove all sugars (even white potatoes, rice, breads, etc) and start him on a probiotic. Yogurt will help if you feed him the unflavored kind. The fever is doing its job. 103 is not too high, but it is telling you something is going on. With his immune system so weak, he open to pick up more.
Regards,
M.