D.W.
I'd check with your pediatrician to see what their recommendation is, but usually it's ~24oz of whole milk/day at that age. Our daughter would consume twice as much if we'd let her.
If he doesn't enjoy it anymore or just won't take anymore, I'd ask your pediatrician on their recommendations on replacing the nutrients lost on milk. Perhaps they'll recommend rice/soy milk if the taste is more pleasing to your son. They may recommend adding more broccoli, cheese, yogurt, etc. to replace the calcium/fat of whole milk.
I'd also ask if they recommend a multi vitamin at his age. Our daughter started on 1/2 of a Flintstone's vitamin around 18 months.
I don't know if that helps, but that would be my best guess.