From what I've seen and heard, pediatrician recommendation is all over the board on this one. My opinion is that you should base your choice on the rest of his diet. We are still giving our 3 year-old sons whole milk because they get very little fat in the rest of their diet (and they need some fat for development). We aren't over the top on fat-free, nor do we give them artificial sweeteners, but we feed them a lot of fresh fruits and vegetables, whole grains, etc. They both still have no body fat at all, so I'm not concerned with the fat from their milk. We will continue to re-evaluate as they grow to see when to wean them back.
It's great that you are seeking what's best for your little guy, but don't stress over the decision - he's probably only drinking a few cups of milk each day, so either choice will help him to get the calcium and vitamin D that he needs . . .