I also agree, I doubt he has had an ear infection for 4 years. My son has a similiar habit, but he bites the back of his hand to the point that he has huge callouses. I would suggest a substitution plan. Give him different things to do with his hands at the time that he usually plays with his ear. Like squeezing a stress ball, or even putting a small amount of play doy in a ziploc bag and letting him maneuver it around in his pocket. It might just be a game to him at first, but it might be a good distraction from his ears. Be patient, its going to take a while to break this habit, but easing into it now may make a big difference by the time school starts. Be very careful not to make pulling his ear something "bad" that he does. He wont understand the concept of socially acceptable yet and he definitely doesnt need to learn that "different" things are "bad".
On the other hand, he might also be pulling at his ears for a different reason, some kids do this when they have sinus issues, allergies or even dry skin/scalp. It wouldnt hurt to mention to your pediatrician about possibly seeing an ear, nose and throat doctor just to relieve any concerns. He would know more about possible effects of this habit too.
Good luck!