First I would ask you is it important to you and your family for your son to leave the tv buttons alone? If it doesn't bother you then you know it simply isn't a "battle" you need to engage in with a toddler. Be mindful, with that age group, what is okay at their home is fair game in other peoples homes - so you may encounter issues there. Each family has things that are non-issues in their home which could be major issues to another family.
My household prefers that children do not play with objects that aren't designed for them - but that doesn't make that right for everyone - It simply works for us. The way I handled it varied from child to child, but for the most part I simply said no and then redirected the child to something far more interesting - even if that meant we needed to head out doors or get out arts and craft type stuff - or even blowing bubbles INDOORS - another example of one of those things that is okay at our house but not in others.
Good luck and rest assured once you truly assess your family's needs then you know you will have found the absolute right decision for you all!