I don't really have any restrictions on my daughter's (who is almost 5) TV watching - but when I feel she's had enough, it gets switched off. Even when it's on though, she's really not just passively watching it or turned into a zombie. She's still drawing, playing with toys, wants to play a game with me, etc. Therefore, I don't think it holds that much fascination for her. She spends the majority of her time during the week in preschool/day care and then will be in kindergarten in the fall, so it's not as if she has all day to watch TV anyway. If it's on one of the kids' programming channels for an hour or two in the morning, and a couple of hours after she gets home, I don't see it as a big deal (and yeah, it does end up being more on the weekends). She still has plenty of other activities to fill her time and the watching of TV has not hurt her in any way. Some of it is Daddy too though, when I have to work and he's in charge and he restricts TV even less. Watching TV is how hubby and I unwind at night.
ETA: The only channels she watches are Nick Jr., Disney Jr., PBS Kids, Sprout, and the occasional Disney cartoon movie. And no TV in her bedroom!