THANK YOU for asking this question and starting this thread- I feel the exact same way and get SO much flack from other people about how we need to just go on "vacations" and leave our kids behind... Our children are almost 1 and almost 3, and have never been left with anyone other than family. In fact, when our first child was a few weeks old, I got talked into leaving her with inlaws while my husband and I went to a movie. When we came home 3 hours later, they said our baby was perfectly happy- OF COURSE she was perfectly happy- she was propped up in front of the TV!!!
I was absolutely crushed and I think this played a huge role in why I am so careful about who we ask to watch our babies now! Those people rarely ever get a call to watch our children, but we do utilize other family members and are blessed to have loving, trusting people in our lives who want to spend time with our babies!
I know our babies are still young, but I don't feel like they are going to "suffer" in any way because of this, other than they might be missing out on a lot of inappropriate television shows... :)
In all honesty, I think you just do what you are comfortable with and what you feel is best for your family- even if people judge you for it, it doesn't matter. These are your children and you have the right to raise your family however you feel is best.