We went on our first vacation, just the two of us, last August when our son was about 20 months old. We were gone for 3.5 days, and my mom stayed with him at our house while we were gone. And he was fine! We were super worried about it too, but it all worked out great. So just breathe, and know that while it may be hard, you can do this. =)
I think what helped is my mom came and stayed with us for a few days before we left (she lives 9 hours away, so it's not like he sees her very often), so she got used to his routines and whatnot. Then we put him to bed Wednesday night and left right after that, so it wasn't like he saw us leaving (not sure if you can do that), and when he woke up the next morning, my mom was there. We did take a laptop and webcam with us and tried to use Skype--we couldn't get him to show up on our screen, but he could see us so I think that helped him as well.
As for the family who is against it, I'm sorry to hear that. If they give you grief, let them know you are doing what you feel will help your marriage--which, and I may get in trouble for saying this, sometimes needs to come first anyway. Your daughter will be fine, and she'll be so happy to see you when you get back. Enjoy the kid-free time--it's so rare that you get it, and you deserve it! =)