Until recently, leaving our son with someone else wasn't an option, we just didn't live close enough. So, if you want to take him, I'd say go for it. We took our son to Vegas last year, he was almost 3. He enjoyed seeing the lights and all the action of the city. We're not gamblers either, so that wasn't an issue for us. We went to M&M World and Everything Coke and they were both fun. He loved the Bellagio fountains and we watched them for a long time. He also enjoyed the moving statues at Ceasars' Palace, he still talks about them. The funniest thing to us was that as we walked down the street the people handing out the calling cards avoided us like the plague. It just got funny watching them avoid us!
I can also see the advantage of leaving him with grandparents, but just wanted to let you know that if you decide to take him, it's not bad!