I'm in a similiar situation. My father in law is a smoker and my mother-in-law drinks in the eveings. I feel okay with leaving my kids, 2 years old and a fourth month old, with her during the day in my home. The kids are NOT allowed at their home because of the smoke, and I cannot monitor her drinking. My mother is busy, and doesnt see the kids as much as I would like. My father is not well, and he enjoys his grandkids, but we all understand that he cannot watch them alone. (My parents are divorced) SO...
My Mother-in-law has this bright idea she is going to retire and watch my kids, so I can go back to work. Um, HELLO, I never said I wanted to go back to work. Luckily, my husband is understanding of the situation. Although, he thinks the kids should see his mother more.
Bottom line...Those are my babies and they are the most important thing in my life. And, I will NOT put them in harms way. His mother may think I am a witch, but I dont care. Keeping my kids safe is much more important.
So, my advice to you, is to stand your ground and to do what you believe it best for your kids. If you inlaws dont babysitt as much, maybe you can bring the kids over for supervised visits?? Or maybe just drop them off while you grocery shop, or run some quick errands. That way they arent there for along time, and hopefully the visits will keep everyone happy.