I would first suggest that both of your children and yourself start family therapy. This is one of the first things a child should start when divorce happens. This is something I never got when I was a child, and I cannot say enough on the importance of it.
Secondly, you are really in a tough position. Especially with your ex's mother putting things into your daughter heads, and asking questions. Totally not fair.
Thirdly, holidays are always iffy when divorce happens. Either way, children are always put in the middle, and issues arise. So, prepare for it. The best thing to do is to not put any pressure on your children, and do not engage long term in any situation with your ex's family.
I would drop them off at his mother's and pick them up. Set this up with your ex ahead of time, so their is no surprises, and his mother knows her place. She shouldn't be involved, but it sounds like she wants to know what is going on. Set a time,date and then a pick up time and date. Keep it short with conversations. The less the better.
Your children are 4 years old. So keep it simple and in their understanding. I was 5 when my parents divorced. My mother told me that my father was going to live in his own house, and she was going to live in another house. That I could see my father anytime I wanted. My mother told me that she loved my father, but not in a way a person should when they are married. More like a friend or a brother. Mommy and Daddy wanted to be better people to me and my brother, and so they needed to not live together anymore. That she hoped when I was older that I would understand better, and that she loves me very much.
Do what you feel is right. I would keep it simple and whatever you do, do not list out the reasons why daddy and you cannot live together. When they are much older you can have a different conversation.
I would also add that you should let them talk to you about how they feel at any time.