I say enjoy family while they're here. My family is very close.
When we have holidays, it's [most often] in my home so my inlaws, my EX inlaws, my step-parents, my parents, my aunts and uncles, my cousins, close friends, and all the kids can come and have a wonderful time together. And yes... My ex's family are part of my family still too. Just because I divorced their son, brother, whatever... Doesn't mean I divorced them too.
My mom irritates me to no end (calling me all the time and many other things), but that doesn't mean I don't want to spend time with her as much as I can... You never know how long someone will be around. My best friend's mother was only a few years older than my mom... Died of lung cancer. A friend of mine, 45yrs old... Died of a brain aneurism (sp?). Her daughter is 15.
Don't sweat the small stuff. Just let him know that you're not thrilled having your 3yr old over night anywhere yet.
I say, what's the huge deal about having holidays at his house this year? It's not like you cannot visit other family other days.
The affairs are none of your concern [IMHO]. This should have nothing to do with how you view your father. My dad had affairs on my mother... It had nothing to do with me... I love him as my dad. Not as my mother's conscience, savior, or defendant. I learned as I became an adult that both of my parents have issues that caused a chain reaction ending in their divorce.