"Son, the reason Daddy and I live apart is because we weren't getting along living together. For a while, it's true, Daddy did come over more often to help with you and your sister and to spend time with you. Now that you are older, I think we need to make a new arrangement, because while Daddy loves you and sister very much, he and I still do not get along in a friendly way.
Your father is not a bad person, either. He can be a good person, and I can still find it hard to be around him. This doesn't mean that Daddy is bad, or that I am bad-- it's just that we are not good together as a couple. And we both love you very, very much."
I would do some proactive things to make sure that this conversation has some support: I would be clear with your husband about making a schedule so that the kids get dropped off at his place more often..and then stick with it. Consider going to the Goodwill and buying some duplicates of games/toys that your kids enjoy, or extras of paper and markers, any supplies for activities they like that they can do with dad or at Dad's. You know what the kids most like to do, so if it's Playdough-- put together a container with some playdough and some supplies-- stuff like that.
I'm not going to make a blanket statement about all fathers, but I noticed that when my folks divorced-- my own dad was pretty clueless about 'what' to do for us at his house. It got better when he remarried (his wife is an elementary school teacher), but for a lot of our early visits, we'd pack up some extra toys, our suitcase of Barbies and their clothes, our stickers and paper, etc. So, if you can offer that sort of support for your kids, and get a schedule going of 'who's house' the visits are happening at, you'll be in a better position to move forward.