D.B.
I've been thinking about you since your other post about this visit.
I know it's so hard. Here's the only way I know to get through it: understand that we make stronger kids when they don't have everything so easy. It's okay, in the long run (though very hard in the short run), when they learn to deal with disappointment and imperfection in others. The challenge is to get them to talk about it, ask them to verbalize their feelings, and separate out someone else's undesirable behavior from the child's own worth as a person. So sometimes dads and moms and grandparents do dumb or hurtful stuff, but it's their problem and not an indication of a major flaw in the child. The other thing is that kids go through phases when not everything bothers them the same way it did a year ago or will a year or two in the future. Try really hard not to project your own frustration onto you child - he may not be as annoyed with certain things as you are. Sometimes an occasional "good time Dad" can be okay without the pressure of regular visits. If he's not crazy about the grandparents (especially the grandfather, if I recall correctly), then it may be a blessing that they live far away and your child doesn't have to deal with them often. And you'll be spared the tough times a lot of single moms face every other weekend when they have to make a child go visit the parent they don't love being with.
We have to teach our kids to deal with other people's shortcomings or bad habits without it crushing them. In fact, long term, it makes them more resilient. There's no perfect childhood - everybody's got something they need to deal with. This is your reality - if you can come to terms with it, and try to fill your own day with something rewarding or productive instead of just being miserable with worry over the time your child is having, you'll do better as well. My husband has an impossible ex, so I understand - but the worry and the anger winds up eating you up and doesn't do a thing to the other person!