I agree that you need to plan a time, without the kids, when you can talk, on the phone or in person,with them in an honest way about how you feel. Use I statements. "I feel pressured when you keep asking to get together. This.....happened when we did get together and I feel ........."
Make a list before you get together of the things that are upsetting to you. then narrow that list down to a couple of things that are the most important to you and plan a way to talk about them in a calm and nonjudgmental way. Practice what you want to say. Include recent, emphasis on recent, examples. Yes, you can bring up how you fought all the time when you visited often but don't make that the central issue. Talk about what has happened recently.
Also plan for them not understanding you and know what you're going to say to explain your point of view again in a nonjudgmental way using I statements. Psych yourself up so that you don't get emotional.
I suggest that they may not understand or choose to not understand. In that case stop trying to communicate with them.
I suggest it's also alright to just be blunt and say that you don't want to visit. If they ask why, calmly and without judgment state what has happened in the recent past, how you feel about it, and that as a result you find visits unpleasant and don't want to visit. * Do not get into an argument with them. If necessary hang up the phone after saying good bye or walk away.
*Or not. I might just say, "I don't want to visit" without giving an explanation. You do not have to give reasons for your choices. Choices are yours to make. They don't even have to make sense. Picture yourself with a fence around you. You choose who you let inside the fence. Giving an explanation is letting that person inside your fence.
Or if you want, tell them you will visit once a month and in the meantime do not want to have phone conversations. You decide what you want, tell them in a calm and nonjudgmental way using I statements. "I want to visit you next month and suggest a day. Is that date OK with you? Please don't call me in the meantime." Their remarks. " I feel pressured when you call. It feels like you want more from me than I'm able to give. I need my space." End the conversation and hang up.
I often write down what I want to say first. Sometimes, after I've had a friend read it and determine that I have been able to use I statements, not criticized them, been nonjudgmental, etc sent the letter. Sometimes the other person does understand what I've written and we've been able to talk about the issues in a calm way.
You have the right to not visit. You are not responsible for their happiness. And......it sounds like the three of you have different personalities and expectations from your relationships. It is OK to say, this is not working. Let's take a break.
Here are web sites about a way of talking that may help you in this conversation. The method is called nonviolent communication. It's a way of acknowledging each person's feelings and wording things so that the other person is more likely to feel heard and thus better able to agree with you.
http://www.cnvc.org/
http://www.wikihow.com/Practice-Nonviolent-Communication