I've been in that situation unfortunately... and I pray it's not the same reason I had. What worked for me... just be unavailable. It will take time... but each time a child from that family wants to come over or do something, be unavailable... make up something important that you already have planned. Make a list now... going to the beach/park/picnic. Have family movie or game night. Plan a suprise trip to a family member. Have several "home projects" ready... spring cleaning, make t-shirts, crafts, etc. "Oh, I'm sorry, but we already have plans..." You have to be prepared to do things on short notice, and explain to your kids that you're sorry you forgot to tell them, you wanted to suprise them, etc.... Eventually the other kids will stop calling, and you are not the bad guy. If you try to put a definite stop to it, then your kids might rebel, and you may not want to explain the reason why...
It took a few months for me, but my kids now hang out with other friends, and there was no big blow up over it... Hope this helps...