I don't know that you can decline "once and for all" without the other family at least wondering what happened. If you continue to dodge playdates I would think they'd get the message and give up. If your son has had lots of play dates at their home but very few at yours, I myself would probably host at least one more play date and observe the boys during that. Frankly sometimes kids say "I don't want any more play dates at X's house" because they just find X's toys boring -- not because they have any issue with X. Have you asked your son about that?
There's a lot we don't know and maybe a few things you don't know yet either. It's really hard to comment when we don't know the boys' ages or what they attempt to play or whether they know each other in school as well, etc.
Why does your son say he doesn't care for this kid? Is he specific about problems he has or does he just shrug and say "It's boring"?
Have you observed them playing together? Do they just have different interests, or do they actively clash or even argue? Does one boy like to be outside and running all the time, while the other prefers doing quieter things like Legos on the floor?
Did playdates with this child get started just because of proximity -- in other words, because he was a classmate of your son's or because he lives nearby? Or did they start because the boys seemed to get along well in class or elsewhere, or seemed to share some interest? If it's just proximity and nothing else, well, you tried, but they don't click. If they started out with somethiing shared like an interest, what happened with that?
You're absolutely not obliged to have your son play with a kid with whom he just doesn't click or with whom he has nothing in common. I'm just offering the questions so maybe you can probe if the issue really is just "they have different styles" or something else.