Hmmmm . . . I'm not sure why a kid needs to be around an adult who is so obviously emotionally stunted, parent or not?
Yes, I generally agree that kids need both parents. But I would question whether your ex has the capability to be even a half-way decent PARENT (not a friend). Your son does not exist to coddle his feelings. Was that really the best way (yelling, threatening to sell his stuff and cursing) to handle your son's obvious distress? For an eleven year old boy to cry, especially one that's so into sports - generally that's a big deal.
Obviously you didn't want to be around this guy - and it sounds like for good reason. Maybe your son has noticed some of the same less-than-desirable personality traits? Have you delved into what, exactly, goes on when he's over there?
I'm not talking about a situation where one parent actively alienates or trashes the other. It doesn't sound like that's the situation here.
Yes, I agree some counseling would be good to sort all this out. But I would not make my son feel like HE is the one with the problem. I also would make sure that I'm not pushing the situation for my own benefit (i.e., getting a break).
Good luck and I hope all this works out for the best.