That sounds really tough. Have you tried a babysitter, even for a few hours? You could invite the babysitter over the first time and stay and plan projects that only the babysitter does.
Then, sometimes you have to initiate the playdates. Even if it is a kid that is somewhat rough, I'm betting it isn't as bad as you would think and you could always try.
I think that slowly introducing new things, outside interests will help. You also might try to drop him off at his grandparents for a Sat. afternoon. He might cry and whine, but I bet you after you are gone, he has fun. Right now he probably knows that if he protests, he wins.
Another idea is that after your son goes to bed, which I know my 4 1/2 year old is down by 8, don't do laundry, other chores or TV. Make a pack with your husband that you will spend 1/2 hour or and hour talking to each other - and that you won't do any kid speak! My husband and I have started to do that because we have two and its easy after coming home from a long day, putting two to bed to not communicate or focus on kids.
Hope this helps.