K.V.
I second Martha's response. I am also a parent of a child like this and it can be very isolating. Kids don't understand that mine doesn't make a correlation between actions and consequences like most people. For example, he'll push kids on the playground, know he's pushing them but not understand that because he's on one of those elevated all-in-one playsets that he could potentially cause the other kid to fall off. His thinking doesn't get that elaborate. Playdates have to be heavily supervised - you obviously have to protect your kid - and will require a lot of patience. He needs to be shown how to play with your child, telling him no won't change his behavior. Instead of a regular child who would understand after the first couple times, it will take many many more times for this child to understand but it will happen. I have other children who are not disabled and having them see me show him how to play rather than yelling is teaching them to be patient and tolerant. And when he is doing something harmful and I do put him on timeout and tell him firmly that's not acceptable, I'm teaching them to stand up for themselves. It really could be a benefit to your child to have these playdates if you can get them under control.
My child has buttons that can be pushed to help make him choose to behave. If he's doing an activity he loves and it's taken away because he's misbehaving, the next time he's in that situation he's more likely to stop if reminded it will be taken away. Watching TV is considered a privilege and I use that quite a bit. If I tell him to sit in timeout and he runs away from me, I will tell him to go sit or TV gets taken away. He has hearing tests and he knows if he does a good job he gets to play on the playground afterwards. The important thing is consistant follow through so he knows you mean business.
Try to pick playdate activities that will result in the best outcome. Ask the grandma if there are certain things that he behaves better in doing. Find things that don't require a lot of physical contact between her grandson and your child if that's part of the problem. I know for me, it's easier for playdates to occur at my house because my house is set up for my son and he knows the rules here. If we go to someone else's house, they may not be as good at child proofing or their backyard might not be fenced (he's an escape artist) and in his mind, all rules we have at home are thrown out the window because this isn't home. Would this child understand if you were to tell him the rules of your house? That could help.
As a last thought, I would rather have someone be direct and say to me that things weren't working out than have them slowly drift off and eventually never talk to me. Like Martha said, if you're honest and don't place judgement I never find that offensive or insensitive. I know the realities of my situation and my child. Not everyone can handle it, not everyone knows what to do and that's ok. Would I if the situation were reversed? Probably not.