Two issues going on here. One is the boy himself and his behaviors. The other is the untenable "babysitting" situation, which is really about the dad and not the boy.
Has the dad ever actually asked you HIMSELF to look after his son? Even a casual "Hey, would you mind watching Bobby for an hour if he gets home some days and I'm not home from work yet?" Or has this entirely arisen because the poor kid, himself, is coming to your door?
You're right; the dad is frankly flaking out and expecting you to be his "village" and look after his kid. First, I'd tell the dad, "Bill, I know you want Bobby to be supervised when he comes home from school, but starting (date) I am going to be very busy in the after-school hours, and won't be able to watch him, and it's not going to work for us to have him and my child play together at that time. If he comes to the house on his own, I'm going to have to call you to come get him then. I hope you understand." And then-- DO be busy. Have activities for your own son to do, so he will make new friends outside the neighborhood and eventually this boy will be less of an attractive playmate. If you are going out the door to a class or event or sport with your own son, you can legitimately tell Bobby, "I'm phoning your dad right now. We can stay only until he picks you up." A few times of this and dad will maybe get the message. But you do need to make a friendly but direct statement to the dad that you cannot be on call every day for Bobby to just drop in.
(And as your son gets older, this has to stop anyway; your son's homework will increase and honestly, gobs of play with the neighbor doesn't work most weekdays when your kid has plenty of homework!)
I also might tell dad that if grandma is there, Bobby shouldn't come over. Not sure at all what that's about--why can't grandma go into the house with him? Sounds like she's a reluctant sitter herself, or like dad does not trust her somehow...?
As for the poor kid himself, he needs some guidance. It sounds like he's aimless and unsupervised and probably undisciplined if he's destroying things. I wonder if any of the other families who say he's destroyed their stuff have ever told the dad face to face and asked what he planned to do to have the boy clean up, make restitution to them, earn money to repay, etc.? If they had it would have done the boy AND the dad good. If you hear other families say the kid did something like this, I'd advise them to tell the dad -- not angrily but in a factual way -- and say they need to talk to the boy about making things right.
You can be a somewhat soft landing place but don't be too soft-hearted either. Let your son play with him only at your house, with you present, a very limited number of days a week, maybe even one day. Perhaps even tell the dad, "Our family schedule's changing (true, because you're doing it right now) and Bobby is welcome to come over to play each Wednesday after school until 5:30" or whatever. But at the first sign of destructiveness, tell the dad. At the first sign of "Let's wreck the squirrel nest," tell the dad, and ask him kindly if he knows that harming animals is not a good sign. The first bad word gets a warning from you and the second, he gets sent home and is told next week he can't come over. And so on.
If your own son is busy with school and gets into other activities, eventually this boy will fade for him, and if your son is just not around as much because he's busy, the boy will not have as much reason to come over. These things will happen naturally as your son gets older and gets into other things that interest him, and leaves the idea of neighborhood buddies behind in favor of buddies who share his interests and activities. But meanwhile I'd be kind to this boy but very firm about when and where he comes over and what he can say and do when at your house.