"If I stop letting my son go over to spend the weekends or overnight in the summer then I'm punishing the kids by keeping them apart."
I hope you can see that you need to keep them apart, period, even if you feel you're "punishing the kids." No more sleepovers (or day-long visits) at all -- you have zero idea what is going on all night long: What are he and the kids talking about? What might he be telling them or demonstrating (i.e., drinking, even other drug use you don't know about?)? What videos might they be watching, or what web sites might they be visiting, on these overnights when you are not around? Please do not give this man any more time like this with your child. I know you don't want to "punish the kids" but you have to be the adult here (even if you are accused of being a mean mama, of being overprotective, of keeping your son in a bubble--you MUST toughen up and ignore all that and you WILL hear it, even from your parents, possibly).
You make no mention of the cousins. How old are they? Surely they are aping their dad's behavior. Your son's own behavior may come from copying not just his uncle but also his cousins, who are learning how to behave from their dad. Yes, it's not their fault dad's like this, but you cannot be concerned with them right now. You need to get your son away from them as well as from their dad, unless they're angelic compared to their dad.
You say you "want to be able" to visit your father. Are you concerned that if you start to turn down all sleepovers or don't let your child see uncle and cousins except with you there (and even then it should be limited!) then somehow your brother will prevent you from seeing your father? If that is the case, you have an even bigger problem -- that would indicate your brother is possibly manipulating and controlling your parents against you.
Your son is at a very difficult and highly impressionable age. What he picks up now will be much more difficult for you to talk or teach him out of; he is no longer a young child who will amend his ways to please mommy and daddy. If something attracts him or is funny, he's going to do it. I'd remove the attractive, funny temptation and start just seeing your dad and leaving. Can your dad leave the house and come to you at all or does his cancer mean he is house-bound? I would find ways to focus your son on his grandad in these visits (can granddad and grandson play a game together, etc.) and then leave promptly.
When you are at your parents' house, don't let your son go off with the cousins or uncle into another room. Make sure you or your husband are IN the room where your son is. And do be sure YOU are getting visits alone with your dad, where you do not bring your son -- because you are a daughter here as well as a mother, and you need some time with your father right now.
You know your son best. How would he respond to a serious talk along the lines of: "Granddad's days are few and you need to know and understand that. Once he has passed away you won't have the chance to see him any more. I'd like you to consider focusing on spending time just with him, one on one, in the coming weeks/months." If he is close to granddad he may find this a spur to doing just that and you can at the same time limit his time with the other family.
Meanwhile, be sure your son has other activities that really interest him and to which he feels committed. Does he have an activity, a sport, a church youth group if that's your thing, any outside thing he attends and where he has peers who are better influences than his cousins? Focus him more on that. Or find him a new activity (with his input -- he is too old for you to pick these things for him entirely). Make him too busy to visit, frankly, other than visits focused on granddad.
I would be concerned that your brother, living with your parents and already having mom on his side, is setting himself up for taking their house or otherwise running the show entirely once your father has passed away. I'm sorry if that sounds harsh or is hard to hear but it's something to think about now, before the time comes and you find mom has put everything into brother's name. I am not thinking here about inheritance at all, but about whether your brother is controlling your mom and will do so even more once your dad is gone.