We have this same issue with my husband's sister. She is a severe drunk and we did put a stipulation on her that if she is drunk or had been drinking then she could not join us. It is not that we do not have drinks once and a while, she just gets out of control and I do not want my three kids exposed to that.
I would talk to your husband about that as a solution and then clarify with your husband on the other line or in front of your older son that if he drinks or has any pot, then you will immediately take him home.
The BIG bummer about all of this is, your son is a grown man and needs to take responsibility for himself. Your responsibility is with your husband now. You did your raising of your son, and unfortunately we can never predict how they will choose to spent their adult life.
As much as you love him, you have got to give him the option to be sober or not join you and this will work ONLY if your husband agrees to the terms. If your husband is still dead set on your son not joining you even if he is sober then you need to respect that and make other plans with your son. Your husband is right, it is not good for your child to be exposed to self-destructive people even if it is his half-brother or in our case their aunt.
Good luck, I know how hard it is!