WOW!! First of all, CONGRATS on nursing your son this long!! That's AWESOME!!! Second, they are YOUR breasts, NOT his! Third, ask yourself this question: How do I feel about nursing my son?? If you're still comfortable with it, then go for it. Either way, tell your husband to back off, especially if he isn't going to participate as it seems he doesn't want to. It may not make a difference, but try explaining to him that there are ALWAYS health benefits to nursing, and when you're both ready, then you'll wean. It may not seem like it, but your son WILL wean! He's probably not going to be nursing at 5, although that's very normal in other countries. You need to be strong in your convictions whatever they are. If you're ready to give up nursing, then stay strong with your son. If you're not ready, stay strong with your husband.
Honestly, I was so sad when my 2.5yo son weaned himself :( I was 4mo pregnant with his baby sister, and it hurt really bad to nurse him, but I miss the time. Obviously, we found other things to do to promote and strengthen our bond, and we still have cuddle time, etc. Now, I look at this almost 3.5yo and see this independent, strong-willed, smart, caring, sensitive boy that I've helped parent, and it's just awesome!!!
My husband was ready for me to give up nursing when our son turned 2, and I thought I would be, too, but we weren't. My husband just had to accept that it was ultimately my decision. I know that sounds harsh, but that's how strong my convictions are about nursing. Good luck whatever you decide! You've gotten some great suggestions from other mamas about weaning!!