J.B.
I don't know how old your son is and I am guessing you work during the week. If that is the case you have every right to want to keep your son with you. Maybe you can suggest taking him once a month or once every other month or something like that. Discuss with your husband how you feel. Because I would be exactly the same way. Although I am a SAHM, but I use to work and cherished every minute we had together. My oldest daughter is 2 and she knows her grandma and grandpa who live in New York and we maybe see them once or twice a year. She knows them just as she would my parents who live in the same town we do. Just because your son doesn't hang out with them every day he wont forget them or love them less. Just try discussing options with your husband. I hope all works out for you.