My "situation" as they call it was quite a bit different. My ex was never around for my daughter. I'm wondering, though, if your son is concerned that, because your husband isn't living in your house, that maybe he won't be either. When he leaves, maybe he's not so sure he'll be coming back. At 4 1/2, he's probably old enough to have a real heart-to-heart with. I think it's just time to sit down with him and assure him that he's not going anywhere for more than a few hours at a time. Talk about how families are different and how families and family life can change (a conversation I had w/ my 4 1/2 year old granddaughter regarding her non-involved father and her sister's very-involved father). Just keep letting him know how much you love him. In time, and with a lot of love and continued consistency, it'll get better.
My heart goes out to him. This is a very tough time for your son. He's scared. He's probably angry. Confused. Sad. At 4 1/2, he doesn't always know what to do with all of the emotions. He's doing what he knows how to do right now. All he needs is a little guidance on how to handle them in a healthier manner.
As another poster wrote, I think those people that now feel sorry for you will have nothing but admiration and respect for you in the future. Just keep doing what you're doing, seek advice from others, find out what works for them and what doesn't. You're a strong and courageous woman, and don't let anyone's sympathy bring you down!