M.V.
i totally understand. my daughter is now almost 5 and we went through that for a while at different stages. my son too, who is now close to 3. what worked best for us was to have consistent short practices with clear communication before i would leave that i would be back...then when i came back, i would verbalize that i always come back.
my mom suggested too that i start treating babysitting times as treats...telling my kids that whoever is coming over is special and the time is so much fun. one of my children needed fair warning of babysitting, while the other was better off being told closer to the actual time...
i would also suggest leaving the room when your husband is trying to soothe your baby so you aren't a temptation...but communicating that to daddy first is really helpful for him to know his role.
good luck!