P.M.
You have perfectly legitimate needs. You have an obligation to your child and to yourself to honor those needs. You have every right to protect your health, to be realistic about your budget, and to take care of your precious daughter's health and safety. Once you take your needs seriously, an expensive trip across country with a small child to spend time in the home of people with unhealthy habits seems pretty absurd, doesn't it?
Of course, complications arise around feelings. Feeling obligated, feeling guilt, feeling afraid of how others will see you, and trying to take responsibility for your husband's and in-laws' feelings. The good and bad news is, you can't be responsible for them. No matter what you do, you will always run the risk of offending or hurting the feelings of people you scarcely know. So my suggestion is that you get really clear about what you and your daughter actually need, and stick with that.
Of course you want your husband to be happy and get his needs met, too. You might gently ask him what he really needs – to jump through his parents' "guilt" hoops, thus setting himself up for repeat performances in the future, or to be an effective, responsible dad and mate. He's in a tough spot with his parents. You can acknowledge that, and support him in taking a stand for sanity.
It's possible, even easy, to tell people what you need, simply, clearly, and powerfully, by saying some version of, "Mom and Dad, I hear that you would like John and me to come across country to visit you. It's sweet that you want to meet your grandchild and spend time with us, but no. That will not work for us." Don't start giving all your reasons. They will be seen as excuses, and the in-laws will ramp up the pressure. Simply say no, you won't do it. (Try saying that to a mirror, and see how good it feels!)
Then renew your invitation to have them come and visit you. Be warm and sincere. They indicated a willingness to travel to you once, so it is probably not beyond their means.
Go ahead and be the heavy, H.. The world won't end. If you can let go of your worry about being seen as unloving, you might be surprised to discover the sanity and delight of being loving in the most responsible way, to yourself and your child, and in the process, to your husband.