Ooooh! Flashbacks to my childhood! When I was in jr. high school, my (first) stepmom badgered my dad in to taking us each weekend, as was stated in the divorce papers (my folks split when I was three). It was really awkward at first...she had four kids of her own and I didn't know them OR my dad all that well.
But, I'm so glad she did this because after a while the awkwardness went away. As a teen, I made more "sense" to my dad and we developed (and still have) and excellent relationship. My second stepmom worked on him to spend more time with family and when she passed, my dad started make twice yearly visits to get to know my sons. This wouldn't have happened without my stepmoms' influence.
I get that you don't have a good relationship with your dad, that you disapprove of his past actions. My dad did a lot of crummy things, too. He left my mom, he was a total jerk about child support for 18+ years, he never made an effort to know us, everything seemed to be more important than me and my sister. But the strangest thing happened. When I let go of the past and started to get to know my dad--first as a teen, then as a college student, then as an adult--I discovered that while he did a lot of really crappy things in his past (who hasn't?) that he was, in fact, a person worth knowing. I discovered that we shared a passion for history, a love of accounting and historical fiction films. We both love reading and telling stories. We're both Anglophiles and we both love to travel (well, he used to). Best of all, we both love my sons.
I guess what I'm trying to say is that maybe, just maybe, it's time to let the past go and see what the adult you and the older (maybe wiser) dad can do in terms of developing a relationship. Over the years, my dad and I have talked about his past and its effects on me and I learned so much about him, both good and bad. He's apologized for a lot of things (something I never expected). Because my dad is a self-avowed curmudgeon, I've had to be the adult and work the situation in to a positive. Maybe your dad is as scared of your rejection as you are about being hurt by him again. And your husband...he loves you so much he doesn't want to see you hurt again, too.
For me, I'd rather make the effort. I would hate to die, or have my dad die, knowing that I didn't take advantage of the opportunity when I had it. It wasn't always easy but I'm glad I did try. It's been so worth it. My dad still has his faults--we all do--but in his older age, I discovered a man who faced his mistakes, admitted them, married a wonderful, wonderful woman who made him very happy (and he nursed her through a bad case of MS and ultimately buried her) and has reconnected with his family and daughters.
Good luck with your decision. Be sure it's one you can live with in peace.