Sorry this story is a little long.
I am 34 weeks pregnant. At the mid-trimester ultrasound my baby was seen to have "mild pyelectis" (sp?) which is basically extra fluid in a kidney or two. I went back for a second ultrasound a month later and all was back to normal with the kidney fluid so no worry there. BUT at the first US they told me that it was one of the "soft markers" for Downs and my chances of that were now tripled. I didn't even know what my current odds were so the Dr. looked at the age chart and told me that at age 36 the odds were 1:270, so tripling that I am now down to 1:90.
I still didn't have the blood test or whatever further measures to rule it out. I didn't want to be further hyped up for no reason (like they rule it down to odds 1:48, and now I am even more stressed/worried, and all turns out to be well in the end anyway). I never did that testing with my other two kids either. But I know once they start putting the odds in your face and bringing that question into your mind it is hard to get it out.
For me, I figure I will find out in the end what my kid is like. Whether that means Downs syndrome, autism spectrum disorders, health problem, mental disorder... my child is a gift from God and I will walk whatever parenting path He wants me to walk. That's just me though. I know many other people who would prefer as much advance notice as possible to wrap their heads around what's coming. Unless it's something that would require immediate medical intervention at the birth, I am ok with not knowing in advance. The reality is you are never truly in the clear of what you are getting as a parent. I've got two healthy kids right now but who knows what's to come in the form of challenges for them, that will try my meddle as a parent.
Hope this puts you at ease a bit. I have noticed in the last 7 years since my first baby, they have gotten increasingly vigilant with pre-natal care. I went through a lot of extra tests this time around for something that came up with measurements in my babies brain (not even seen at the first US, only noticed when I went back for the 2nd one about the kidney issues that turned out to be fine). After several more Ultrasounds by further specialists and an MRI, I am back to square 1. baby is looking pretty good, all is well in that particular department. So I just think they overzealous in this day and age. But that's a good thing I suppose, the more care and monitoring the better.
Good luck, I'm with your husband, don't waste time worrying on a 2% chance. If you do you might as well worry about the other 3000 chances of things that can go wrong with a human life. If we did that we wouldn't enjoy pregnancy or parenthood at all. And that would be sad :(