J.D.
I don't know what to tell you about this one, as my hubby and I go back and forth about things like this all the time. lol
I am currently pregnant with baby #4, our first girl. We always said we would have 4 children, but since this pregnancy began, I find myself really wanting a 5th child. He goes back and forth about it for some of the same reasons your hubby does, yet he knows I will always feel that I am missing a piece of the puzzle if we don't.
What about talking with your hubby about just letting things happen? You can decide on a time frame together, and if you get pregnant, great!
Good Luck!