I think you have to look at this objectively, and I know that's hard because this is not what you envisioned. I think it's important that you ask yourself some questions here.
What kind of father is he to your kids? Is he a dad to all 5 of them and not just his 2? Is he a good provider for you? Is he a partner in the parenting? Does he treat your kids with value and build their self esteem? Does he guide and teach them? Does he support you in your rearing of the kids? Does he help when you need it?
If the answer to any of these questions is "no" then I think you have things to think about. But, if the answer to these questions is "yes" then you have to consider the fact that he's pretty much a keeper.
If he is done building the family and you force the issue what you could end up with is a husband who is overwhelmed mentally, physically, and financially. You have to ask yourself the question "Would I rather have a terrific father to 5 kids or would I rather have an average or worse father to a larger family? Is my own happiness worth another person's distress?"
What good is having a larger family if the man you are making them with only became their father because he was backed into a corner for making a decision before knowing what it would be like to have a large family.
I think I would let it quiet down for awhile. Take time to look at it from his point of view and really look at his feelings in this and see if you can see his side. And if he's a good dad, keep him.
L.