In some ways it is not as bad as it seems. In other ways, it is far worse....
1. Wait. Give it time. Most of the time kids of divorce put a parent's new significant other through Hell as a test. "I'm not going to allow you in until you've proven you're in for the long haul. (I know. I was one, and I did the same thing." The wait may not be until they're out of the house. They may grow out of whatever this is in a year or two. It's possible. You should first be willing to give time to all the people you are asking to accept major life changes.
2. Work on your relationship in the meantime. A few things strike me about your fella. (1) He said no to a house big enough for all the children you currently have so it seems obvious he doesn't want more, Are you on the same page? (2) He seems to assume that all the children will leave the nest at high school graduation and never come back home. Did you have different plans/hopes for your son? (3) Is he often not open to discussing things you have different opinions on? I would not take kindly to an edict on the size house we would get.
3. Is there any extended family that can help? The kids may appreciate the continuity of a grandparent or aunt/uncle while you two get some time together.
4. Don't resent the kids, his or yours. They didn't ask for one of their parents to suck.
Good luck to all of you.