I'm a stepmother and I understand the blame game. But these stepchildren are adults, and at some point, they should be able to figure out that no third party breaks up a good marriage. Ever. If they can't do that, their chances for a good relationship themselves are pretty slim. Fearing a third party is just saying you don't trust your own spouse/partner/boyfriend/girlfriend and therefore that you don't trust your own ability to choose a worthy person. So hopefully they will get past this.
This is still new to them in many ways. Some may be mad at dad for leaving and finding someone else "so soon" (I know it's not soon in your mind, but for those who never saw any option but "Mom and Dad," it may be.) Some, though adults, have problems envisioning sex between their father and anyone else, especially one who is "not Mom." Some may be less hostile than you think, but just feeling awkward.
The main thing is not to give the impression that you are forcing new traditions on them. Their old traditions are already in upheaval - they used to have Mom & Dad together, then they had to learn to juggle Mom in House 1 and Dad in House 2. Now they are learning to incorporate you, your child at home, and your 2 oldest. That's a lot of adjustment. They don't want to leave either parent alone on a holiday, and because they are adults, they may feel they have to "take care" of one or both parents when they still haven't figured out how to take care of themselves.
I think you let their father invite them and figure out what would be the most comfortable. If you can adjust your own traditions with your kids to include whichever of his kids want to come, great. If everyone compromises a little, that's the best thing. If it doesn't happen right away, give it time.
I don't know what holidays you celebrate or how you do so, but if you do Thanksgiving but they want to go to their mother's, then you can do a fun "Leftover Day" on Friday or Saturday, with each of them invited to bring something they just love on the day after. I know a synagogue that does a Friday Pot Luck with all the leftover pies and cranberry breads, and everyone loves getting rid of their leftovers!
If you celebrate Hanukkah, there are 8 nights and it's easy to find a night everyone can come. We usually do a charity night, where we defer gifts and gather things to give away to a chosen good cause (women's shelter, food pantry, whatever). Everyone brings something to contribute. Pick something non-controversial to all concerned, though. Better yet, let them pick. You could do this on Thanksgiving weekend (or day, if you see them then) and pick something related to Christmas if that's your holiday - all kinds of organizations have "giving trees" to help buy things for kids who would otherwise go without. It's nice to do something for older kids too, who often get forgotten in the "toys for tots" type drives. All of the kids/stepkids are closer in age to needy teens than you are anyway, so let them be the guiding force. Your husband can set aside a budget and let them spend it as they wish, then everyone takes part in delivery the choices or wrapping them, whatever the recipient organization wants. Food pantries get a lot of food in November and December (and really struggle the rest of the year), but you could do something like not-so-perishable fruits (oranges, clementines, apples) and make up some little baskets or bags with ribbons (enough for one family), and donate them to the local pantry to be given right out. You could also do toiletries - women's sanitary supplies, deodorant, toothpaste, toothbrushes, hand lotion and more. This gets all of your children out of their "it's about me" mindset and into the "help someone else" mindset, without them focusing on making life better for you. They're all old enough to submit their own ideas here, so let their father brainstorm it with them and you with your own kids. These little projects also work fine if not everyone can get together on the same day, but you still have one activity.
If the stepkids have favorite recipes that don't conflict with their mother's favorites, you can do a recipe swap or ask them to provide the recipe to you in advance for any meal they are attending. Your husband could invite them to help you all pick out or decorate your Christmas tree, unless that's too emotional for them. Everyone could bring a new ornament so it's not just your family's traditional decor and it's not what they're missing from their mother's home - it's all new stuff.
Do they do old-fashioned board games or jigsaw puzzles? Those might be great to connect 20-somethings to the child you have at home. They are communal by nature, and puzzles can be gone back to again and again, without completing them on day 1. If you don't have a card table, borrow one from a friend or buy your own, just so the games aren't on the dining room or coffee table when you want to do other things.
Mostly, try not to solve everything in one year, and try to let things evolve as you get to know them better.