I've experienced this with my daughter who started with me as a foster child. She was 7-10 in age. She was very difficult for a day or two or three after an hour visit with her birth mom. You've tried everything I'd suggest.
I'm also beginning to experience attitude from my 10 yo granddaughter. We suspect a lot of her being difficult to get along with from time to time is related to hormones and being aware that she's growing up. It feels to me that at times she's caught between being a child and a teen. She wants to be considered more grown up but she's having difficulty letting go of being a child.
I suggest that your step-daughter is experiencing a double whammy of being confused about being older and confusion about why she has to spend time where she feels she's unwanted. Coming back home to be with the family that wants her may be too great a shock with which to deal. I suggest trying treating her return in a more laid back way. Perhaps when you try so hard it makes her feel that you're trying too hard and what does that mean? Are you trying to prove you want her which could mean that it's not enough to just be loved? More confusion.
Tell her you're glad she's back but then treat her like you do the rest of the time. Let her have time to decompress in whatever way she wants to do so. Take her lead. I suggest you try continuing the same routine all of the time.
Also let her talk about what happened and how she feels without making any comment. Just listen. Never make judgmental statements about her Mom or the other household. Remain as neutral as you can be.
I also suggest that getting her into counseling could help. My foster daughter and I both went to the same counselor to process our feelings about many things. The having to visit this mother with whom she'd most likely never again live but with whom she had to spend supervised visits was a really big issue for both of us. My foster daughter and then adopted daughter has only since reaching adulthood been able to talk about her feelings with me. This has happened only thru counseling.
Does her mom know how she feels? Are you or her father able to talk with her or facilitate your step-daughter talking with her about possible ways to change just one or two things so that she's more comfortable. For example: perhaps your step-daughter could have a special place for her toys so that the little kids could not play with them unless they have her permission. We can only learn to share when we have control over our own things.
I can understand how the mother is overwhelmed with two babies. Perhaps she just doesn't know how to make this better for her daughter. If your husband can't talk with her perhaps there's a third person who could be sympathetic with the mother and helpful at the same time.
If your step-daughter doesn't have her own space (bed, drawers, portion of closet) she's really just visiting. But she's expected to act as if she's living there. How confusing! And then you treat her special when she gets to your house where she knows she lives. Again, confusing. Make her life as routine as possible. A part of that routine is doing things to help her feel special. Do those all of the time. Treat homecoming as routine and no big deal. You're then letting her know that you know she can handle it. She'll figure it out.
One thing that I did eventually do was to make a routine out of returning after a visit. We had dinner at McDonald's. We frequently had dinner at McDonald's. Inability to eat home cooked was one of her emotional issues. It's taken me years to once again willingly eat a McDonald's hamburger. lol And then we went home and did whatever we usually did in the evening.
My grandson goes to his father's every weekend. He's 7. He frequently is cranky and difficult when he returns. My daughter and her husband consciously try to have chores done so that they can have a quiet at home time when he gets there. For example: as a family they can sit down and watch a movie together.. Sometimes he just wants to play alone in his bedroom. They are able to take their cue from him because they have nothing else that they have to do tho there are always things around the house that they can and do do when he's wanting to be left alone.
Sometimes he's just a bear to be around no matter what and they let him know, just as they do when he's like that during the week, that his behavior is unacceptable and he goes to his room until he's able to be reasonable. When he comes out he gets hugs and attention just as he does during the week.
helps most of the time but not always. Just as your step-daughter has to find a way to adjust so do we. Let her know that you know this is difficult but that you trust her to be able to work it out with your help when she asks for it.