I don't have stepchildren, but I grew up in a home where my sister and I had different Dad's (our Mom was married to my Dad). Having watched my sister struggle with her identity as a kid and the outcome as an adult, I thought I could add a different perspective.
She's a kid and she wants you to love her. She's 12 - of course she told you to back off but she also probably doesn't want to be "disloyal" to her Mom. Her birth Mom's obvious about why she'd rather you not be a parent and I'm willing to bet that Dad is too worried about being the bad guy for "replacing" her Mom that he's doing more to help create the canyon that's being built between you two than really finding a solution that works for everyone.
As an adult, I have now been able to speak openly about how my sister felt without getting defensive about my very existence. But she felt really lost because the family she had no longer existed and she didn't feel like she fully belonged in the one that included my Dad and I. This was ground zero - meaning, it didn't matter how my Dad was, that was where she was coming from.
So what do you do? I would treat her like the grown up she's longing to be (at least for the purposes of this conversation). Take her out to lunch and pedicure's yourself. Not to compete with her Mom - just because it's something that can be a distraction but offer opportunity for a real talk. Take her to lunch first and talk to her as if she were an adult. Don't talk down to her like she's still 4. Be honest with her (without the negative stuff and NO BLAMING ANYONE - she will only want to defend her Mom and stop listening to you because it's her Mom and she will love her regardless of Disney tendancies). Such as:
"I wanted to take you to lunch today so we could have a girl chat about our little family - your brother, me, your Dad and your Mom. We are all a family. It's important for you to know how much I love you and [don't use the word 'but' - it automatically discredits what you just said] I'm feeling like you and I are having a hard time lately. I was hoping to get your thoughts about what you want our relationship to be."....
"I know that I'm not your Mom. I do love you as much as your brother though and I love being able to be a part of your life. Sometimes we are going to argue and sometimes I'm going to have to lay down the rules. But that doesn't mean that I'm trying to replace your Mom. It means that I'm treating you the exact same way I would treat your brother. Because we're all a family. You, your brother, your Mom, me and your Dad. We all love you and we all want to help your grow up. So you tell me - how can I help you grow up? What can I do that would make you feel better?"
It doesn't mean that you HAVE to do what she says. But if you actually listen to what she's saying, she'll feel respected and open up.That way you're not left trying to guess what she wants. You'll have a baseline and the proper background to assess why she's acting the way she is. Some days it will be you, some days it will be the cute boy in Math class. You don't know.
Good luck!