First, I have two distinct categories of backtalk: there's the grumbling, self-indulgent talk ( like "you're not my friend"), which I consider to be just letting off steam. Yes, it's juvenile, but kids are juveniles so I don't punish for this. They're mad, they say stupid stuff-- so do adults, right? As long as it doesn't fall into the category of name-calling, I ignore it if I can or sometimes send him to his room "because you are really grumpy at me and you need a break." He is allowed to come out whenever he is ready to be pleasant.
And then, there's being non-compliant. "No-- I won't" sort of backtalk.
Sometimes, my son (5) will say "no" to a direction, and then just go do it anyway. So I often keep my mouth shut for a moment or two and look to see what his actions are before deciding to address the situation. Sometimes, it really doesn't need addressing and he's just grumbling.
However, if I asked my son to take a time out and he refused, I'd send him to his room. "Come back out when you are ready to do what I ask." So, while it doesn't seem like a punishment, it is allowing him to take a break and get his focus together. If he wants to be back out with us, he'll *still* have to take the time out time before resuming any fun in the common areas of the house. This also eliminates a huge power struggle-- instead of physically forcing him to sit in a chair and using up MY time (and we moms have plenty else to do, don't we?), I make the noncompliance be HIS problem.
There's a great deal of motivation to be had when it comes to being allowed to be part of the social activity of the house. I want my son to learn to take breaks when he's feeling uncooperative socially; I want him to learn to go play by himself sometimes if it's hard to get along with other kids. I want him to learn that if he's in a bad mood, he's welcome to take it elsewhere and come back when he's ready to participate in the ways we need him to. Don't we wish more adults knew how to do this?:)
And you might try turning it on her: "You know, you are being so unpleasant, I will take a time-out in my own room. I'll come out when I'm ready for your company." Depending on what's happened, if you go off and disappear for ten or fifteen minutes, this may be enough for her to check in on you. Then you can decide if you still need her to take that time out or if you two are okay with each other. Of course, this only works if you have an only child or older kids who are engaged, etc. You can take a baby or toddler to your room with you if need be. (This is called 'room clearing' and is sometimes a great response to big, bad acting out or tantrums. Everyone leaves the angry child to yell or carry on without any attention.)