Well...none of it sounds too good to me, except that it is unlikely that the central issue is anyone's opinion of you. It probably isn't about anyone's opinion of your husband, either. Any half-way decent attorney can force the transfer of custody now (if all the things you say about your husband are true), but you might want to think about this-
1. Her grandparents raised her mother, and that (apparently) did not work out well. They must feel bad about that.
2. They have had her since she was born. It would be extremely difficult to love a child and care for them as your own from birth and then have to give that child up. Particularly if you were focused on "making up" for the bad job you did raising the child's mother.
3. They are human, and humans make mistakes. It takes an exceptional person to put someone else's interests ahead of your own. Perhaps they are not exceptional people. Perhaps they are average, like most of us.
4. They probably fear they will lose her if you want to take her, but they KNOW they will lose her if you want to take her and she wants to go. The only "weapon" they have is giving her whatever she wants in the hopes that you won't make her do something she doesn't want to do. That is not in her best interest. Try to help them shut that down if you can.
Go through the courts if you feel she is in danger but remember that children are not property, they are human beings, and that these people were kind enough and loved her enough to open their home and their hearts to this child when no one else would or could, or at least, did. God bless them for that.
When she is older, she will come to her own conclusions and she will remember that her father and his wife were kind to her, loved her always, supported what was best for HER and always supported the people she has come to know and love as parents. She will either choose or not choose to accept your other children as siblings, no matter whether or not she lives with them on a daily basis. (Who wouldn't LOVE the opportunity to do that?) Don't force her to choose parents. Let her have a mom and a dad and you and the grandparents and help her understand that all of you love her and want her always. Families aren't just about who lives where, they are about who loves whom. Reach out to her grandparents with this attitude and you will probably find a very generous response from them. Somewhere in their hearts I'm sure they know the child's relationship with her father is very important and they want what is best for her, too.
Good luck, may God bless you and your family, included the inlaws, and may you find a way to work together to help this little one grow into a strong, wonderful person that all of you will enjoy loving for the rest of your lives. (You might even find a couple of friends yourself in the process and your other children might find some pretty nice surrogate grandparents.)