Sit down. Write down what the minimum you require from her for sanity if she is in your space. And it has to be concrete. "Be respectful" has no specific meaning and is useless to this exercise. It should include things like "No one will be coming and going from the house between midnight and 4 am unless x amount of rent is being paid and you are paying for and only consuming your own food. Requests to babysit will be made x amount of time in advance, or generally turned down. You agree not to be grouchy or nasty about refusals at the last minute" etc. Whatever it is that is your absolute literal bottom line. Write it down as a contract. If she wishes to live in your house, she agrees to that contract, and any consequences written therein for failure to comply. You agree that you will give her 48 hours notice to leave, and if she fails to comply you will notify the police, etc.
You also say in writing that you are willing to assume full custody of her daughter and will get a lawyer to arrange the paperwork for her if she wishes, or if she cannot find acceptable and consistent living arrangements for them. If she chooses not to turn over custody, and appears to be homeless, you will report the situation to to CPS to ensure safety of the children.
Don't get exciting, don't argue, don't back down. She's an adult and can comply with the terms of agreements she signs, or find a new place to be.