I used to think things like CPR were important. Now I think it's all about distraction. Teens are on iPads and cell phones constantly and they THINK they are focused on 2 things at once, but they are not. I'd tell her you pay a higher rate and re-hire sitters who are not on the phone. They are there to ENGAGE with your kids, not just be available to call 911 if the house catches fire.
Ask about experience & references, yes. Basic first aid, yes. Ask what they like about sitting, what activities they like to do with a child your children's ages, find out about HER. See if she has any clue of what to do with them.
You can ask her how she handles disputes and fights between kids. Mostly give her your rules. I was much more safety and activity oriented, and I cared a lot less about whether all the dinner got eaten. I left a listed taped to the inside of a cupboard door about where bandaids were, that they could have snacks (including the sitter), what TV channels were allowed, how much TV was allowed (depended on the time of day - on a beautiful day, nothing. Rainy day or nighttime, different story), where the board games were located, where the bike helmet was, neighbors' numbers, etc. I found a sitter was more likely to call a neighbor than the pediatrician - and it made more sense anyway.
Don't make it the third degree - it's not a job interview. I also started my sitters when I was home so I could get stuff done and I could still overhear the interaction. I posed it as "start slowly to be sure you like these kids and so that I am here if you need to know where anything is or ask what's allowed" rather than "I want to be here to supervise your every move."
Also find out whether she needs rides to & from, what her rate is, what she needs from you. (Does she like stuff written or verbal, for example.)