Hi J.,
I think you will know when you've found the right nanny. We found ours through a referral from our cleaning lady, who had been with us for 5 years. Our nanny had raised her own 7 children herself, so I asked what her children do now - 2 of them are ministers, 1 doctor, 2 at-home moms, 1 nurse, 1 teacher. All of them went to college (and she raised them in Mexico, poverty stricken). So that told me something about her resourcefulness and her ability to enrich a child's life. Next I asked to speak with people who had known her for a while - spoke with 4 or 5 people including the pastor at her church, people she had worked for before. I did not bother with a credit check or criminal check because she had only been in the US for a year at that point. Then, the most important thing, I took a week off from work during her first week with us and watched her interact with my baby. The second week, my husband and I dropped in on her at least once a day unannounced, and continued with that less and less often as we became comfortable with her. We were heartbroken when we moved 3 hours away, but we still exchange Christmas cards and see her when we are in the Bay Area.
We are on nanny #2 in our new town, and we found her the same way - through my mom's cleaning lady. This nanny had a lot of nanny experience so I spoke with her former employers, all of whom gave her glowing references.
Both nannies of ours have had infant/child CPR and first aid from the Red Cross. (If they are not already certified, pay for the class for them.)
If you are unable to drop in unannounced on your nanny, you should get a nanny cam. Your home alarm company can set it up for you and give you some peace of mind to be able to "drop in" whenever you like. Make it clear to your nanny that you will be dropping in to observe, and/or you have nanny cam - I do not know of any nanny who would be offended by this. Also you will need to make clear how you would like things to run around your house. For instance, our first nanny would gladly sit there for an hour rocking our daughter to sleep and would seriously even hold her while she slept! That didn't really work for me because I did not have time to do that at night or on weekends. So I had to walk her through how I wanted my daughter put down for her naps (awake). I don't think she agreed with me, but she did what I asked cheerfully.
Good luck with your nanny search. I hope your nanny is as wonderful an addition to your family as ours have been to us!