I know you wanted a doctor's perspective, which I'm not, but.... I switched doctors once - actually happened kind of like your situation - my usual doctor was on vacation and I was seen by another doctor within the same practice. I wasn't happy with my original doctor anyways (terrible bedside manner!) so any time I needed an appointment I just continued with the "new" one! No drama - no one ever questioned why I switched. If they would have, I was prepared to tell them! To be honest, I don't think the other doctor even noticed!!
It's your body - you should be seen by someone you're comfortable with and not worry about hurting anyone's feelings.
Good luck!!