I don't know this doctor, but I will say that, if you were uncomfortable on the phone (and I don't blame you), you might consider making an appointment. Tell him you were uncomfortable with the resolution of the issue, felt you were perhaps bothering him, and you want to resolve it. Ask him if there was a reason for his silence or if there's something you could have done differently. Now, frankly, I think HE'S the professional and YOU were in pain and a bit scared, so it's not up to you to please him. But it's a way of getting into the conversation. Tell him you don't want to feel that you can't call your doctor - you're a mature woman and you deserve to be respected and listened to.
Tell him you've heard great things about him so his reaction surprised you and you really didn't/don't know what to do next. Remind him that patients who are sick and in pain and scared don't always ask things just the way he might want, and it's hard over the telephone when people can't see each other's facial expressions. Also ask him specifically what can cause that kind of pain. Is the Clomid, which stimulates ovulation, causing some kind of major action in the ovary which caused the pain? What should you do/not do if it happens again? If you're still not satisfied with him, then switch doctors. If, on the other hand, he apologizes or explains things better, you will have a better relationship and he'll maybe learn something about how his phone manner can easily be misunderstood.
Trying to get pregnant is really frustrating - I went thru it too. Clomid worked for me with some other things and it's frustrating when each month brings a period. Be good to yourself.