You are NOT over-reacting. I am married to a divorced man, and he has an 8-year-old son. If he talks to her too long on the phone about their son, I get annoyed. I feel she does NOT need to know any details about our life, our plans or anything at all. He has almost nothing to do with her, except for arranging to pick up/drop off his son. It would make me extremely uncomfortable to know he was talking to her about me, and spending time with her. Afterall, they were in love, have had sex, and had a child together. Throw a little emotion in there, and things can easily get out of control. My husband tells me that men love to be the "hero" and rescue women. Calling for a panic attack? Sounds like she is looking for attention. What were the circumstances of their divorce? How long have they been apart? Who filed for the divorce? If you don't think you can give him a "it is her or me" type of deal, then try making friends with her yourself. Keep your friends close, but your enemies closer type of thing. If he is a typical man, the idea of you two being friends will freak him out. No man wants two women who have been intimate with him to be friends. Tell him conversations about him that you two have, and maybe that will have him saying... don't do that. Then you can say, if you stay away from her, I will too! I'm divorced, too. And my husband would really freak out if I was contacting him/staying in touch and hanging out. In this type of situation, you can't look like the bad guy. Try to finesse your way out of it, instead of throwing a fit. I love the saying from "My Big, Fat, Greek Wedding." A man is the head of the family, but the woman can turn the neck any way she wants to!!
Good luck!