He tells you that YOU are crazy and that YOU need to see a counselor.
He is throwing blame and guilt onto you. That is classic behavior of someone who is hiding something.
He may not be having an affair but may be feeling guilty about just hanging out. Or he may be having an affair after all. But either way -- his telling you that you're crazy is so disrespectful and cruel that it's time for you to tell him that you BOTH will start counseling the day he returns -- the same day! -- and that even if he is pure as can be, his reactions to your questions are a problem in themselves. If he is utterly innocent, why react angrily and rudely and with disrespect for his wife?
Call today and set up the counseling appointment. Don't ask him to go. Tell him you both are going and if he will not go to the counselor, you have another appointment set up for yourself alone -- with a lawyer.
If he insists "I'm not having an affair!" tell him, fine, then this is about the way you speak to me, which is by itself a problem.