L.A.
Just ignore it EVERY time and continue the conversation. Do not reward him with any expiation or come back. It sounds like it has become a bad habit to make this comment to you.
I do not know about where you live, but here we have constant molds, pollens and allergies.. It is the price we pay for living in a nice mild climate with lots of trees.