Well, I don't think it's gender-related or age-related, necessarily. My friend is 40 and he cannot stand loud noise. I cannot have him in my car as he will turn down my music, which irritates me. He claims it is too loud, yet when I speak to him, he is always saying "what?" and says he cannot hear me. People think I am screaming at him when we hang out because that is the only way he can hear me, I think he has hearing problems. He also avoids rock concerts and pretty much anything he considers loud, crying children etc.
I have excellent hearing, can hear things from miles away, love loud music, concerts, and hanging with musicians, it's a positive energy I get from all these activities. What irritates me the most are people who honk, it puts me in a bad mood. Also, yelling. I cannot be anywhere where people are yelling or slamming doors, it raises my blood pressure and makes me feel trapped. All these things are associated with negativity. Curiously, I can tune out the crying kids at restaurants, even bad music in a restaurant (which my friend cannot seem to tune out).
My mother has confirmed hearing problems. I must scream when I talk to her, she blasts the TV and still manages to fall asleep with it so loud, and cannot hear a lot of small noises, like crickets (which are very high-pitched to me). Based on all these things, my theory is that it's more of a selective thing, like "this is what irritates me and I cannot tolerate" and pick and choose versus "I cannot stand any noises, they are all so very loud and everything upsets me." Notice you said only certain things irritate you, not all noises or that you have difficulty hearing or everything sounds magnified.