H.W.
Sometimes, old hurts can manifest when we are better able to cope and deal with them.
As some wise woman on this site wrote recently (and I can't remember who, so speak up and claim it if this idea is yours): Work as hard as you can to 'earn' the divorce'. That is to say, unless you do that hard work of trying to mend those hurts via couples counseling, working on improving communication, etc., you may be making a huge mistake.
I can tell that communication is certainly an issue. Your statement :"The times that I would just sit and cry he'd ask what's the matter... umm duhhh you should know that already. Granted he is a man and they don't always know how we feel. " -- this says a lot to me. Couples get into the bad but understandable habit of assuming their spouse should just "know"... E., I've been with my husband for nearly 12 years and we still joke to each other "oh, wait, I'm not a mind reader babe!"
Couples have come back from deep, hurtful grudges. I have seen it. If both parties are willing to do the work and accept their part of the responsibility for getting the relationship to the place it is-- and then really work on changing how they speak to and respond to each other-- I've seen a LOT of good come from this. If both parties are willing to invest in the progress of the family as a whole, it CAN be done.
You and he have BOTH had very real, daunting challenges E.. Guys typically want to fix things and sometimes, when they can't, they act out in the worst ways and do things which drive us nuts. I know this firsthand with both my ex-husband and current husband, especially during the heartbreak of miscarriage. I won't go into it, but there was such a fundamental disconnect with my spouse in BOTH cases, and they are very different men. (My ex is self-absorbed while my husband now is very pragmatic and 'big picture' when sometimes I'd prefer he were more narrow in his focus.)
All that to say-- even very good guys make mistakes. Don't give up on what you have if you truly want it to get better. Marriages are sometimes challenged with very hard, good fights. This is a fight worth fighting. Good luck, get some professional help (or consider your clergy if you attend a church) and don't walk away until you know for sure that you've done everything you can.
ETA: I want to add to something I've seen repeatedly on this thread, which is the possibility of PTSD. Being a PTSD survivor myself, I would certainly encourage you to get help with this. I could not have overcome it alone; my triggers were very deep and brain connections had to be reworked. It can happen, but it does take time and a trusting relationship with a counselor. Please take good care of yourself, E..