No, it shouldn't be that hard. I tried for 17 years to save a bad marriage, nearly split at the same time you did (the 6 year mark), stayed and toughed it out, did counseling, had a child, etc., and I've finally thrown in the towel. It took seeing my husband be snippy and mean to our child, making her cry over the smallest, stupidest thing ever, for me to decide to leave. And once I decided that, I found such peace. We're still going through our divorce, still living in the same house (though different rooms) while I look for a place, and I can't get over how happy I am. Even my child seems happier, which is probably because 1) there's less tension, and 2) my husband is finally being a good dad now that he knows he won't have her around all the time. He's so much more patient with her and interested in her now. I know the actual moving out part will be hard for her, but she'll be back at this house nearly half the time, and I've told her she will have two houses of her own instead of just one. For all his many faults, I think my husband will do his best to make it a smooth transition for her, too. It's funny--we had a minor spat last night because I was busy with something and couldn't walk away fast enough before I got sucked in, and as unpleasant as that was, it reminded me of how every single day used to be like that. He would get home, and if the house was spotless, he'd complain about dinner ("did you use unusual spices in this?" to let me know he didn't like it). If dinner was great, he'd complain about the bed being rumpled and not perfectly smooth. Or he'd fuss at our daughter to put her toys away even if she was still playing with them. It was nuts. I've spared myself a few months of that insanity since I filed for divorce, and the peace has been so welcome and healing.
Anyway, right now you and your husband are not good for each other or your kids. Fix it and stay, don't fix it and leave, but do something. I'm glad I stayed longer, if only so I could have my wonderful daughter, and it's nice to know I really did give the marriage the ol' college try, but those extra years did take a toll on me. Best of luck to you with whatever you decide.