Hang in there! What you're feeling is totally normal. I remember right before my son turned three, many of my friends whose first kids are a few years older and their second children just a few months older than my son would say often, "I just hate 3! I can't wait until he/she turns 4!" I didn't know what they meant until my son, who is my firstborn was close to three, and then I knew! Luckily I had all these friends with 5 and 6 year olds to reassure me it was only a phase.
My son who is now a mostly-wonderful 4 was just awful at 3! Our daughter was born when he was 2.5, and he became clingy, needy, mouthy, refused to be potty trained until after he turned 4.............there were days I didn't like him very much either. I think he was bored, and at the same time, although he never purposely hurt her, somewhat jealous of his sister and the attention of mine she was getting. He's always been a little more difficult in comparison to his sister (more needy, sensitive, etc....), and it's hard not to compare and sometimes like her more. Like you, I have a wonderfully patient husband, and he really helped during that year.
I'm not sure where you live, but perhaps you can check into some ECFE or preschool programs where you can drop him off for a few hours of "boy play time" for a reasonable fee, or use this site to hook up with some other stay-at-home-moms for some play dates. Sometimes just a little time away makes both parties appreciate each other.
One place my son just LOVED was the Edina Family Center. We still go to their summer programs. I don't live in Edina, but they accept nonresidents and have just wonderful programs for kids of all ages. They have "block time" which is up to 3 hours, up to 3 days a week, where for an hourly fee, you can drop your kids off with child care professionals at the family center. They also have other preschool-type programs. They have a web site connected to Edina Public Schools - don't have the site address in front of me, but are easy to find.
If that doesn't work, check out your public schools' ECFE programs and local churches for preschool programs for kids as young as 2. There are several music classes through the Family Music Center (they require parents to attend, but they are a great way for both of your children to experience music together) and lots of programs through local nature centers and libraries too.
However, If your feelings get more negative and serious and you feel like you want to hurt your child, please call someone and ask for help.
Take care and know you're not alone.