Hi A.. I feel the same way sometimes. I wish there was a cure for this.... My boys are almost 5 and 2 and a half and it wasn't until after my younger son turned 2 that I started to feel a little like I can wrap my head around things. I feel better when I exercise too, but I don't put too much pressure on myself (like if a wrench gets thrown in my day and can't work out, I don't stress about it). However, I do try to run with the boys in the double stroller even if it's cold outside, usually I just go for 30 minutes. So I'm with my kids and it's not a huge production to get to the gym.
Also, I started sharing a sitter with some of my son's classmates. Sometimes on a Saturday we'll go out to dinner with a couple from school and they will drop their kids off at my place. We'll go out somewhat early - 5 or 6 pm and we are back by 9pm, but it gives the kids a chance to play together (they have a blast), it cuts down on babysitting money, gives me time with my husband and also we get to know new people/families from school too.
I've also learned to ask my family and friends for help. My son broke his femer and was in a cast for 5 weeks. A bunch of people asked to help and at first I turned them down, but after I started to go a little crazy, I decided to take people up on their offers to help - drive my other son to school, etc.
My mother in law comes by once a week for a few hours and I used to feel like I needed to stay with her and have some quality time with the kids..... but now I do my own thing when she is here and give her alone time with my kids. I can finally do this without feeling guilty, etc. It's funny I used to feel bad if I wasn't spending a lot of time with them, but I think they actually have a better time when I give them some space.
I think the bottom line is that whatever you are doing, just do it, 100%, even if it is something for yourself and try not to feel guilty about something you are not doing. You can only do so much. Good Luck!