Just my two cents as a mom who has worked her way up the corporate ladder... when kids are young, (like under 7 or 8), they need you a lot less than they do when they're older. An infant can be cared for by anyone, really. As long as that person is loving, attentive, makes sure they are fed, changed, and take their naps, it really doesn't matter if you're doing the caregiving or someone else. (I know every SAHM in the world is howling right now, but this has been my experience.) So I don't think you need to worry so much about that, or feel guilty.
Surprisingly, once your kids get older, they need your attention more. My older daughter is now 9, and when she got to be about 7, she started telling me she didn't like daycare, she wanted ME to help her with her homework, she missed me while I was at work (and I worked A LOT, most weeks 60+ hours, plus travel). Around this age, kids also start getting involved in more activities, and are more serious about the activities they're in than they were when they were tiny. In the past year, our household has been a madhouse with me working crazy hours, the kids being in after-school activities (that required me to rush home from work, pick them up from daycare, rush off to dance classes or whatever, then come home, rush through dinner which was usually takeout, etc etc). Anyway, a few weeks ago I was laid off, which was a HUGE blessing. I had felt for a while like I was ready to be home with the kids for a while, and just couldn't justify taking the leap (financially) to do it. I'm SO GLAD to be home with them now! Our lives are calmer, the kids are even sleeping better at night (less nightmares, less craziness right before bedtime). Anyway - I think my job was creating a lot more havoc than I realized.
Bottom line, your kids are only young once. Even if you go all-out and work 60 hour weeks, you'll still make less than your male counterparts. You may as well do what you want to do, even if it means your career will be stuck in slow-motion for a while. Life is too short to spend it questioning your choices. As of this moment (granted, it's only been a few weeks), I'm thrilled to be able to stay home with the kids. Life is a happier place for sure. I'm dreading having to go back to work eventually! But if you feel like you don't want to pass the opportunity at work by, then go for it. Your child will be okay no matter what you do. It's really up to you, and what you can handle at this point in your life. Best of luck with your decision!