That's a personal decision, but I'll add my own story.
I never used to get the flu shot. I am, in general, a very healthy person with a strong immune system. I just figured I didn't need it, I was careful, and really, how bad is the flu? Then, one year, I got the flu. And it was awful. It lasted for a full 7 days, and it felt like I had been beaten with a stick - every part of me felt heavy and I ached terribly. I was weakened so dramatically by it that my hand would shake when holding a pencil for at least the next month or so, and I didn't get my full strength back for several months. I was really shocked by the severity. The true influenza viruses (as opposed to what we call the stomach flu, which is actually not an influenza virus) usually last at least a week, and differs from the 24-48 hour bugs that people catch on occasion in severity.
I am very careful with vaccines, and my kids are on an alternative vaccination schedule, but I will confess that we always get the flu shot now. If the disease could bring a healthy 20-something in her prime to her knees, I don't even want to think about what it could do to a baby. However, I did wait with my son because he stayed at home with me and wasn't high risk at the time.
Also, the flu shot contains an inactive virus, unlike several other vaccines like the MMR. It cannot give you the flu. It can compromise your immune system temporarily, but in that case, you would be catching something else, and would be at risk from any vaccine. The other alternative would be an allergic reaction - if you think that is a possibility, you should look into it, because there is a possibility that it could be genetic.
Good luck with your decision!