My kids were 6, 4, and 2 the first time we could afford to go. We were going to go when they were 5, 3, and 1...but hubby lost his job and we couldn't afford it.
We had planned to go again when they were 8, 6, and 4, but hubby and I both got new jobs a few weeks before we were supposed to go.
As of now, we have a room booked for next March to go back, but that's assuming nothing big happens!
I saw tons of families there with little babies. 2 is a great age to go too though and they will remember it...if your child is anything like my youngest. He still tells me all about it and he is 5 now. So basically, go whenever! I think it's fun for all ages and even if the baby can't go on a lot of rides, they will still be there and it's a great family memory.