I had my fourth child (BIG surprise!) at 40. I had some problems that I did not experience with any of my other children, but nothing serious that required bedrest. I was in great shape, exercising 4 or 5 times a week, but I still was tired a lot and it just hit me more than my other pregnancies. So here's my personal experience.....
Going from 2-3 kids is a BIG change. Basically you go from man-to-man defense to a zone. There will always be a child not getting attention because there's only 2 of you. That's something to think about. So this will be a HUGE change to your kids. There's also the car factor, there won't be many cars that all the carseats will fit in. You're going to have to do the SUV or minivan.
My last one is lots of fun. There is a 10 year age gap between him and my oldest, 8 years between #2 and him and 6 years between #3 and him. It took them awhile to get used to the crying and distration of a baby in the house. They have lots of fun with him, but they still complain.
I am usually the "old" mom at any classes or playgroups with him. Not that it bothers me, but it's another factor. I keep in shape, so it's not like I can't keep up with him or anything.
That being said, I wouldn't give him up, but there are lots of things to think about. I was actively working on not getting pregnant when I found out about Luke, so I figure that he's set to do something very important! Maybe he has just by being in my house! :-)