My son's teacher usually starts them around mid-2nd grade. However, my son started at 6 - at the beginning of 1st grade. She agreed to try it because he's very bright, and was already reading at almost 4th grade level. She said the time to start is mainly related to attention span and responsibility. Lucky for us, it's all worked out.
As for too late, there's probably not a limit. That said, I didn't start until I was 18 and at university (had to take piano as part of the music ed program). After playing clarinet for 10 years, I had a terrible time get my hands/fingers to cooperate - and we'll not even talk about adding my feet! lol
I suppose I'm lucky and on the very low end in terms of price. I pay $13 per 30 minute weekly lesson, and my son goes year-round. I don't know if summer lesson are standard or not.