It sounds like a much longer, franker discussion's needed here. Have you asked the preschool teacher/director the questions the way you have asked them here? "I don't know what goes on in the 4-year-old class that would make it too difficult for him to succeed in. What is that program like? What will he be expected to do and accomplish?" And "Why are we discussing this so early? Are you seeing behaviors or delays that I should know about and if so, what are they, in detail?"
The fact they want to discuss this now could mean that they are seeing things they truly believe mean he should stay in the 3s class, but if so, they need to tell you these things clearly.
You have the right questions in mind but I'm not clear if you've asked them explicitly to the preschool. You mention that his speech is delayed; that may be a bigger concern for the school than they have told you, despite his progress.
I would say that overall, as someone else put it, why rush him? Though you haven't said so here, are you on some level concerned that he wouldn't start kindergarten "on time"? Please don't be, and give serious consideration to letting him repeat. Better by far to repeat some preschool than to get to school and have issues. I can only go by the experience of a boy we know well: His parents were determined that, because he "tested smart," he go to kindergarten right on time, and then move on to first grade on time, etc. "so he won't be bored academically." But he was not ready in other ways, was the smallest and youngest, lacked the needed ability to organize himself for school that's expected by about second grade now, and despite his bright intellect, he struggled until about fourth grade. If he'd repeated kindergarten as was recommended by his teachers, it might have helped so much. So another year of just maturing, playing with others, getting used to routines and getting the rambunctiosness out can probably only help any 3-year-old boy.