In most States... Kindergarten is not 'mandatory.'
If a child enters school... at say turning 6 years old... some schools will then put that child into 1st grade.
Maybe, you could home-school him... to prep for 1st grade... or, just enter him into Kinder now... you said, your son is ALSO 'socially' fine. Not only academically.
Being youngest or smallest... has not hampered other kids, entering Kinder at turning 5 years old.
My daughter, entered Kinder at turning 5 years old... she entered at 4 years old...then a few months later, turned 5. She did fine. About half or less than half of her classmates... were late-born kids who were turning 5 upon entering Kindergarten. Physical Size.... was not confined to only the late-born kids being small. Even some of the turning 6 year old kids... were small too. Which I witnessed myself. Age has nothing to do with 'size' of a child.
Just some thoughts. My daughter is now in 3rd grade and although born late... is doing fine.
Here in my State... the age cut-off for Kindergarten... is in August. If turning 5 by August... the child can enter Kinder. BUT I hear, that in the future (who knows when next year or the years after), they 'may' be changing the cut-off date... to June or July (summer). Which I think is questionable... and I do not know of any other State, with age cut-offs that... way.
all the best,
Susan