5th to 6th grade is a huge adjustment even for advanced students in general. It is about maturity, self responsibility, socialization, organization as well as subject content.
If he moved up, would he be able to be in the accelerated 6th grade classes? These classes move much quicker with a lot more homework. More involved reading and longer reports and more depth for the projects. I bet his friends would all be in the accelerated classes and he may not. Or he may struggle from not experiencing the transition from fifth to sixth grade work.
Add in hormones of the girls, and boys starting to get facial hair and voices changing.. Would you son fit into this?
Dances, football games, gym with older boys.
Not sure this would be easy enough to justify a move for friends.
Take a good look at all of this and speak with the teachers you trust. Go n your own to the middle school,and ask if you can observe the hallways for changing of classes, the lunch room experience, a PE class and the 6th grade advanced math, language arts and social studies classes..
Some of he kids will be taller than you and your husband.