He may be struggling with a learning disorder. You can have him evaluated by your school district's Educational Service District Office. THis service is mandated by law and is without charge to you.
I would also consider the possibility that he is reacting to feeling pressure to learn and/or perform. This could be a power struggle between the two of you.
When you say "he refuses to sit down and learn" I wonder what that means. My grandchildren never sat down and learned. I sang the ABC song to my granddaughter as a lullaby from the week she came home from the hospital. We counted things as a part of play. She learned colors as she put together puzzles. Learning was a part of everyday life. We did nothing specific like sitting down to learn. We were relaxed and undemanding. If she said her ABCs we praised her. If she didn't, no big deal.
Has your 5yo never talked about letters, numbers, and colors in the process of living? How was he acting at 3. Does he know his name, ABCs, numbers, colors and just refuses to say them?
There are several learning disabilities and/or emotional issues that could be causing his behavior. I highly recommend getting an evaluation with the school district. If he does have difficulties that will negatively impact his ability to learn the district will provide treatment, also free to you.
My 4 yo son went thru this process just before he turned 3 because he still wasn't talking. He goes to a therapeutic preschool paid for by the school district. The school is also addressing his anger issues. And his mother is getting support from a team of professionals.
Also keep in mind that children all mature at different rates. Not all children are ready for kindergarten at 5. An evaluation would tell you that too.