This is just off the top of my head, so it may not be worth much.
It sounds to me that he might be anxious, perhaps scared. He wouldn't know how to talk about that if he wanted to. Boys are not supposed to be scared. It's not cool. That's why some of them turn to alcohol and/or drugs.
It's a scary thing, though, to think that in not too long a time he'll have to take on grownup responsibilities, and won't be a kid any more. Advanced classes are, well, advanced, and perhaps he's afraid he can't cut it, so he's cutting the ground out from under his feet instead. No, that's not too bright, but the thing about teens is that they're not bright enough to be adults yet. (I think some older people aren't either, but that's a different matter.)
He might even feel as if he might fail the family he loves. Not that he could ever be a loser in your eyes, but it might be on his mind.
I wish you could somehow take him to the mountains for the weekend... or take him out of school for a "mental health day" and go play miniature golf or something ... anything to say, "Let's just get out from under for a while." No lectures, no questions (unless they're his), no manipulation - just rest, a change of environment, maybe a little fun with Mom and Dad. You can't make him talk to you, but you can be with him.
A dozen years ago, a career teacher (at a boys' school) named James Kelly wrote a book called "Respecting the Man the Boy Will Become." He taught (teaches?) at a Catholic school, and I'm not Catholic, but I like the book so much I wish I had found it while my sons were teens. He says over and over that "rarely will a kid tell you what he really needs and what he really feels. Frequently he doesn't know himself." But what he needs to know most is that he is loved. All other things being equal, hang in there, let him know that he's loved, and hopefully he'll get through this "dark night" soon. He can always catch up on his grades. They're not the most important part of him.