Some camps just aren't good fits. I always found the camps where the kids were grouped in an age group such as 5-7 were good. Sometimes they end up lumped together at lunch time, etc. but to be hanging out with 11 year olds at 6 - not so good. I hear you.
None of my kids would have enjoyed the academic part (even very light academics).
I found the camps that were the best were camps such as our local rec center provides. They have about 6 camps per week - you can choose which one suits your kid's personality. So for example there was a baking one (they'd do a new cupcake every day), there was a Survivor one (every day they acted out activities like on Survivor), hockey camp, Sports camp, etc. It would change weekly too - so a lot of kids went all summer as daycare. They would do swim in afternoon, and played outside too.
So maybe check your rec centers. Here there are also camps at the big swimming pools that have gyms attached - same idea. So check out sports places.
Our area had a listing of camps - from there we found a drama camp that suited one of our kids.
A lot of the private schools and daycare centers have summer camps. They tend to rotate them too - and kids grouped max to 2 year age difference. They do science weeks, Lego week, etc.
We did a soccer camp, hockey camp, etc. You would just have to look up the local league and see what's offered.
What's big here are canoeing/kayaking camps. Kids as young as preschool can go.
It's usually only a portion of the day they do the actual water sports - the rest is activities and playing outside.
We had a few hit and miss tries - one of my kids hated art camp. But art galleries sometimes have camps. I wonder if there is a listing or directory for your area that you could get a hold of, and talk to the directors to make sure it would be a better fit? Good luck
Sometimes big kid camps can be exhausting - I found age 6 hard because they were the little ones at camp, and it's a long day.
Worst comes to worst - would you be able to hire a teenager to watch him during the day, and have them take him out? I did that all through my teen years. I was a summer time nanny.