Go with your gut, mom.
You need to get dad on board.
Why does he think this is OK? When he was this age, was he allowed to be places like this alone, so his attitude is, "I was fine, he'll be fine too"? That is a trap that parents sometimes fall into, not realizing that (a) times have changed, and many places like rec centers don't allow kids to be unattended and (b) time have changed, and parents need to be more attentive to placing their kids into situation where either the kids are vulnerable OR the kids are set up to get themselves into trouble -- even the best, nicest kid can yield to temptation to get up to mischief in a bustling public place with a pool. I guarantee that if your son gets nabbed by a lifeguard for roughhousing with buddies who turned up at the pool, or kids he just met there, the lifeguard is going to say, "Where are your parents?" Does dad want to be the one to handle that phone call? Or worse, does dad want to be the one to deal with it when he goes back to pick up son and son isn't there -- most likely because he walked off to get a drink at the store next door, but....
I also wonder: If the rec center is this cool, with rock climbing walls etc., why the heck doesn't dad want to spend time there with his son doing those things? Dad's missing out on what sounds like potential good times with his own son.
Does dad think maybe that son "needs to learn to handle himself alone" and this is a good way to teach him that? It's not the rec center's job.
Also, for things like a workout room or rock climbing wall, around here a kid of 10 would have to have a parent present to sign a legal waiver before a child was allowed to use those things. And many workout rooms don't allow kids as young as 10 on the equipment at all. The age limit in our area is 13 for weight rooms and classes, and we've used rock climbing walls that had a lower age limit but required a lot of waivers.
Dad and you need to go down to the rec center and talk to the manager about each activity. I bet you'll find that son can't just stroll into the weight room by himself, or get on a climbing wall, or even go into the pool, possibly.
And all this is without even thinking about the whole "stranger danger" aspect of it. That alone would make me be there with my kid. Dad needs to get some facts and not set a precedent with your son that it's OK for son to be there alone.
Will dad also be OK with dropping son at the mall soon too? That's not even allowed -- malls are private property and have rules that kids can't be dumped there alone, despite the fact that many parents do it.