If it were me, I'd ask your aftercare provider to allow your son to bring a small, healthy snack to eat in the afternoon. Many providers won't furnish a snack, but if a child brings one they should allow them to eat it. It's unreasonable to expect a child to go six hours without eating and there might be child care legislation to support that.
To answer your question directly-- sandwiches with mayo need either a cold pack in an insulated bag OR should be frozen the night before and allowed to thaw out during the day. Otherwise, yes, you do risk food poisoning/spoilage.
I pack my son a half-sandwich (usually sun butter and jam) , a fruit option, and a small snack bag with a crunchy snack (chips, veggie booty) and almonds, plus a thermos with about a half-cup of milk. I should add that they allow a snack earlier in the morning, so he had a portion of trail mix then to tide him over. (They eat at 12:15)
Also, consider how excited and talkative he must have been to see you today. He may do better when he is focused more on eating-- I'm sure he was just tickled that you came to have lunch with him. And don't forget to ask about a snack at 3 or so. That's WAY too long for a kid not to eat.