What the heck is "semi-supervised"?
Well, for sure it translates as "not right for a child this young."
Does Dad really understand that a child this age needs supervision on a real basis -- not a "semi" basis?
Maybe Dad was raised in a casual, free-for-all place where kids roamed free all summer so he's picturing the kids having a blast. And they would. But do you really want your child coming home each night and you're not there? Would he be around water? Allowed...to do what, exactly? How old are these other kids? Is the expectation that the other kids are going to watch the younger ones like him? (Recipe for disaster, by the way, especially if you do not know the older kids VERY well.) Does your son need his routines at bedtime, which will fall apart if you are not around? Will he be overtired all day, allowed to stay up later than you know is good for him, allowed to eat whatever, whenever?
This sounds like a no-brainer to me. If he goes, you go too. If Dad is all in favor of this, you need to sit Dad down and go over what your son is like when he lacks his sleep; when he eats certain stuff; when he doesn't listen to YOU (no four-year-old listens well all the time--is he really going to obey these other adults when they tell him "No, don't do that"??). You are not overprotective; Dad is overly rosy in his outlook on what larks they'll all have.
Oh, and you do know that a four-year-old is going to get homesick, right? We go overseas with our daughter every single year. At that age she loved the trip but also said after the first few days, every time, "I miss my room and my toys." If we had not been there with her, I feel sure she would have been miserable at that age without one of us around, especially at bedtime. Kids need to feel secure. Does your son know his hosts so very, very well that you are certain he will feel secure being semi-supervised?