In that situation I'd go, but we started leaving our kids with their grandparents as young as 1yo for overnights and 2-3yo for 3+days-long stays. and so they wouldn't have minded being left behind if it meant an awesome vacation with the other gps.
With four kids, we've done plenty of activities with two of them and sent two to do something else, we don't always expect to travel to every single thing together. I don't think your in-laws are being unreasonable in their expectations, there will be plenty of time for the 4yo to take part in things like that as he gets older. It will also be nice for the older ones to be "free" from their little brother for a while. I never thought about it as being unfair, just that they were ready for different things at different ages. Our kids tend to roll with the punches this way, though, I know not all kids do that; and I don't have anxiety about leaving them, never really have.
BUT.
If this is going to cause an extraordinary amount of anxiety on your part or your son's, I'm not sure leaving him behind is the best choice in your case. I wouldn't take him, I'd probably let the others go and take him to do something else while they were gone.