ADDED: Just saw the SWH. I get the frustration. Next time you try a museum that's kid-friendly, I'd divide and conquer: "SIL, I'm taking (my kids) to a particular exhibit they read about before we came. We will meet you and Nephews at 11:30 by the clock in the lobby." Then you nip off quickly with your own kids. They may protest that they want to be with Nephews but the kids just do not have to be together every second of every visit. I would carve out time where you and the kids go off and say firmly, and smiling, that you'll meet the others at a specific time later. Taking your kids to do something separate like that is also a great job for your husband and his brother, for instance, and you can spend time with MIL since you seem to like her. Plan it out in advance so you and husband are on the same page. Since your own son is 11, I also suggest you let husband and uncle take the nephews plus your son somewhere while you take the younger child somewhere more age-appropriate. Let dad go off and deal with the boys including your own older one so you don't see or hear what stresses you out anyway. Your younger child will complain but don't cave; find an appropriate but fun thing to do together.
Regarding hotels -- that would be the thing that drove me most crazy. If you are staying in the same hotels, I would arrange -- quietly, by calling the hotel before you ever get there -- to have your room on another floor or in another wing so your own kids don't hear the hallway shenanigans.
The restaurant story with the boy whipping other kids with part of a plant etc, and melting down like a toddler -- be prepared, if that happens again, to leave. Just say, "I'm sorry but it's clear Nephew is tired out, and it's time to get our own kids home." And go, swiftly and without more discussion. That level of scene in public, with a child as old as 10, is troubling, and I would just tell the parents that you're sorry but you feel Nephew might be calmer if the meal were over. Wow.....
ORIGINAL: It's possible that the mom/your SIL is much firmer at home than you see her being at these family outings. She might be afraid or embarrassed to give consequences on the spot, or maybe she's trying to act calm and neutral and not correct in public. You and I might not react like that but she might be more like my SIL, who goes very quiet when our niece acts up in public places. Yes, I'd prefer SIL was much firmer (and more consistent) but it's not up to me.
And I can tell you, your nephews do not care when you correct them, or even if you nicely ask them to stop doing something. Why? They only see you these four times a year, and they have no sense whatsoever that you have any authority to correct them. You may feel you do, because you are an adult and their older relative; I totally get that because it's MY gut feeling around our niece. But it does not mean the kid will ever acknowledge that anything you say is something he or she must actually heed. Can you see that? It's why I no longer try to direct or correct my niece. Not worth the breath wasted.
Has anyone considered that the locations and sight-seeing are not very engaging for these boys of 10 and 12? Think through where you have met and whether those were locations with things for these boys to actually DO in an active, hands-on way. I wonder if the locations chosen are for the adults rather than the kids, so the kids get bored and run around? While children (especially by age 10 to 12) should not have to be entertained all the time, and should be able to control themselves in public, I would look into doing something next time that would get the kids out of the adults' hair and give the kids more to do. Maybe meet somewhere that an adult (and there's usually one adult who would love to skip out on adult family time) can take the kids to play laser tag or something like that. Get the energy out.
And remember this: Those nephews could be great guys in a few years, or right now, and you won't know it. You only see them in a kind of artificial situation. I'd ask their parents what they're into, what their activities and interests are, and I'd ask the boys about it next time you see them. "Your mom said you made the travel soccer team--how many times a week do you play?" "I heard you really like science class; do you get to do experiments?" Etc. It won't make for long conversations but it might help you see beyond the behaviors that come out when they are dragged along on family outings.
Maybe it's also time to switch to meetings that are not about a destination and sight-seeing. Find a large park with good picnic facilities and bring a ton of game stuff with you. It's also MUCH easier to leave when you choose in that kind of situation than to get out of a day's planned activities at some special destination.
You mention the nephews as the source of your frustration. If you actually like SIL and/or MIL's company as adult women, arrange to do something with just them, no kids along, at another time. No reason to not have a good and grown-up relationship with adults in the family.