I do say go with your gut and keep the kids away from any solo outings, but I also would echo what Jo says in the very first response: Could it possibly be that he has Asperger's or a disability or some other condition where he is socially awkward? Whether due to a condition or not, there are adults who don't read social cues well at all (for instance, not taking any hints about "personal space" though you keep moving away from them), or who prefer kids to adults in a group of both kids and adults, because kids just don't notice the awkwardness.
I want to say I definitely agree that you should go with your gut and absolutely should not have your kids around him alone if you have even a hint of a concern, but it's just something to consider. The vibe may not be (let's just say it out loud because no one has yet) a "child abuser" vibe but could be a matter of his being "off" or "weird" or awkward due to a mental/physical condition, or due to his upbringing, even.
I once went on an all-day church outing with a large group where I didn't know everyone, and felt all day that one guy was "off" but later found out that he was slightly developmentally disabled, so his responses were slow and sometimes he said things that didn't entirely make sense in the moment. But it was a function of his mild disability.
If you say to your sister "He's off and I won't leave my kids with him ever" she likely will get furious and that's the end of any talking to her. She'll get defensive and think you are questioning her judgement in men and her choices (which really you are -- but for good reason).
But perhaps you can consider approaching her along the lines of, "Is Sam OK? I've noticed that he sometimes seems awkward around people. Does he have anything going on that make it hard for him to read social cues?" And so on. That way you're making her feel that you're concerned for him (and by extension, for her) and you might have a better chance at finding out something without alienating and angering her.