Dealing with issues with family members often feels very difficult because of the years of history (and baggage?) and expectations, but what would you do if this were your friend or neighbor's child who came over to play? Wouldn't you talk to the mom and say, "Hey, when little Johnny was here today he was having a hard time sharing/playing fairly/getting too rowdy, etc. Normally when my kids get that way, I do X (fill in the way you handle your own kids). If this situation comes up again, are you okay with me doing X, or do you have a different way you'd like me to try dealing with it?" That way you alert the other mom to the fact that there's been a problem, you make it clear to her your expectations for the kids, and she gets an understanding for how you plan to handle problems when they come up. But it also gives her the chance for input, and it makes the process more of a collaboration between the two of you to solve the problem. AND it keeps you both from assuming that you handle problems in the same manner since you were presumably brought up the same way by the same parents.
I would recommend talking to your sister like she were simply your neighbor. Try to get away from pointing fingers or placing blame or trying to figure out WHY your nephew does what he does (even if you don't come out and say those things to her, you are probably thinking them), and simply focus on what practical steps you guys can take to ensure that the cousins can play well together. After you have that first conversation, the next time the kids play, let your sister know if there were any problems, how you dealt with them based on what you two agreed, and what the result was. If the agreed upon approach didn't seem to work, hit her up for new suggestions. And if your nephew is truly out of control and nothing you try gets compliance on his part, let him know that if he can't follow the rules and behave appropriately to you or your kids, then you'll have to call his mom to come get him. Then follow through with the call if you have to. Chances are it will only take once for him to see you mean what you say.
A couple more practical suggestions: If there are certain house rules that are different for your kids, ask your sister if she can remind your nephew about Aunt S.'s rules before he comes over (keep it to one or two and only focus on the ones that really cause problems), and then you remind him too once he gets to your house ("Little Johnny, don't forget that at our house we don't jump on the furniture," for example). Sometimes all a kid needs is a reminder up front.
Also, another mom pointed out that you should teach your daughter how to stand up to her cousin--and that is so true! Role play with your daughter ways she can respond to her cousin in common play scenarios, and if you know of a particular situation that they have trouble with, role play with both of them. Maybe your nephew is still unsure how to ask to join in or how to negotiate in deciding what and how to play. For example, on more than one occasion my son's friend just started crashing his toy car into my son's. That started a fight the first time it happened, but when I asked the boy why he did it he said he just wanted to crash the cars (I could tell from the way he answered that he wasn't trying to be mean or provocative), so I explained how he needed to ask first, "Hey, do you want to crash our cars together?" instead of assuming that my son wanted to play the same way he did, and if the answer was no he needed to say okay and do something else. I still have to remind him to ask first though from time to time.
It may also be a good idea for you to play some with them--at least until things get better. Play a game together and help model turn-taking and fair play. Work on a collaborative project that requires some teamwork like building a fort or baking cookies. Maybe your nephew isn't getting the guidance or social practice or attention or discipline he needs at home, but you are his aunt, and it's great when cousins can be friends for life, so maybe you can find gentle, loving ways that you can make positive changes in his life.
Good luck!