A. make sure there actually are appropriate activities, toys, people to talk to, etc. to keep him engaged. Any person would act out if they are bored or being ignored.
B. Be willing to follow through with significant consequences at the FIRST SIGN of inappropriate behavior. Removing him from the situation may be required, so if you are not willing to do that, you may not have much success.
1) The next time you attend a family get together, just before you get there, such as in the car, talk to your child. Get down on his eye level, make sure he is listening to you, and tell him calmly and clearly what you expect: "I want you to be respectful of other people. There is no running in the house. There is no hitting other people." Tell him the consequence for misbehavior. "If you ..." (a suggestion is to have him sit outside for five minutes.) End with a positive affirmation: "I know that you can have self control in there. Let's go to it!" This should take five minutes.
2) The VERY FIRST TIME he violates the expectations you have clearly explained, put the consequence into place. Do not give him a pass, or else he will learn that you don't mean what you said, and so of course this will encourage him to continue ignoring what you said and misbehaving. Tell him why you are instituting the consequence: "I told you ... you just... this is wrong. If you cannot respect other people, you will have to be alone for a while." Do not get into a discussion justifying your actions. Just explain and institute.
3) Make sure the consequence is short enough so that he has the opportunity to try again. For example, if you have him sit outside for five minutes, then after five minutes, speak pleasantly to him and say, "I had you sit outside here by yourself because you were not being respectful to other people. What is it that you did wrong?" Have THEM explain their offense so he has to take responsibility for it and so you both know he clearly understand right behavior. Then say, "I believe that you can be successful here. Are you ready to try again?" If he says yes, give him another chance. If not, let him remain isolated.
This process does take an investment of your own time and energy. It may not be your favorite thing to do while visiting family, but once you make this investment, it will pay off. Sometimes what I do is practice at home beforehand. Say, "Let's pretend..." Practice the appropriate behavior. It really helps kids remember in the heat of the moment.
Hope this helps.