I actually have a similar issue with my 4-year-old son and 2-year-old daughter. My son will often hit his sister. Usually, she hits back. I have to agree that one threat that seems to settle things down is what P.O. mentioned. He likes playing with his sister, even if he does hit her and pick on her. So the idea of being separated will make him settle down, at least for a while...
Actually, my son will be 5 in April. Another thing that I find is that as he is getting older, he is getting better. He is better about controlling his anger and can be reasoned with more. Often if I'm paying closer attention, I can head off the problem before it becomes a big situation.
I think things have been just a bit worse lately because we've been inside with the weather and because I'm expecting our third baby within the next 2 weeks. So I've been focusing a bit on getting stuff ready for the baby and trying to tie up other loose ends before the baby is born. This means that I haven't been paying as much attention to him, so he then acts out to get my attention.
Basically, I think what you are doing is right: removing him from the situation so he can settle down. Then if in addition, you can spend just a little bit more one-on-one time with him. In time, it should get better!