You have gotten some great advice about his age and stage and how this does turn up at around five; he's aware that a change (kindergarten)is coming and is acting out possibly because of that, and/or not wanting to grow up just yet, and/or needing to assert his wishes and not yet knowing quite how to do it (despite your surely telling him to use words and not hands).
I would add one thing -- time outs in his room should end. You did remove the toys, and that's excellent, but time outs in a child's own room can do one of two different things: : They make their room, which should be their own special place, associated with unpleasant things or, on the opposite end, they have little effect because the child has his own fun or comforting stuff around him. Removing the toys helps with the latter but instead, I would have his time outs in a very boring, neutral place that does not have any view of the TV, or of any of his stuff, or anything interesting - the end of a hallway (doors into rooms shut) or the bottom step of the stairs are some places that parents use. He should be in an area where there is...nothing even to look at. That way he feels the time out more. And be sure that when the time out is over, a required part of the time out is that you get down on his level and he tells you why he was in time out, what he did, and he apologizes to you. (This step can also help bring out the emotions that may be underlying this and can help you work on those with him.)
Also -- it's hard to do this when the hits are outside the house, but be sure that the consequence follows immediately. Sounds like you are doing that already. Regarding the t-ball game, I would not take that away unless he hits that day or the day before and it's crystal clear -- in advance -- that the t-ball game is on the line as a consequence. I would tell him the day of the game, "If you hit anyone today, you are not going to the game," rather than waiting until he hits and then saying, "That's it, no game." He needs the warning and the knowledge of what is at risk -- but in a kid his age, that warning won't stick very long in the heat of the moment if he's mad. I would wager that you are doing all this already, though!
This will change. Stick to your very good ideas about keeping your own temper under control (so hard to do, I know!) and ensuring he feels he can talk to you, in case the hitting is a result of feelings he doesn't understand yet, like fear of starting K.