Hi M., I am mom to 3, grown and gma to 6. I had one who did this, We sat down and talked. I told him how much I loved him, but that him being in our bed at night was so uncomfortable for me that I was not sleeping and actually, my body was aching. He stopped coming in!
Now in your case, he is very used to doing this, you have let this go on way tot long, so, he may need a reminder, a physical one.
Buy a gate for his room and put it in the doorway when he goes to sleep. Tell him it is a reminder that he is to return to bed when he walks up to the gate. I figure that he is a pretty sound sleeper.
If this doesn't work, take a week off of work, and take him back to bed every time he comes in. I think if you do this for a week, it should stop. If it doesn't, you can get more direct, by removing a toy from his room every morning that he still comes to your room. I really don't think you have to resort to that.
Also, I was thinking of a monitor that beeps, when he gets out of bed, reminding him to return, and waking him a little. Good luck. Problems with children sometimes take perseverance, lots of love, and understanding.