Hi N.!
Good luck with this. First, keep in mind that kids outgrow every stage, as mentioned by some others. It always seems like it will never end, when you are in the middle of it! But it always does.
I ‘d sure be careful about the advice to swat a kid while telling them you love them, and while taking them from the place they feel is safe, back to the place they feel isn’t safe. Wow, what a bunch of mixed messages – that kid is learning that love hurts, and abandonment is what you get when you honestly ask for comfort. Those kinds of early perceptions tend to color later ideas about love and relationships, and can haunt people throughout adulthood.
Just because a kid follows orders and doesn’t come back into the parent’s room at night when told not to, or threatened with a “swat”, doesn’t mean they are "over it." It can often simply mean that they are just getting better at stuffing their feelings. What happens to a little boy who gets good at this, and “behaves”? He grows up, gets married, and then his wife says, “why are you so disconnected with your feelings?” Well, he learned it a long time ago - so long ago that he doesn't even remember what it was like to really be in touch with his feelings.
Raising well-adjusted kids just takes the ability to put ourselves in their shoes. And they are hardly able to “manipulate” at that age. They don’t learn that until we teach it to them.
Plenty of people choose not to include their kids in their bed, and that is certainly understandable, but it is not natural. So kids who want to be in their parents room at night are just responding to their instinct – in prehistoric times, staying close to parents while sleeping could mean the difference between life and death for a child. And just because a society changes its ideas about that, doesn’t change the deep seated instinct.
However, here is a practical solution. Blue light is calming and comforting, and helps promote restful sleep. Try putting a blue lightbulb in the nightlight in your son's bedroom, it works wonders. (Or cover a dim lamp with blue cellophane, or blue theater gels.)
(I'm mom of 8 grown kids, and a bunch o' grand younguns)