My first question would be = how long is Daddy gone? If it’s only a few nights, I see no harm.
If it’s more than a week, your son may become adapted to your bed, and harder to get back in his own.
We're a co-sleeping family but my kids now sleep in their own beds, in their own rooms. My youngest is 4. When Dad is out of town I absolutely let him sleep with me if he wants to. He sees it as a super treat that he gets to sleep with me. Our routine continues as far as talking and reading before falling asleep. I wouldn't ever change this "tradition" we have.
My kids know full well that once Dad gets home, its back in their own beds. Never a problem.
My daughter is 12, and STILL wants to sleep with me when she is sick. I love it that we are so close. If either of my kids are sick, they sleep with me and my DH encourages it.
By the way in reading your post wherein you said “I felt a small little body hugged up against me”…. I just smiled! That must have been such a wonderful feeling!!
You know your son best. What works for one family, may not for you.