one thing it could be is that on you, your all nice and cuddly warm and then you put him down and the sheet is cold. try to put a blanket between you and the baby and then when you put him down take the blanket with you - that way he's still on his warm spot.
also, when he's on you, he's probably on his belly, and then when you put him down are you putting him on his back? i found it was always easier to put them down on their side. and i put a blanket roll behind them to keep from rolling on their back.
and are you still swaddling? when the baby was asleep on me, i would drape a blanket (with the corner down) across their shoulders. then when i went to lay them down, i took the bottom "wing" and brought it up, and then took the top wing brought it down and tucked it under. so that they were wrapped up.
one thing i can say, is that my son was my second too and i was too afraid to let him cry, for fear of waking my daughter up (their rooms are right next to each other) so i always rocked him to sleep. by 6 mos he was waking up every 1 - 1.5 hours all night long b/c he'd move position and then not know how to get himself back to sleep.
1 night of cry it out made a dramatic difference. altough 5 weeks may be a bit young... it's good practice. i'd say by 8-10 weeks he should be able to do it.