M.C.
Hello L. -
LET HIM SLEEP!!
When my son (6mo now) was still calm enough to sit in his swing, he was in it all the time. I put him in just to stretch my arms for a minute, and he fell asleep. So I left him there. He needs sleep and food right now. I say as long as he sleeps in his crib at night, let him sleep in the swing all day. I did. You'll know when he is ready to sleep in his crib in the afternoons. He will not fall asleep in his swing, because he will most likely need more room to move around.
M.