Transition to Crib for Bedtime & Naps

Hi mommies,
I just need advice or support in helping my 3.5 mo old son transition to the crib in his room. My son has been a great sleeper. He has been sleeping for 7 -8 hours a night from about 10pm - 6am in our travel swing, but NOT turned on. For naps he has been sleeping in the swing but it needs to be turned on. Because he has been sleeping so well at night we decided to move him to his room in his crib. Starting four days ago.
The first day, I just did bedtime he slept from 8-11 and then woke and didn't want anything to do with being put in the crib so we took him back to our room.
Second night, I moved the cover from the swing and laid it in the crib. I follwed our usual routine of swaddle & nurse and put him in and he slept 7 hours.
Thrid day, I tried naps and he took 3 forty min naps. Then for bed time I follwed same routine swaddle and nurse put him in and he was awake YEAH and he fell asleep on his own but only slept 5 hours.
So just wondering should I keep with it or is he to young. I know that having the swing cover in the crib isn't the best but I'm hoping it will help the transition. I hope he will get used to being in the crib and won't need it soon . It has just been difficult because I'm waking up again and it's rough with getting little sleep. I know he does so well in the swing in our room should I just wait or if I wait will it be harder, so just tough it out now. We are also working with moving his bedtime earlier he doesn't really want to be up until 10 now.
Any advice is helpful.