The intimacy with your husband is REALLY important, and you don't want the child to be the reason that intimacy is not happening. There are so many parents of K through 2nd grade parents at our child's school who are divorced and you have to wonder if this co-sleeping thing sends the marriage relationship down a slippery slope. Obviously that is not the reason for all divorce, but clearly when children come into the relationship, these sorts of stresses occur. The marriage needs to be a priority as well - and it's good for the child if it is.
I'm a step-parent so I had to be the "bad guy" when it came to setting limits with the child in the adult bed. My step-son was 7 years old and still sleeping in his dad's bed whenever he could (both dad and son liked it). But it didn't leave much room for a new marriage.
It seems to me that parents think that they are doing a good thing for the child by keeping them in the adult bed and letting them think that it is their right to be there. I think that having that as a "treat" is a good thing once in a while... but not as an expectation. It's not healthy for the whole family if the kids rule the bedroom and sleeping arrangement decisions.
And no intimacy in the relationship... one partner is bound to get resentful, and you really don't want that happening.
The natural course of things is that kids should sleep in their own beds. If they are sick, or have some special circumstance or want to cuddle in the morning in the big bed... that's fine. But the expectation is that kids sleep in their own bed. It's good for the kids (even if they say they don't like it) and good for the marriage and intimacy.
My parents never once that I can remember allowed any of their kids to stay in their bedroom ever. I always hated that. Even if I woke up with a nightmare at a really young age I was sent back to my bed within about 10 minutes. I think that's a little extreme. And of course now I'm a fiercely independent adult. Do some cuddling and make some exceptions to the rule. I also recently learned in a conversation with my mom that my parents had sex almost every night. So that explains it. And it's good for the marriage to have that - especially if it prevents divorce.
Sorry if this sounds like a lecture. I just have some strong feelings on the topic. And you said that your relationship was feeling "cold." That's a terrible sign, and you really should be more alarmed by that then you seem to be. You should be checking in with your husband immediately and finding out what he would like to happen. Because a "cold" relationship will lead to things that are not good - and your child will not want divorced parents. Don't put the child's desires over what is healthy for the marriage 100% of the time, or you will no longer have a marriage.