The one bit I would add is don't make a big deal out of departures and returns (even with his dad). When you leave, tell him you're going, give him a quick kiss or pat, tell him how long until you get back and then leave. When you come back say hello, then put down your things, do something else (go to the bathroom, whatever) then give him cuddles and attention. You might try giving him a clock or a timer and showing him when you'll be back though he's a little young for that.
And small slow steps. If he does cry when you come back don't overcomfort him. Distract him with fun things to do, quick pat, quick hug then treat him as if leaving is no big deal.
Try to find one trusted person to really bond him too.
But also understand that some kids are just wired that way and it takes them a really long time to get over it. But you should encourage them as much as possible. Talk about being a big boy and how good he is. Also talk to him about it ahead of time, tell him what he's going to do, what you're going to do (make his sound like more fun) and how long you'll be gone, and what you'll do together when you get back.