All three of mine have done the same thing at about the same age. I've never forced them to go to someone else, I never left them to tough it out. They're all three turning out pretty awesome. :) Not a single one is permanently attached to me; nor are they afraid of new situations or unable to function without me standing watch.
I don't think anything he does now is foreshadowing of future issues. He's telling you he needs to be near you, so listen. :) You're obviously a devoted, loving momma & he loves to be with you. Don't leave him. It DOES NOT mean he's going to grow up attached to you & incapable of independence. It means he'll have the confidence to gain that independence when he's ready, knowing that his momma will help him when he needs it. You're like the net at the bottom of the trapeze; he knows it's there, so it's easier for him to be brave about swinging out those first few times. :) Ofcourse, he's only 4 months old now, so that's still a bit far off. For now, all he needs is snuggles & lovins.
Have you thought about getting a wrap & wearing him? Babies do wonderfully, snuggled next to mom & dad. Infact, that might be a great way for him to do better with his dad. Wrap them up together! The Moby Wrap is great for dads, as it's just a long piece of fabric that can be easily wrapped to fit whomever is wearing it.
So, don't worry about it & don't think there's anything to you have to do besides listen to your little man.