I would talk to your son, privately and see why he feels that way about your Husband.
Next, you have been working for 1 year already... so any morning "routine" should be routine already. Your son knows that.
Maybe its normal developmental stuff, and maybe its not.
For me, ONCE in a while, my daughter would say that... but it was ONLY when she had been scolded about something. And then she would explain to me why she didn't want to "see" Daddy.
Then we all talked about it, Daddy included. And then it was fine.
Or maybe your son misses you, and rather be with you. Kids do have "preferences" sometimes, and at certain ages.
Or, do you know what your son's day is like, at home with Daddy? Maybe you should find out.
What do they do? Does your son get enough attention? Or is he scolded all the time? Or is he left to play by himself more often than not? Or, is your husband different with your son, when you are not home? Does your son and his Dad basically get along well and "bond" well??? Or not? He is after all 3.5 years old... and BY NOW he should be "warmed up" to Daddy.
Or, maybe your son knows that once he goes to see Daddy in the morning, that it means you have to get ready for work... and he doesn't want to see you leave.
Its hard to figure out... unless maybe you see what your son says himself. But at this age, depending on their communication level, he may or may not be super articulate about it. But TEACH him, that he CAN express himself openly, without 'fear' of anything. Boys NEED to learn HOW to express themselves.... without fear of being scolded or criticized.
All the best,
Susan