Sounds like he feels the lack of personal time with you. He sees you giving "good energy" in the form of kind words and hugs to the other kids, and he sees that you are often very busy when he would like some attention. So, perhaps you can arrange some one-on-one time with him to have a little fun together? Even on a regular basis? Easy to say, hard to do. Maybe he feels he is not important to you. If you can give him extra hug time during the workday (perhaps a near-impossibility!)he might feel that he is indeed important and loved by you.
I had trouble with my kids when I taught pre-school music classes that they attended. I felt I shouldn't give them extra attention because that would be showing favoritism. Each child's behavior went from less-than-desirable to downright hidous. My son could never explain his feelings, but my daughter was able to articulate it at age four. She said she secretly wished that mommy was there just for her, but that she knew it was wrong to feel that way.
I wish I could say that I found a good solution, but I didn't really. Looking back, I wish I had given each of them *more* attention, not less; i.e., holding his hand during circle time, putting her on my lap, etc.
Also, by the time I got home and was fixing dinner, etc., I was more grouchy. So, they saw me at my best when I was relating to the other kids (I was getting paid to do that, after all), and at my worst when I was relating to them at home. I'm not proud of it, but they did turn out very well in the end - they're in 10th and 12th grade now, and really good kids. I don't deserve it!
So, my final advice is to give him MORE attention each day, before he starts to act up (so you don't reward bad behavior). It's very hard to share mommy. Imagine how you'd feel if you saw your husband flirting with beautiful women all day every day. You'd want to pinch him - and perhaps yourself - too! Best wishes . . .