Marda and Diane are spot on. Like most things in life, choose to focus on your son's actions, not the words.
(Think about it this way-- how many people have you met in your life who will easily say "I love you"-- and then turn around and act in ways which do not support their statement?)
My son is five and is still very much like this. He's all over his buddies, professing his love, and only once in a while lobs an "I love you" over to me. "Show me with your actions" is a big phrase in our house, and I do actually get lots of real, physical love and affection from him. You will do well if you choose to focus on how *he* expresses affection. With boys, it is often very different from the more verbal way girls have. This is due to communication styles.
And don't get too complicated or overthink it. Just assume that he loves you unless he tells you otherwise.:)