D.C.
No, don't worry. Their appetites can vary day to day just like ours can. My daughter is 6 1/2 months old. She normally takes around 6 oz of expressed breast milk at lunchtime; today she only took around 3 1/2 oz.
If he's happy, alert, and has lots of wet diapers, he's doing just fine.
It can also depend on his size; a 12 lb baby isn't going to eat as much as a 17 lb baby, even if they are both 3 months old. The lactation consultant I saw said the amount they take can vary by their weight. Take the # of lbs your son weighs, multiply it by 2.5, and divide that total by the # of feedings he takes in a day.