He got really sick if he ended up having a kidney infection. That's rather serious for his age. He sounds completely exhausted, from the chronic sicknesses complicated by an infection. Poor little guys is probably sick and tired of being sick and tired. I would try to schedule down time...like I would not have had lunch with mom out and about with a tired, sick little one tagging along.
One big difference with your 6 yr. old at 4 yrs of age, he did not have an older sibling to keep up with. Your 4 yr. old most certainly looks up to his older brother and wants to do what he is doing, but is not developmentally as ready, so pushes himself. And that will go on and on.
Did your son complete the full course of his antibiotics and meds from the kidney infection? Were there any reaction there?
His lethargy is nothing to dismiss yet. If it continues, be sure to discuss with your pediatrician. But make sure he has plenty of time to rest and heal still.