If that is the intuition you have about it, then listen to it.
Not all 7 year olds, "know" how to play with a toddler or baby or younger child. And you CANNOT assume that they 'are" responsible enough. They are not.
The age difference, to me, is too much. Unless the 7 year old was his own sibling... but still. A 7 year old and a 2 year old do NOT have the same developmental abilities. AND, your 2 year old "cannot" defend himself nor know fully what is right/wrong or fair or not.
AND you would not want your 2 year old to pick up bad/negative habits, from t he 7 year old. Since you feel, that the 7 year old is not appropriate.
CERTAINLY, there should ALWAYS be direct supervision, IF the 7 year old comes over. Never left alone with your son, even for a second. Accidents can happen.
Listen to your red-flags about it. Never mind about just letting it happen out of being "polite" or social pressures. YOUR son's safety and development comes first.
All the best,
Susan