I don't think you're being silly at all, but I also hope you won't let things from your past and your own boundary issues make this more than it needs to be, especially with someone who is working so closely with your son.
I doubt she meant any harm by this whatsoever and her first thought was to keep her eye on your son at all times...even though he'd entered your own home. Not knowing you well, she basically had a 50/50 chance of handling it right. You could have been upset with her that she just allowed him to come in while she stood outside.
I would just talk to her. I'm wondering though, why you would feel violated by someone you entrusted your son with. She might wonder that too. I mean, violated is a pretty strong word.
You can tell her that you are uncomfortable with people that you don't know well coming in. You can tell her that she needs to wait outside until you respond to her. You can tell her to let your son go in the house on his own and she can leave.
This is something you have to communicate with her or she has no way of knowing how dropping him off works according to your comfort level.
This is just my opinion and don't take it the wrong way, but you basically need to rely on your son being comfortable with the people who work with him. I think it would be easier for him if you could show an amount of comfort with it as well. Feeling violated when you need people to be pretty closely involved with your son is something you have to work on for his best interests.
Long story, but my daughter developed very close relationships with "professionals" in her life and she saw them as friends and part of her family to an extent. My comfort level with them really helped her. It was tough in the beginning because many I dealt with were only due to the fact that her father and grandparents tried to prove me unfit. My life became an open book and although it was not easy, they knew that none of my daughter's issues stemmed from any of my own issues. They were able to see that for themselves.
I had to let go of feeling like my every move was being judged, because in the end, all the professionals were allies, not just for myself, but for my daughter. And, I was definitely found to be a fit mother.
I would just be open about your comfort levels so that you don't come off in a way that looks like you have something to hide. Which you don't.
You might need to talk to someone about your feelings of violation so you can get past that too.
They are valid feelings.
But, they most likely weren't intended.
If she's great and your son likes her, that's a plus.
Be open and honest.
Best wishes.