I hear ya, this is your live, not theirs. No, you did not handle this wrong. You are in the right, they need to respect your family’s privacy, parenting, and not get involved. THEY NEED A HOBBY! If you don’t put a stop to it now, they will continue to be overbearing.
I have been dealing with similar situations with my mother and her boyfriend for years!! They call all the time, complain when we do not answer (hello- we are at work and school.) They stop/drive by all the time (even when we are at home-being nosy, going through my items,. etc.) They expect you to drop everything to come over to help them with stuff they can do themselves (laundry, cutting grass etc.-even if we are at work or school they expect us to come over right after to help-even in the dark) Since my kids are older (21, 18) and moving on and away from home they have to know what is going on every second and they are now "lurking" around my oldest son’s house to find out what is going on!!! It is getting to be ridiculous!! They complain about everything we do or say. God forbid if you do something that they do not agree too, they will not talk to you for weeks (this is a good thing!) Even my neighbors avoid them as they are so nosy, they need to know everything in their lives too.