You allowed him extra time so he thought your time wasn't worth anything.
You sat and waited for him instead of going in and getting him and telling him off in front of his friends so he thought your time was not worth anything.
So, you have to teach him that your time is worth something. You have to teach him to respect you because he just showed you how much he doesn't respect you or value you.
Your problems are much deeper than this situation. I imagine this situation is just the tip of the iceberg.
You and him need to work through this where you both will learn boundaries.
My more huge learning experience happened in therapy with my daughter. We were in with the doc and she asked if she could walk to a fast food place and get some fries. I said no, that she was too young to go by herself. She kept bugging me and bugging me while I was trying to talk to the doc about important stuff.
Finally I let her go just to shut her up bugging me so I could talk without her constant "Can I please go...".
She went out in the lobby and sat down. The therapist asked me what had just happened. I was confused. My brain was mush from her badgering and I couldn't think straight. I was so angry she wouldn't just be quiet.
He told me the only way I was going to learn was to show me. He told me he had set this up. That he'd asked my daughter how many times out of a hundred could she change my mind even when I knew I was right if she just kept bugging me. She said over 80 times, at least 80 times out of a hundred should could manipulate me into letting her do something that was a bad choice if she just kept asking me if she could do it.
I stopped right then. From that point forward I told her one of three answers. Either yes, no, or I needed time to think about it and if she asked again the absolute answer would be no. She started hating me right then and there. But she stopped bugging me for the most part. She'd still test it now and then but then she would NOT get to go do the activity she was hoping for. That sucked for her so she'd sit by my and stare...her way of bugging me. I'd laugh inside because I could see she was frantically trying to remember to not speak to me again because she really really really wanted to go do this thing.
So this is what your son has learned. The more he manipulates you the more you're going to give in so he's got it down.
I can't say you did wrong or right. He was in someone else's house, where were the parents? If you'd have knocked on the door wouldn't they have answered and told your son it was time for him to go because his mom was there? Wouldn't they have gone to get him? If they were gone then why was your son there in the first place?