Hi T.,
You've received some solid advice, so I'll just add one thing.
I recently saw a parenting speaker who talked about "teachable moments." Her advice for poor public behavior focused on setting up opportunities to teach children how to behave in certain situations.
She suggested picking a day when you don't actually have scheduled errands to run to teach. Go to the store when you don't need to buy anything. Let him practice being obedient under the threat of leaving the store immediately if he doesn't. When he pushes your buttons, pick him up and leave. He'll know you aren't joking, and you won't be leaving a cart full of groceries behind.
I haven't used this technique in public yet (I still confine my 2 year old to a cart or stroller for most trips), but I have done something similar at home. Following through on a big threat (for us...no stories at bedtime for running away from bedtime routine) once or twice was enough to get her attention.
I also have a friend who uses "sidewalk timeouts" with her son. He has to stand outside the store, restaurant, etc, facing the wall. He knows that everyone passing by can see him being punished, and it's enough to make him listen to his mom most days. Again, practice it on a day when you're not on a schedule.
The last piece of advice I've received that I'll pass on is to make sure your son touches you and makes eye contact with you when you give him instructions. Then have him repeat what you told him. "Yes, Mommy. I will stay next to you." It helps somewhat.
Best wishes!