We have a 3 1/2 yo son who did the same thing at times in the past, but not always...just on occasion he insisted he did not need to go. One thing which worked on particularly tough days is when I would tell him that the buzzer will go off and when it dings, it's time to go! I acted like it's no big deal, and also told him that it's not ME who wants him to go, but it's absolutely necessary when the dinger goes...for some reason this works. It's like it lets him off the hook from having to obey mom, and he can maintain his independent decision-making.
He obviously hasn't associated it yet with mom being the person setting the timer! But I say whatever works!
No guarantees, but it's worth a try...good luck!