B.M.
That is kinda weird. This might not be it at all but how is his neck? When he's in your arms or laying down or is even held out under his armpits does he favor one side? Does his head turn one way most of the time? Some babies early on can develop torticollis. This is when their neck muscles on one side get tight. So they have a hard time turning their head to that side. This causes them to not want to nurse on that side of the breast because it hurts the neck. If he does have these signs just call your doc. They will refer you to a PT. They will just need to do exercises and stretches on the neck to relax and loosen those muscles.
My other thought is maybe you don't make enough milk on that side? Have you tried feeding him from the left breast first, then once he's sucking starts to slow down and he's getting somewhat full switch him to the right side. Maybe he just gets frustrated on the right side because he's hungry and it just doesn't come out as fast as the left side?