M.P.
I would be more proactive. I'd have said, NO! we're going home and got in the car both with the lunch and at the house. No discussion. When we try to convince the other person, who is dense, we lose. Be firm and take action.
At the house, you could've gathered everything up and said, I'll wait for you in the car. We leave in 15 minutes and then if he doesn't show up, leave.
I suggest that because you ultimately go along with what he wants he's not feeling any reason to change his ways. You're throwing him under the bus because you're feeling powerless and angry. Change the dynamics by being assertive and insisting that he take notice.
Talk with him now, when everyone is calm. Tell him how you feel, using I statements, and how you're going to handle such situations from now on. Ask for his help in making this transition.
Later: It will help if you can stop labeling his behavior as selfish. Negative terms and way of thinking makes doing something positive more difficult. I suggest that he's not selfish. He has a different way of looking at things than you do. He's OK. You're OK. You're just different.