J., I am right there with you. And yes, I understand the clinical definitions of narcissist. My ex-husband sat in counseling with me and proved it to the therapist himself every time he opened his mouth and blamed me for all our problems. (She told me this later when she counselled me through our divorce.)
From my experience, "The point" is whatever makes them right. If it's 'not the point', that's was shorthand for "don't call me on my own sh*t" or " that's not what I meant to pontificate on" (and to appear 'right' in doing so). I could site real conversations we had when all logic flew out his window because it wasn't "the point" of what he was saying. Don't want to visit my former in-laws because his brother is a sex offender? "That's not the point. It's meant to be a family time and you have a bad attitude about it."
It's a device to place blame on others and to ensure rightness on any absolutely riduculous position they choose. When most people I know tell me that "it's not the point", I am willing to open my mind and backtrack and try to see where they're coming from. With my ex, it was "okay, I see that the conversation's pretty much over now and I'm still going to be wrong, even if God sends down a memo carved in stone telling you differently..."
Added: I'm really quite surprised at all the derisive comments about J. putting the Narcissist label out there. For those of us who do have to deal with bona-fide narcissists, it can be incredibly painful. Perhaps they are rare, but I had three of them prominently in my life: my ex-husband, my stepfather (for six years) and my own birth mother, who has Personality Disorder with tendencies toward Narcissism and who falls into a class of Borderline Personality Disorder which that is nothing short of terrifying at times. I'm not being dramatic in describing that. Please don't insult someone's experience or descriptions unless you know that they are, indeed, off base. Some of us do know, all too well, what we are talking about.