I think that it's terribly rude, and I say something when it's a small group. For my baby shower, some people brought extras. (I use the term RUDE, but I realize that I am likely making it extreme and polarizing, based on how it hits me. Actual rudeness--I think--is more about intent, and I don't ever assume that someone means to be rude.) I thought it rude, but the gathering was large enough for it not to feel so intrusive. I said nothing. When my friend brought her sister to my house for an intimate dinner for four that I had cooked and set the table for--among girlfriends--I said something. I told her that I hadn't prepared for five and she should have let me know ahead of time. (Since there were only four of us, I had set up the kitchen instead of the dining room. I made four dessert cups and used the four-setting dishes. I took a couple of topics off the agenda. Her sister dominated some of the more touchy conversation. I gave her my seat and dessert, and we moved to another room after dinner.)
Next time you plan something, say pointedly to the group that this gathering will include exactly X number of guests. Have the table set for exactly that number, with no room for more. Make it uncomfortable and crowded for more to join. Do this a couple of times to show that you're serious.
It's rude for a number of reasons. In addition to the obvious ones involving preparation, it affects the dynamics of the group, the level of intimacy. These people feel like intruders, and it can cause resentment. Socially, I don't like surprises, and I don't divulge certain information with unvetted ears and hearts around. When you get together with your small circle, you want to let your hair down in a way that you don't do with just anybody. You might even have an agenda of things to discuss that you haven't been able to open up about. That gets disrupted when "outsiders" are there.
I've had someone close to me bring new people around and then open topics that are personal for me. "Oh, you can talk around her." Yeah, but I don't want to. YOU know her. YOU have vetted her, and she meets whatever standards you have in place. Not so for me. I have actually said, "No, she doesn't know me like that."
I haven't had anybody do it when it was supposed to be just the two of us.
Oh, did I answer your question? My vote: RUDE.
ETA: After reading JessicaWessica's reply, I'm trying to imagine someone inviting himself/herself along. I've never been that presumptuous, even with close friends. Seriously, people just think that it's okay to just invite themselves to an intimate event? Not a concert, but a dinner? Huh?