I also think it's just a descriptive term. It can be difficult to talk about skin colors with children. I've tried to get my kids realizing that there is a huge variety of skin colors and tones in the world, just like there is difference in hair colors and stuff. It's hard, though -- sometimes I'll draw attention to it on pictures in books or the internet (mostly around here we have "black" and "white" so not a huge variety), and I try not to use the terms "black" and "white" because everyone is a shade of brown, except for a few extremely dark people and albinos who are truly white. My skin isn't white -- it's sort of peach colored for lack of a better word. Right now my arms are tanned so they are more brown. The funny thing is, my son's favorite "Little People" character is and always has been Michael, the black boy; and when we play Chutes & Ladders, he always chooses the black boy as his game-piece. I point out that his skin is darker brown and say, "People have skin that's all different colors of brown."
I don't want to implant some idea about major differences, though, and make something in his brain that wasn't there before. Like telling kids the first time they go to the dentist, "This isn't going to hurt," may make them say, "Why are they saying that?? They're lying!! It's going to hurt! I'm running away!!" That sort of thing. So, I don't want him to think that people are really that different, just based on skin color, because we're all the same inside.
Sometimes, my son will say something about "that brown boy" he played with at the park or church camp or whatever; if I know the kid's name, I'll use it, because I assume he used the descriptive term since he didn't know the name.
But I think the biggest change that has to happen is in the adults' brains, and that is to realize that we are all basically the same. We're all related. Just as in some families, there can be a wide variety of skin tone and hair color (I know a family with 11 children, and the hair colors include red, blond, light brown and dark brown; and skin tones range from Mediterranean [the grandfather was Spanish] to very pale), even so, we're the human family, and have a wide variety of skin tones, hair color, nose shapes, eye shapes, heights, weights, etc. We can let that divide us, or unite us. A lot of the world's problems would go away if we weren't so focused on outward differences.