Yes, it's probably worms, but no you do NOT need to be freaking out. It is highly unlikely that your son is in any danger. I have, unfortunately, endured both tapeworm AND pinworm, and I can assure you that it's going to be okay.
Tapeworm - do you have a dog or cat? We found tapeworm moving in my son's poopy diaper just after his first birthday. The P.A. at the Dr.'s office insisted that it could NOT be tapeworm, and treated our entire family for pinworm THREE times before I took a specimen to the veterinarian for diagnosis. Tapeworm is self-limiting - eventually the worm itself will die and pass through. The "worms" that are seen are actually living egg-sacs. Those egg-sacs must be eaten by an intermediate host such as a mouse or a flea. Animals with fleas re-infect themselves by biting at their fleas. The egg-sacs themselves cannot infect them or you. My son used to pull at the hair of our long-hair cat, and of course would get some in his mouth. Although we had never seen any fleas, apparently he managed to get one. Certain tapeworms can be dangerous, but regular canine tapeworm (yep even in a cat it's still called canine tapeworm) is not one of them.
Pinworms - Has he been sleeping well? Typically, pinworms leave the anus at night to lay eggs in the surrounding skin, causing intense itching. Infection is caused by ingesting the eggs. Scratching gets the eggs on the fingers, and they go everywhere from there. Has your daughter been itching as well? That would be a big clue, and your entire family will need to be treated as well.
I have so much more information for you, but every time I try to send it AOL bumps me offline, so please send me a message if you'd like more info.
It could be just a thread, by the way. Good luck, and let me know and I hope this goes through this time.