I'm with you and personally, if it were me, I would probably put my foot down and make the consequences for unauthorized use of the dirt bike severe. However, I like the other poster's response to sign him up for lessons or safety classes. Also, set some rules or guidelines--location for where and when he can ride, helmet useage, who he can ride with, etc. That is probably a more practical and realistic approach than the one I would want to take.
Here's my background on the situation--my husband is a product liability attorney for a firm representing manufacturers of ATVs, motorcyles, boats, etc. We hear the worst stories about those type of vehicles. However, the majority of accidents involve misuse or innappropriate use. That said, to my family's dismay, we don't allow our children to ride ATVs at the family cabin. Under the manufacturer's rules they are too young, but most people don't follow the manufacturer's guidelines and that's when tragedy happens. Our kids are allowed to drive golf carts.
My brother was also involved in a dirt bike accident as a kid. Somehow the bike slid out from under him and landed on top of him when he fell. The bike sliced through his calf, severing his tendon and requiring surgery. Fortunately, he was with a friend at the time who was able to quickly get help. My brother has also rescued people 2x from ATV accidents. As an adult, he is an avid ATV and snowmobile rider, but doesn't have anything to do with motorcycles.
The bottom line is they can be safe and fun if used properly, with proper equipment and proper training. I'm with you though--they are scary. Good luck.