When your pet seems depressed or 'absent,' or is acting like it's in pain, constant panting, flinching, or other unusual behavior, and there's little hope of turning its situation around, it's time.
I don't know for sure if most dogs have a sense of dignity, but we had an Old English Sheep Dog who certainly did, and when her hind legs began to give out, we'd find her lying in a puddle of urine. She was obviously unhappy and uncomfortable, and seemed to feel ashamed, avoiding looking at our faces. Even when we spend hours comforting her and tending to her needs, she was depressed.
It became quickly obvious that there was little or nothing left of value for her. It was a hard and painful decision, but I'm convinced it was the right thing to do.
I wish we could extend that kindness to people. My grandmother suffered terribly in her final nine days, but she didn't have an advance directive, so we didn't have much choice but to let the doctors fight her approaching death. I hope I'll be able to opt out when I reach that point, or that my loved ones will fight for my right to die. I have established a DNR order and advance directive. I know that's not what you asked, but I have heard about some truly terrible situatiions among friends' families that could have been mitigated by an advance directive.