No, I don't think you're worrying too much - it's a broken bone and at his age, while the bones are still soft - if it doesn't heal or set right - it will just lead to problems down the road...
However, I think you SHOULD listen to the Dr and just keep an eye on him...we now know he has a high tolerance for pain as he was walking on it over the weekend....YIKES!!!! If it hurts too much - he won't walk on it so this is a good thing! :) he MIGHT mellow out! (HAHAHA!! One could only hope!!) So bottom line? I think it will be like with the splint - he will adapt, improvise and overcome - I think he's gonna be a Marine mama!!! :)