I would do my best to convince her that maybe this is some time that she may have a few choices of what to do now....
1. Concentrate on what she has left in life and make a better effort to do what she needs to do by her son.
2. If she can't or won't do that then tell her that you would understand if she needed to have some time to figure things out and that while she is doing this you would be more than happy to help her out (whether that be with her or with only your nephew is up to her).
3. If she refuses your help and will not improve her son's life then call CPS. I have had to do that one time when there was nothing else to do and just the first visit helped them realize that they needed help. Maybe at that point she would be more willing to shape up ... or ship him out to you.
Also I personally think your brother needs a swift kick in the butt. I am sure you probably agree but figured I would put my two cents in there. He is just as responsible for him as she is and he is just leaving him to whatever happens to him (that's what it sounds like). That's awful and I can't imagine how your SIL feels or how his son feels about all this.
Good Luck!