I agree with you (though it's your mom's call in the end). She has returned the dog twice, which was beyond the call of duty. And next time, all it would take would be for the dog to be agitated over seeing a squirrel or anything and its "personality" could change; she could get bitten. That would really open up a dreadful can of worms and create possibly years of strained relations with the neighbors--maybe even legal problems. Yikes.
If it were me, next time I saw the dog I'd phone them immediately and say the dog is in my yard right now; I cannot return it to you this time (no need to explain why); I'm concerned it will wander farther and be hit by car; please retrieve it now. Then when they come to retrieve it, I would come outside and say, "This is the third time he's jumped the fence and I am very concerned for his safety. Not all the neighbors are going to return him like I did or call you like I did just now. Please consider looking into a higher fence, or an "invisible fence" or some other way to keep him in your yard. If he can get into my yard, he can get into other yards or the street as well. I would hate for him to be lost or injured, but I am not always going to be around to return him or to call you, and other neighbors may well call animal control instead of calling you." I'd say it nicely but firmly and emphasize that if the dog gets into MY yard it's going to get into other yards and other neighbors may not be so accommodating.
If they don't do anything, and the dog does it again -- I would call them maybe one last time, for the sake of peace (not return or touch the dog, call them) and then any further times, I'd call animal control. I would frankly want animal control to keep my call anonymous and just handle it as "we heard there was a dog roaming the area and that you had a dog like this one."
Does your mom have fenced yard? They may think it's all fine since the dog is going into a yard with perhaps a better fence than their own!
But I definitely would never again touch a dog that wasn't my own. I don't care how he seems to be -- he could decide next time she's coming at him too fast, or he doesn't like something else in the yard, or he's startled by her. It's not worth gettting bitten.