D.M.
If this man is good to your kids and to you then don't give up because chances are the kids will not be happy with anyone you date if you let them call the shots I am sure their dad walking out had a huge impact on them even though they were little when it happened kids hear things over the years and they pay attention more than we think they do their anger is probably with their dad but he isn't around to deal with it so they are going with the next best thing.I would look into family counseling and try to let get to the root of the problem but don't give up.