I have had this happen to me...
I don't know the laws in your state, they may be different, but until you have an order, from a judge in your hands, legally, he has just as much right to them as you do. He CAN keep them and there's not a single thing the police can do about it.
My hubbs and I were separated and I let our 3 year old go visit him. He normally had nothing to do with him but wasn't a horrible dad or anything. He wasn't a bad person, in other words. But once he got him, HE ran to the court and filed papers as is my son had been living with him and he was caring for him the whole time. There wasn't anything I could do for a week. I couldn't even go around him.
Fortunately he is stupid and thought once the judge signed the papers and set a hearing date, that those papers were his custody order and he let the baby come 'visit' me. I went and picked him up just like I was taking him for a visit, I even played the part of looking defeated. The police knew what I was doing as well as my lawyer and that's how they advised I do it.
I got my child back and immediately filed for an expedited hearing. I let the judge know what all happened as well. Because up until that week, my son was never away from me for more than a night or 2 and that was only with my sister or mother in law.
Of course I got custody after that.
So be careful. If you are fearful and there's no order yet, don't let them go. If they want to see their dad, go meet somewhere in public and let them visit. Don't let them out of your sight with him bc he can just take them.
As long as he doesn't physically rip them from your arms, they are his kids too and he take just as easily as you can.
I learned my lesson the hard way bc I wanted my little man to get to see his dad. He is entitled to a father, after all, and he loves his daddy. But I can't let feelings get in the way of 'what's best for my child'.
Good luck.