M.W.
S., the laws on child visitation and protection vary from state to state. And the outcomes will differ depending on the judge who applies the law to the facts in your case. I strongly suggest that you seek legal advice so you know exactly what your rights are. If you can't afford an attorney, there is a VA legal aid system available to you, here is their URL http://www.valegalaid.org/VA/index.cfm . But you definitely need to speak with an attorney, because the laws on custody and visitation are very tricky. Did an attorney handle your divorce? Maybe that attorney will help you with this and it won't cost you as much because they already know the situation.
What is always important in any legal case is to show documentation of the violations of the other party. So, you need to start documenting each and every time that your ex does not abide by the court order. Have your mother do the same. Write them down with the dates in a bound notebook. Keep this as a journal. That may not be admissible as evidence, but it may be able to be used to refresh your memory, so you can testify to dates and exact events. You also can document anything that he does that might be a danger to the child. Courts are usually protective of the welfare of the child, but what each sees as harmful varies. The three most important things to do for legal issues are document, document, document. But please check with attorney as to what your rights are. Only an attorney can tell you that, no one else can give you legal advice. And only an attorney knows how the courts usually rule.
I totally agree with Jessica, she gave you good advice as well.