I used to be best friends with a lady that had the ex from hell. She worked at a glass manufacturing company here in Oklahoma. They would often have stuff coming down the line and need to finish that before they left. So she might be 20-40 minutes late. Sometimes it was something else, traffic, some boss stopping her on the way out to speak with her about something, or even her hurrying to get home would get her a speeding ticket.
He would often meet her at the door with a shot gun. Accusing her of cheating and all sorts of things. She had a bunch of kids with him because he refused to let her use birth control and she felt trapped.
She left him and took her kids one day. He raged and hunted them and didn't find them until the divorce was final and he had visitation.
Years later, when they kids were starting to be teenagers, she would still show her friends and co-workers his picture and tell them if they ever saw him anywhere they should let her know right away. If he was asking questions about her they should say they didn't know that name or if shown a photo and they couldn't lie they were to say they didn't know her well.
She was still afraid of him years later is my point. Things like this, the accusations and more that you're not saying, are only going to grow and grow. Please go to counseling as soon as possible. If he/she won't go with you then go for yourself.
For you, so you can grow and learn skills to recognize when it's normal concern and when it's not.