I would talk to the members on the HOA board and explain what’s going on. Tell them you are planning on replacing/fixing the fence once the grading problem is resolved. If you need to take pictures of people going over your fence so they will also see why it is being damaged.
The HOA's have good lawyers working for them so IMO it's not worth fighting (you will end up paying more with legal costs). As for making the people climbing over the fence pay, you need to catch them in the act and call the police for trespassing.
My HOA is very strict, but they will always first work with the homeowners to resolve any issues.