There is no concern going on a date. This is a time you really need the stress reliever. You don't need to go to the ER. You do need to find out what's causing them, though. Is it times when he gets really mad? My son seizes when he gets mad. Add that too a really long list of other issues! (Think $2700 in a trip to the pharmacy kind of issues.) I have not had a problem with babysitters as long as I tell them how to tell it's coming on, what to expect, and what to do if he does actually seize.
If it's caused by anger, what works for us is, very calmly and quietly, take him in our arms and gently ask him to calm down. The trick is catching it early enough. I wrap my arms around his with his down, and give him a gently but firm hug while I do this. I just hold him then until he can breathe normally again. Works about 99% of the time. Funny thing, though, it only works for me. I ask babysitters to just lay him down while calming him down so he doesn't hurt himself. Two days ago, he had one on the stairs and I couldn't get to him in time. It was interesting, with his body so stiff, he just gently glided down the stairs. No pain or injury at all. His are considered grand mal, although they are not severe. Interestingly enough, in my research, I came across something that said some kids find a seizure very quieting and comforting, and will make themselves go into a seizure. Seems that's my kid. :)
Hang in there! It's going to get better! Let me know if you're ever just needing to "chat" for encouragement. I can't babysit from this far away, but I'd love to help out in any way I can. P.S. I fit happens again, you might want him to be someplace where it wouldn't be a problem if he wets his pants. My little guy is also 4, and he does that sometimes as he comes back out of it (maybe 10-25% of the time).