You do not have enough information here.
If you truly have joint custody then he cannot limit the time spent with you, you can go over anytime you want and pick them up. They cannot NOT see you, they are under court order.
My next thought is that joint custody is just that, joint. It doesn't sound like you have any real custody at all, actually, not even visitation. With joint custody you should be consulted on each activity, given a schedule of their weekly appointments, practices, games, parties, etc...if you truly had joint custody you would know where they were and who they were with nearly all day and could walk over and join them at any time, plus go to their activities along with them or take them.
If he has custody and you have liberal visitation then you must get this scheduled and on the court order.
Also, is there any reason you do not attend their activities? Why not? That is a great way of showing them love and support. They will know you care because you show up and make time for them. That is a big deal, if other parents are there and week after week you don't attend they become good witnesses for the ex hubby to say they don't know you and you are an absent parent.
How old are the kids? If they are under 12 years old they do not have a say in if you get scheduled visits or not. If they are teens then they are probably not going to visit with either parent, they will want to do stuff with their friends only...typical teen.
Talk to the counselor about this, if you are attending this in joint sessions then you are seeing them then too, that is time whether is has much freedom or not. He will argue that you have plenty of opportunity but are not taking advantage of it.