First, let me say how sorry I am that you had to go through a divorce and have this situation to deal with. Nothing is harder than to work with custody issues, especially when you are miles apart from an ex-spouse.
My first impression is that your husband is not thinking about what is best for your son. What your child needs is a stable home with a routine and a schedule. It has been proven that children thrive when there lives are scheduled and routine in that they know what to expect in their everyday lives. Why do you both live so far apart? It sounds as if your ex-husband's home life is other than desirable in terms of what your son is learning (and not learning). However, your ex-husband would probably try to think of reasons why your son should be with him and not you all the time. Your job as parents is to raise your child to be a well-adjusted adult who can survive in this difficult place we call "the world". I could probably go on and on, but it sounds like you already have an idea of what being a good parent is.
My advice...first, can you move closer? Also, the whole week with one, and then another is not good either. My husband has two boys who were 2 and 4 when he was divorced. I also met him at that time. They have joint custody, and the boys stayed with her most of the week and then with him about 3 days out of the week unless other arrangements were made. Once in school, the boys went to her house during the school week and his house every weekend, Friday, Saturday, and every other Sunday. In the summertime the boys are with us Thur, Fri, Sat, and every other Sunday. In my opinion, this is a good plan. The child needs to feel that there is a "primary" home. This creates security. That doesn't mean your ex can't call him and tell him goodnight or find out how his day was, but one home should be the primary home for him.
If you can't move closer, I would suggest hiring a paralegal to help you figure it out legally. Avoid going to court simply because it is costly. Also, you should seek a family counselor's professional opinion about what is best for your child. That gives you a basis to go from in the legal system. Don't make a plan just between you and your ex. You need the plan to be enforceable, and if it isn't legally backed, your husband can change his mind at anytime and throw your son's life into a tailspin.
I am praying for your family that all comes out the way it should...which is what is best for your son. I hope you will find peace and everything works out.
Sincerely,
A concerned Mom and Step-mom
K.