There are obviously several things your son is responding to here L.:
~ the seperation
~ being forced to stay at his dad's away from you (regardless of the way he is treated there it is stressful for a 3 year old to move from house to house).
~ separation from his brother when he is at his dad's house (unless you send them both).
It is time to get very clear about the separation and whether or not your three year old will tolerate it. You are most likely getting some counseling - or should do so right away. You will have an easier time sorting all of this out if you have a clear head to counsel you.
After two separations at this exact age for both of my kids, I can honestly tell you that it is hardest on the 3 year old. They are traumatized by every aspect of separation at this age.
I wish I could be more comforting. You can love him and hold him and do everything you can... but his world is crashing in around him. His feelings are bound to be big as will be his moods and needs to cling, cry and misbehave.