S.H.
Don't you have custody?
What does the custody and visitation papers say?
To take a child out of State, say on an airplane... there needs to be the permission, ON paper, of BOTH parents. Signed and notarized and he must carry this paper on him. It is a Consent to Travel of a Minor Aged Child With Only One Parent, type of form.
Do your Custody papers, say your Ex can take him out of State? If not then you can tell the authorities.
HOW come, this trip was planned without your consent? and without your knowledge?
A 4 year old is too young to make these decisions.
Your son does not know his Dad very well. Dad is not in the picture on a regular basis.
WHAT do the LAWS say, in your State?
Does your son, have some kind of I.D.?
Before he goes, take a RECENT photo of him, perhaps on the day he leaves.... and print it. That will be a way for you to keep track of your son.
TELL your son/the Dad, that they need to call you during the day and before bed.
Do Skype with them. It is free.
www.skype.com
Also, you really need to see what the legalities of your custody are or not, and of his too.
And, I would hope that, your Ex has insurance etc., in case your son gets sick or something.
Then what?
I personally, would not want my kid going... with a guy that my child does not know well, nor their family.
Mostly you do not trust your Ex or his family.
Too.