If he's acting cold when he comes home he may be in emotional shock. Who are these grandparents that they act this way toward their own grandchild? I'm assuming they are your parents and therefore you know how they are and what they believe.
Do you not have someone else that can care for your child? If you do you need to get him there, assuming it's better. Does he go there to visit? Don't let him visit without you.
Obviously if there is a continuous negative change in his behavior after being subjected to them he needs to stop seeing them. If they are angry that he is bi-racial they need to get over it and NOW. He is already here and their grandchild. And if they are that upset about it that they are abusive to him then they need to stay away from him. Why are they punishing him for being born. If you want to protect your son, you need to have a long talk with them and get it straight with them that you will not tolerate anything negative from them. If they cannot agree to a set of standards and that they will uphold them then you must walk away from them. This is about the child, not them, not you, but the child, and so everyone needs to come together in peace and put down their useless old thinking. He should not grow up believing something is wrong with him, no one should. And if he does, he'll never be comfortable with himself, he'll always thinks something is wrong with his own being, which only causes pain and fear and pain and fear causes acts of negativity.
Were your parents negative with you or abusive towards you as you were growing up? Only you know how they were and how they can be. So you need to decide and do something important and positive for your child and yourself.
Sending the best for you and your little boy