I would take them away now, cold turkey. Once that baby comes it will be even longer for him to give them up.
Are you moving him into his own bed when the baby comes? I just wouldn't do both big things at the same time. By two he shouldn't need a crib I wouldn't think either. Maybe letting him keep the binkie for a while but time to move him into a big boy bed. Behaviors can get worse when a new sibling arrives, if you are tired of him throwing them out then you need to put an end to it. Personally I would look at him and just say "I guess you don't want them anymore" and let him scream. May sound harsh but he is totalling controlling the situation.
You just need to make up your mind what you want to be the outcome. I say take them away, he will adjust to it in a matter of a few days or move him to a big boy bed. He is two and you will have your hands full with the new baby, do you really want to be doing these big changes after the fact?
That way he won't associate the baby with the changes. If you empower him now with being a big boy it will be easier when the baby comes.
I never gave my kids pacifiers, they didn't want them, both babbled too much I think. However if I had and it starts causing issues like them waking, crying or making you jump to find them, it is time to get rid of them. They are not good for his teeth, regardless if you are worried, they can cause growth issues in the roof of the mouth and make teeth stick out. Ask your Dentist.
My daughter was 23 mos when I put her in a big girl bed and my son 25 mos, both transitioned in one night. Get a bed rail, take the pacifiers away, send them to the binkie fairy or whatever works for him. Reward him with a big boy toy, really talk up being a big brother and you would be surprised how easy it can be. I can say when my son came along, my daughter was shy of 3 and she had NO PROBLEM with an adjustment, she was so excited and it was so much easier then I had expected. Good luck and CONGRATS!