T.A.
When my son was potty training, he refused to poop in the potty, and didn't want to go in his pants...so he held it...and we ended up at the doctor's office where an x-ray of his tummy was recommended and done. Yep, bowels were completely full, which is really not good because it stretches them out and leads to big problems. I was told to use stool softener, which is not habit forming, but would make it pretty impossible for him to hold it :) I didn't stress about where he went, as long as he went. We kept him on the stool softener until he decided on his own that the potty was better than his pants, and he had formed a daily habit with bowel movements, which only took a few weeks :)
Since you have a fear issue, I would leave the potty out of the daily conversation for awhile and get a video, book, or toy, and let him explore on his own with it, rather than push it on him. (If he thinks you want him to watch it, hear it read, or play with it, he'll resist, especially when he finds out it has anything to do with that potty!)
And relax...