if you want to breastfeed long term and dont necessarily want him to supplement if not needed, STOP supplementing. you are confusing your body. everytime he needs to "up" the amount, your body wont really know until he sucks and sucks and sucks. by you feeding him, your body thinks "great, i fulfilled him, good job". then you go and feed him the 2 ounces. next time, you will feed him 3 ounces, and on and on. the only way your body will ever know to make more milk, is for him to suck longer than he used to. also, when you say, hold him off, i would say put him on there. let him suck, whether something comes out of not. the only way to make more milk is by letting your body know he needs more.
if you really want to protect your milk, stop pumping between. feed you baby, and even when you think you are empty, keep him on there. then after you are done nursing(whether he is satisfied or not), immediately start pumping. remember, when you pump, it is not always an indicator of having that amount of milk. when you pump out 4 oz, you may have nursed the baby giving him 6 oz. the baby always gets more out of you. so you pump after, both breasts. and each feeding keep him on even if you think you are empty. if you do this, you should increase your milk because you are telling your body that you need more.
the pumpings in between could def have an impact on your supply by feeding. your body is thinking that milk went to the baby in a 24 hour period, but by you removing it from your breast via pump, that milk isnt there for the feeding.
i swear to you, as long as you dont have any medical issues involved, that if you stop supplementing, your baby wont starve and your milk will increase right away. another option is to get his feedings a little closer together. not sure how far apart he is eating, but if you sometimes pump, you need to get that normal pumping time inside his tummy, not the fridge, so you may need to add a feeding or two during 24 hours.
i really hope this helps. from your post, it sounds like you want to be able to just nurse him, and it sounds like you havent any real problems doing so(which can sometimes deters moms from continuing). i am sure if you just maybe take 3 or 4 days, and adjust times and pump a few minutes after, you will have no problems and can get off the supplementing. good luck to you.