We are the only species that consumes the milk of other species. The biological reality is that cows milk is made and designed for calves, not humans. Goats milk is made and designed for baby goats, not humans. The only proper milk that is made and designed for humans is human milk. And it does a pretty good job too.
The amount of people who are intolerant to dairy foods is therefore not surprising.
If he's thirsty give him water. If he takes it easily then you have no problem.
If you provide a variety of healthy foods for him to eat, fresh veges, fruits, meats and grains, he will not need milk. He will get all the calcium he needs from his food. He'll be just fine.
Cutting down breastfeeding can be done just with distraction. Perhaps take him out for a walk or play to a park at times he would want to feed and you don't?
Also, humans have developed to feed our young with our milk for the first few years of life. The World Health Organisation recommends breastfeeding for the first 2 years. The USA is a country where this biological reality is seen as somehow odd, or yuckky, and when you think about it, how weird is that! If you want to continue don't let others opinions pressure you to stop. If you want to stop, same applies. It's your choice. Just be aware of the disconnection between our biology and cultural attitudes to breastfeeding.
Good luck!