Mastitis just happens. All you can do is try to prevent it, but yes, see a Dr if it gets to a point that it needs to be seen. What I would do is to pump a bit at night before you go to bed. Not a whole lot, just a couple ounces or so.
On yeast, you can try to give your baby a bit of probiotic powder (acidophalus) mixed with breastmilk from a bottle before you allow him to nurse. I use a product called Primadophalus made especially for infants and children.
Also, massage, massage, massage. I too had a painful 4 or 5 months after weaning my daughter. We're 6 months out now, and I still occasionally get a bit of pain up near my underarms. I didn't get engorged or knotty, just painful. Massaging and applying warm compresses with massage helps.
A lactation consultant also advocated using raw cabbage leaves in your bra. Something about the oils in the leaf helps to dry you up and prevent mastitis. For the record, I never tried this.