I wouldn't hesitate. It may feel weird to return fruit, but I have returned many food items that were not what I paid for. Once I cut a sweet potato and tiny worms crawled out!! You can bet I returned that - I thought they should know that is what was going on with their sweet potatos.
It all depends upon how much it is worth to you. That sweet potato was worth pennies, I am sure, but I was going to the store anyhow, and I thought they should know about something so unusual. Otherwise, maybe I wouldn't have made the effort. Just depends upon whether the cost of the purchase balances out with the hassle factor of returning the item. But on principle, you should feel fine about returning merchandise that is subpar, even if it is food.