My son did that at about that age. He also has a serious misgiving about grass.
He grew out of both, but it took a long time. And in order to accustom him to even being in the presence of sand or grass, he would suit up like it was nuclear waste we were asking him to go out it. Socks pulled up to his knees, shoes, long pants, long sleeved shirt, gloves, and if he could have convinced us to put a bag over his head, he would have worn that as well. He could. not. stand. the idea of getting dirty, OR the "feel" of sand or grass.
2 years later we were fishing him out of mudholes, and unburying him at the beach, and sweeping grass out of the house and having to check pockets for "treasures" before they went in the wash. ((And 5 years later, even if he JUST had a bath, he still ends up dirty 2 seconds later)).
We just slowly and casually made a big deal about how everything washes off... and that he could change his clothes... and that he didn't have to touch it if he didn't want to... but it wouldn't hurt him AND would wash off if he did touch it.
I know for him, part of his fears were temp based. Grass is cold. Sand is either hot or cold. The next part was texture. & The last bit was the whole "contamination" thing. We also didn't avoid areas he was afraid of. I'd carry him, with a big smile, but we WERE going to walk across the grass and not take the long way around. Sometimes he would freak out... and I'd smile and comfort him as we walked, and at the "end" would set him down and point out none got on him, and that wasn't so bad was it? He'd be horrified for ME having to touch it, but I would lay down and roll in it. "See kiddo? I LIKE sand/grass. YOU don't have to, that's okay, people like different things... but I LIKE _______. But I'll carry you for as long as you want. Mom's tough."
Anyhow... just our story.