Personally, I'd tell him that if he and his girl friend wanted to get high before coming to my house - skip coming to my house. But that's me, and I'm not known for mincing words or remembering manners when it comes to the safety of my children.
That said, based on what you've written they're behavior doesn't concern you with regards to safety - so you might say something along the lines of, "Between her end of the phone conversation and her behavior while you guys were at my house, I'm concerned that Ash was under the influence of ______ while you guys were here. I love you, and I love her - but I can't have that stuff around my kids - then either he will get pissed and hang up, or you can suggest an alternative (shorter visits, if she is an addict - or choosing not to use while she's there, if he insists it's just recreational - as my niece did to me) ... and you simultaneously set the boundaries for his behavior.
Some people might consider it going behind her back - for the record, in my family we deal with the family member and if they're not married, then she's not family.