I see both sides, as a former and sometimes relapsed smoker, and as a mom who doesn't want it around my kid. I wouldn't want it inside my house either.
Not ALL smokers are rude and imposing. Just like not all non-smokers are rude and imposing, ladies. I was always a very courteous smoker. I understand not everyone wants to be around it. But the outside is the outside. You don't say where they smoke, but I am guessing thier patio or something. It isn't really reasonable to expect them to find somewhere else unless they are going in a common area and could just as easily move over 10 feet. If that is the case, they should, I agree.
My suggestion is the fan also. Put a fan in your window blowing out. Also, you could try to work with them on the issue. Is the area set up in a way that you could provide them with a fan also that will blow the smoke in a different direction? I would have tried to find a "move around to a different spot" solution to begin with. Gone down and said, hey, here is what is going on, can we try something and see if rearranging helps? Then have one person stand upstairs and direct them. I would talk to them, don't apologize if you aren't sorry, but just explain and try to make amends and work with them.
As as side note to everyone who hates the smokers being outside of buildings- part of why smokers smoke outside the doors is because that is where the ashtrays are. If businesses provided ashtrays further away, that would be where most of them smoked. And people don't realize that walking across that parking lot in a city to get to those doors makes you breathe in more carcinogens than the seconds it takes to pass through them.