Hey, newmom, I'm sorry you've gotten grief from some folks here about building where you did. You can live next to relatives and not have to have them in your hair every minute, and you have the right to take your family up on the free land, especially in this economy. You do want to be sure, though, that you get your private space without starting bad feelings.
As others have said -- communication. Assuming your BIL is your husband's brother, it's your husband's job to be up front, in a friendly way so no one gets their dander up, about calling first, and not just dropping in without one simple phone call.
And remember, this will pass. Your BIL's three kids include two who are old enough that your constantly saying, "Shhh, the baby's napping" etc. will drive them away for quite a while! And the 2-year-old, well, he or she has no business dropping over anyway without an adult! They will quickly learn that your house isn't an amusement park and they'll drop their comments about riding bikes in the basement and watching TV there. (The TV can be conveniently broken, or off limits, -- "Sorry, we only turn on the TV at night after the baby's gone to sleep. Ever. Want to do homework with me?") They'll lose interest.
Keep everything light and yet firm, showing a lot of love and actually inviting the kids WITH their parents over for some social times, board games, play in the yard, cookouts etc. You DO like them and want to see them at times, after all! But if the kids come over unannounced, smile a big, warm smile, and say, "Nope, not today, I'm studying/playing with the baby/just plain busy; I'll call your folks and we can all get together this weekend" etc.