how long is it sitting out? Sounds like it's just a simple case of getting rid of what's no longer good. If it were an illness, it would be more consistent... and if it were an allergy, you would be more likely to see it before now...
We give our son about 30 minutes to drink his bedtime bottle, and then it gets dumped. It gets warmed in the micro (we shake well to prevent hot spots) and the timer begins.
I've heard that milk starts to go bad at room temperature... but have not been able to find out for sure...
I am paranoid about spoiled milk... it's the last thing I grab at the grocery store, and the first thing I put away when I get home.