Throughout the day, our alertness levels change and our cognitive performance fluctuates. The creation of technology that can adapt to such variations requires reliable measurement with ecological validity. Our study is the first to collect alertness data in the wild using the clinically validated Psychomotor Vigilance Test. With 20 participants over 40 days, we find that alertness can oscillate approximately 30% depending on time and body clock type and that Daylight Savings Time, hours slept, and stimulant intake can influence alertness as well. Based on these findings, we develop novel methods for unobtrusively and continuously assessing alertness. In estimating response time, our model achieves a root-mean-square error of 80:64 milliseconds, which is significantly lower than the 500ms threshold used as a standard indicator of impaired cognitive ability. Finally, we discuss how such real-time detection of alertness is a key first step towards developing systems that are sensitive to our biological variations.