Beyond errors: Measuring reliability for error-prone interaction devices

The development of assistive technologies and ubiquitous computing highlights the need to better understand errors associated with both the limitations of the devices being used and difficulties introduced by the environment in which interactions occur. At the same time, we need to better understand the relationship between the user experience and the consequences users encounter when errors occur. Although error rates are the most common measure of reliability reported in the human-computer interaction literature, this simple metric fails to address the different consequences users may experience. We propose a new metric, leveraging the concepts of entropy and desirability, to quantify the concept of reliability. An empirical study provides a preliminary validation of this new metric, focusing on its ability to describe several aspects of user satisfaction as well as task completion time. Results confirm that our new metric is more effective than error rates when describing user satisfaction and that the metric can also be used to describe task completion times when error rates are high.