A Test for Evaluating Performance in Human-Computer Systems

Understand

The Turing test for comparing computer performance to that of humans is well known, but, surprisingly, there is no widely used test for comparing how much better human-computer systems perform relative to humans alone, computers alone, or other baselines.

• Here, we show how to perform such a test using the ratio of means as a measure of effect size.
• Then we demonstrate the use of this test in three ways.
• First, in an analysis of 79 recently published experimental results, we find that, surprisingly, over half of the studies find a decrease in performance, the mean and median ratios of performance improvement are both approximately 1 (corresponding to no improvement at all), and the maximum ratio is 1.36 (a 36% improvement).