The Limitations of Standardized Science Tests as Benchmarks for Artificial Intelligence Research: Position Paper

Understand

In this position paper, I argue that standardized tests for elementary science such as SAT or Regents tests are not very good benchmarks for measuring the progress of artificial intelligence systems in understanding basic science.

• The primary problem is that these tests are designed to test aspects of knowledge and ability that are challenging for people; the aspects that are challenging for AI systems are very different.
• In particular, standardized tests do not test knowledge that is obvious for people; none of this knowledge can be assumed in AI systems.
• Individual standardized tests also have specific features that are not necessarily appropriate for an AI benchmark.