On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems

Understand

We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components.

• Understanding and fixing errors that arise in such integrative systems is difficult as failures can occur at multiple points in the execution workflow.
• Moreover, errors can propagate, become amplified or be suppressed, making blame assignment difficult.
• We propose a human-in-the-loop methodology which leverages human intellect for troubleshooting system failures.