Holistic Robust Data-Driven Decisions

Understand

The design of data-driven formulations for machine learning and decision-making with good out-of-sample performance is a key challenge.

• The observation that good in-sample performance does not guarantee good out-of-sample performance is generally known as overfitting.
• Practical overfitting can typically not be attributed to a single cause but is caused by several factors simultaneously.
• We consider here three overfitting sources: (i) statistical error as a result of working with finite sample data, (ii) data noise, which occurs when the data points are measured only with finite precision, and finally, (iii) data misspecification in which a small fraction of all data may be wholly corrupted.