On the Joint Interaction of Models, Data, and Features

Understand

Learning features from data is one of the defining characteristics of deep learning, but our theoretical understanding of the role features play in deep learning is still rudimentary.

• To address this gap, we introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features.
• With the interaction tensor, we make several key observations about how features are distributed in data and how models with different random seeds learn different features.
• Based on these observations, we propose a conceptual framework for feature learning.