ModelDiff: A Framework for Comparing Learning Algorithms

Understand

We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms.

• We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other.
• We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data.
• We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters.