SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data

Understand

Recent success in fine-tuning large models, that are pretrained on broad data at scale, on downstream tasks has led to a significant paradigm shift in deep learning, from task-centric model design to task-agnostic representation learning and task-specific fine-tuning.

• As the representations of pretrained models are used as a foundation for different downstream tasks, this paper proposes a new task-agnostic framework, \textit{SynBench}, to measure the quality of pretrained representations using synthetic data.
• We set up a reference by a theoretically-derived robustness-accuracy tradeoff of the class conditional Gaussian mixture.
• Given a pretrained model, the representations of data synthesized from the Gaussian mixture are used to compare with our reference to infer the quality.