Automated Capability Discovery via Foundation Model Self-Exploration

Understand

Foundation models have become general-purpose assistants, exhibiting diverse capabilities across numerous domains through training on web-scale data.

• It remains challenging to precisely characterize even a fraction of the full spectrum of these abilities and potential risks in any new model.
• Existing evaluation approaches often require significant human effort, and it is taking increasing effort to design ever harder challenges for more capable models.
• We introduce Automated Capability Discovery (ACD), a framework that designates one foundation model as a scientist to systematically propose open-ended tasks probing the abilities of a subject model (potentially itself).