Foundation models in brief: A historical, socio-technical focus

Understand

Foundation models can be disruptive for future AI development by scaling up deep learning in terms of model size and training data's breadth and size.

• These models achieve state-of-the-art performance (often through further adaptation) on a variety of tasks in domains such as natural language processing and computer vision.
• Foundational models exhibit a novel {emergent behavior}: {In-context learning} enables users to provide a query and a few examples from which a model derives an answer without being trained on such queries.
• Additionally, {homogenization} of models might replace a myriad of task-specific models with fewer very large models controlled by few corporations leading to a shift in power and control over AI.