What Are Foundation Models?

Foundation models, sometimes called large AI models, are machine learning models trained on broad data at scale, typically through self-supervision, and then adapted to many downstream tasks rather than built for a single one. The term was introduced in 2021 by researchers at Stanford's Center for Research on Foundation Models in a report on the opportunities and risks of foundation models, which argued that a shift had occurred in how machine learning systems are built: instead of training a separate model per task, practitioners now train one general model and specialize it.

Two properties define the category. The first is emergence, meaning that capabilities such as in-context learning appear as a consequence of scale rather than being explicitly designed in. The second is homogenization, meaning that a small number of base models come to underpin a large number of deployed systems, so their strengths and their defects propagate downstream. Both properties follow from the economics of pretraining, where the cost of a single training run is high enough that reuse becomes the dominant pattern.

Pretraining at Scale

A foundation model is produced by a pretraining run over a large, largely unlabeled corpus using an objective that supplies its own supervision, such as predicting a masked or subsequent token, reconstructing a corrupted image, or aligning paired text and pictures through a contrastive loss. The transformer architecture, introduced in the 2017 paper "Attention Is All You Need", is the usual backbone because self-attention parallelizes well across accelerators and scales predictably with parameter count and data volume. Empirical scaling relationships between compute, dataset size, and loss let developers forecast the quality of a model before committing to a training run, which is why compute budgets rather than architectural novelty often set the ceiling on performance.

Adaptation to Downstream Tasks

Adaptation is what distinguishes a foundation model from a conventional pretrained network. Full fine-tuning updates every weight on task-specific data. Parameter-efficient methods such as low-rank adaptation and adapter layers train a small added set of weights while leaving the base model frozen, which cuts storage and lets many specializations share one set of base parameters. Instruction tuning and reinforcement learning from human feedback align the model with a style of response rather than a narrow task. Prompting and retrieval-augmented generation adapt behavior without changing weights at all, by supplying task descriptions, worked examples, or retrieved documents in the input context.

Multimodality and Domain Models

Although the category grew out of language modeling, the same recipe now covers other data types. Vision models pretrained on image-text pairs support open-vocabulary classification and segmentation. Speech models transcribe and translate across languages from a single checkpoint. Scientific variants apply the pattern to protein sequences, weather fields, medical images, and materials data, where the pretraining corpus is a structured measurement archive rather than web text. These domain models inherit both the adaptation advantages and the evaluation difficulties of their language counterparts.

Evaluation and Governance

Because one model serves many uses, evaluating it against a single benchmark says little about deployed behavior. Assessment therefore spans capability suites, red-teaming for misuse, bias and fairness measurement across demographic groups, and documentation of training data provenance. The NIST AI Risk Management Framework provides a voluntary structure for this work, and its generative AI profile addresses risks specific to models of this kind. Transparency indexes published by Stanford's Center for Research on Foundation Models track what developers disclose about data, compute, and downstream use.

Applications

Foundation models have applications in a range of fields, including:

• Conversational assistants and code generation tools
• Document search, summarization, and retrieval systems
• Medical imaging analysis and clinical text processing
• Protein structure prediction and molecular design
• Weather and climate emulation
• Robotic perception and vision-language-action control