Foundation model

A foundation model is a large AI model that has learned from a huge, varied collection of text, images or code, so it can do many different jobs instead of one. The models behind ChatGPT, Claude and Gemini are examples. A business rarely builds one: it uses an existing model and adapts it to its own work, such as answering customer emails or summarising invoices.

Where does the term come from?

Stanford researchers introduced it in August 2021 in On the Opportunities and Risks of Foundation Models (Bommasani et al.). Their definition: any model trained on broad data, generally with self-supervision at scale, that can be adapted to a wide range of downstream tasks. Their examples were BERT, GPT-3 and CLIP. The word "foundation" stresses that such a model is incomplete on its own: it is the common basis from which many task-specific models are built.

The EU AI Act (2024) calls this kind of model a "general-purpose AI model" (European Commission). Since 2 August 2025, providers of new models of this kind must, among other things, give technical documentation to companies that build the model into their own systems.

How is a foundation model made?

Meta's paper on Llama 3 (July 2024) describes two stages:

1. Pre-training: the model learns from a very large amount of data with a simple task, such as predicting the next word of a text. The data supplies its own answers, so nobody has to label it (self-supervised learning). Meta pre-trained its largest Llama 3 model, with 405 billion parameters, on 15.6 trillion tokens.
2. Post-training: the model is tuned to follow instructions, to match human preferences and to get better at specific skills such as coding, with methods such as RLHF or Anthropic's Constitutional AI.

A business then adapts the model to its task: with a prompt that describes the job, with RAG, which adds relevant documents to the question, or with fine-tuning on its own examples. Meta published both the pre-trained and the post-trained version of its largest model, so developers can start from either.

Why does it matter?

The report sums up the significance in two words:

• Emergence: what the model can do follows from training instead of being built in. GPT-3, with 175 billion parameters, could take on a new task from a description in the prompt, which it was neither trained for nor expected to do. Wei et al. (2022) call an ability emergent when smaller models lack it and larger ones have it. Schaeffer et al. (2023) argue that many of those sudden jumps come from the chosen metric.
• Homogenisation: many applications build on the same few models. An improvement in the model helps all of them, and every application built on it inherits its defects, such as a bias.

Where does it go wrong?

In 2021 the report's authors wrote that there was no clear understanding yet of how these models work, when they fail and what they can do. Test a model on your own task before you rely on it, and expect its errors, such as hallucinations, to carry over into your application. Training your own is rarely an option: Stanford's 2024 AI Index estimated the compute for training GPT-4 at 78 million dollars and for Google's Gemini Ultra at 191 million.

Example

Suppose an installation company wants to draft quotes faster. It uses a foundation model through a chat service or an API, supplies its price list and terms through RAG, and puts an example of a good quote in the prompt. If the drafts should match its house style more closely, it can fine-tune the model on its past quotes.