Pre-training

Pre-training is the first and biggest stage in making an AI model. The model reads a huge amount of general text, such as web pages and books, and learns how language works and a lot about the world. After that it is trained further for a task, such as answering customers. That is why a chatbot knows a lot in general and little about your own business.

Where does the term come from?

Pre-training means training a model from scratch: its weights start at random values, and it learns from a very large collection of data, which can take weeks (Hugging Face). The result is then adapted to a task with far less data, a form of transfer learning. OpenAI's first GPT (2018) applied this to language: generative pre-training on unlabelled text, then fine-tuning for each task, which beat the best earlier results on 9 of 12 tasks. The P in GPT stands for it: generative pre-trained transformer (Eloundou et al., 2023).

How does it work?

Pre-training needs no labels, because the text supplies its own answers (self-supervised learning). A GPT-style model predicts the next token from the ones before it. Google's BERT hides some of the words and predicts them from the words on both sides (Devlin et al., 2018). Along the way, in the words of Meta's Llama 3 paper, the model "learns the structure of language and obtains large amounts of knowledge about the world from the text it is 'reading'".

How much data a model needs is the subject of scaling laws. Hoffmann et al. (DeepMind, 2022) found that for the best result on a fixed compute budget, model size and training data should grow at the same rate. Their Chinchilla, with 70 billion parameters and four times more data, beat the 280-billion-parameter Gopher on the same budget.

What comes after pre-training?

A pre-trained model has a rich understanding of language, Meta writes, but does not yet follow instructions or behave like an assistant. Post-training does that: supervised fine-tuning on examples of good answers, then learning from human preferences, for instance with RLHF. In OpenAI's InstructGPT study (Ouyang et al., 2022), people preferred the answers of a 1.3-billion-parameter model trained this way to those of the 175-billion-parameter GPT-3. For GPT-4, the technical report says only that it was pre-trained to predict the next token on public data, such as internet data, and licensed data, and then fine-tuned with RLHF. It gives no model size, compute or dataset details.

What does it cost?

Pre-training is the expensive part. The Stanford AI Index 2024 estimated the compute for training GPT-4 at 78 million dollars and for Google's Gemini Ultra at 191 million. Hugging Face therefore advises starting from a pre-trained model as close to your task as possible and fine-tuning it, which costs far less time, data and money. With a large amount of text from one field, a second round of pre-training on that text can still help: Gururangan et al. (2020) measured gains on classification tasks in four domains with this domain-adaptive pre-training.

Where does it go wrong?

A model only knows what was in its pre-training data. Meta's data for Llama 3 contains knowledge up to the end of 2023, and Anthropic lists a training data cutoff for each Claude model in its model overview (September 2026). The quality of the text matters as well. When OpenAI built the data for GPT-2 (2019), it noted that web scrapes such as Common Crawl have "significant data quality issues".

Example

Meta pre-trained its largest Llama 3 model, with 405 billion parameters, on 15.6 trillion tokens, using up to 16K H100 graphics cards and almost 50 times the compute of the largest Llama 2. It first trained with a context window of 8K tokens and then continued pre-training to reach 128K. Only then came post-training: several rounds of supervised fine-tuning and preference training, which turned the model into an assistant that follows instructions.