Fine-tuning

Fine-tuning means training an existing AI model a little further on your own examples, so that it learns a specific task or style. A webshop could train a language model on a set of approved product descriptions, so that new descriptions come out in the house style without a long instruction every time. For facts that change, such as prices or stock, other methods work better.

How does it work?

Hugging Face's documentation puts it simply: fine-tuning continues training a large pretrained model on a smaller dataset for a specific task or domain. It is the same process as pre-training, except that you do not start from random weights, and it needs far less compute, data and time. The difference in scale is large. Meta pre-trained Llama 2 on 2 trillion tokens (Touvron et al., 2023). OpenAI's fine-tuning guide sets a minimum of 10 examples and recommends starting with 50 well-crafted ones (September 2026).

In supervised fine-tuning, each example pairs an input with the desired output. The model's answer is compared with that output, and the weights are adjusted to reduce the difference. The chat assistants people use are built this way: for InstructGPT (2022), OpenAI fine-tuned GPT-3 on answers written by human labelers and then with RLHF. Raters preferred the answers of the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3.

Do you have to retrain the whole model?

No. Full fine-tuning changes every weight, which gets expensive for large models. LoRA (Hu et al., 2021) freezes the original weights and trains small added matrices. For GPT-3 175B, it cut the number of trainable parameters by a factor of 10,000 and the GPU memory by a factor of 3, with quality on par with full fine-tuning. QLoRA (Dettmers et al., 2023) adds a model compressed to 4 bits, which is enough to fine-tune a 65-billion-parameter model on one 48 GB GPU. Methods like these are known as PEFT (parameter-efficient fine-tuning).

When is it worth it?

OpenAI's model optimization guide starts with measuring and prompt engineering, which "may be all you need". Fine-tuning adds more examples than fit in one prompt, shorter prompts that save tokens at scale, training on sensitive data without sending it with every request, and a smaller, cheaper model that does one task well. The guide names classification, nuanced translation and content in a specific format as uses.

Where does it go wrong?

• New facts do not stick well. Ovadia et al. (2023) found that RAG consistently beat unsupervised fine-tuning for adding knowledge. Gekhman et al. (2024) found that models learn new facts slowly through fine-tuning and hallucinate more once they have learned them.
• The model can forget. Training on one domain can erode what the model could do outside it (catastrophic forgetting). Biderman et al. (2024) found that LoRA forgets less than full fine-tuning, and also learns less.

Example

Suppose a webshop wants product descriptions in its house style: short sentences, specifications in a fixed order and no superlatives. It first tries a prompt with the style guide and a few examples. If the output keeps drifting, it collects approved descriptions together with the product data they were written from, fine-tunes a model on about 50 of them as OpenAI suggests, and checks the results on products the model has not seen. Prices and stock stay out of the training data: the model gets them from the product database when it writes.