Catastrophic forgetting

Catastrophic forgetting is when an AI model loses skills it already had because it was trained further on something new. Suppose a company fine-tunes a general model on its own support tickets: the model gets better at those tickets and can get worse at tasks it used to handle well. A neural network keeps all its knowledge in one shared set of numbers, so new training can overwrite old knowledge.

Why does it happen?

Kirkpatrick et al. (DeepMind, 2017) describe knowledge of an earlier task being "abruptly lost" when a network is trained on tasks one after another: the weights that were important for task A are changed to meet the objectives of task B. When the data of all tasks is mixed in one training run, this does not happen, because the weights can be optimised for all tasks together. McCloskey and Cohen described the problem in 1989 as catastrophic interference, "the sequential learning problem".

Does it happen in large language models?

Yes. Luo et al. (2023) gave models of 1 to 7 billion parameters a series of instruction-tuning tasks and measured what they lost in domain knowledge, reasoning and reading comprehension. Forgetting occurred throughout, and within that range it was stronger in the larger models. Models that had first been through general instruction tuning forgot less during the later fine-tuning.

How do you limit it?

• Replay: mix old training data into the new training. Kirkpatrick et al. note that storing and replaying everything becomes impractical as tasks pile up; Rolnick et al. (2019) found that a limited buffer that discards data at random works almost as well as an unlimited one.
• Protect important weights: EWC (elastic weight consolidation) slows down learning on the weights that mattered for earlier tasks. A penalty that protects all weights equally limits forgetting, but leaves too little room to learn the new task.
• Train a small add-on: LoRA freezes the base model and trains small extra matrices. Biderman et al. (Columbia and Databricks, 2024) found that it learns less than full fine-tuning in the target domain and forgets less outside it.
• Add capacity per task: progressive networks (Rusu et al., 2016) add a new network column for each task and keep the old ones fixed.
• Keep facts out of the weights: with RAG new knowledge goes into documents, and the model stays as it is.

Example

Kirkpatrick et al. trained one agent to play ten Atari games, one after another and then again in turns. Their score adds up performance per game, with 1 point for human level, so the maximum is 10. With plain gradient descent the agent never learned more than one game at a time: each new game pushed out the previous one, and the total stayed below 1. With EWC the same network learned to play several of the games. The research field that studies learning without forgetting is called continual learning.