Continual learning

Continual learning means that an AI model keeps learning from new information after its first training, without losing what it already knew. A model that sorts a webshop's customer emails, for example, should learn questions about a new product line and still handle returns correctly. That second part is hard: a model trained only on new examples tends to forget the old ones.

Why does a model forget?

A neural network stores what it learns in the same shared weights. When new examples differ strongly from the old ones, training on them overwrites what was there: catastrophic forgetting. The review by Parisi et al. (2018) notes that retraining from scratch on all the data solves this, yet is very inefficient and gets in the way of learning in real time. The balance between taking in the new and keeping the old is called the stability-plasticity dilemma.

How does it work?

The survey by De Lange et al. (2019) sorts the methods into three families:

1. Replay: keep a sample of old examples and mix it into the training on new data. With generative replay (Shin et al., 2017), a generative model produces look-alike old examples, so the originals need not be stored.
2. Regularisation: add a term to the training objective that protects what mattered for earlier tasks. EWC (elastic weight consolidation, Kirkpatrick et al., 2016) slows down learning on the weights that were important for old tasks.
3. Parameter isolation: give each task its own part of the model. Progressive networks (Rusu et al., 2016) add a new column of layers per task and freeze the old ones. Adapter modules (Houlsby et al., 2019) add a few trainable parameters per task while the original network stays fixed.

Each family pays somewhere. According to De Lange et al., replay costs extra computation and storage, and storing raw examples can raise privacy issues. Parisi et al. warn that adding capacity for every task can lead to very large networks.

How do you measure it?

Van de Ven and Tolias (2019) describe three test scenarios, based on what the model is told at test time:

• Task-incremental: the model is told which task it is doing.
• Domain-incremental: the task stays the same and only the input changes, so the model does not need to know which situation it is in.
• Class-incremental: new classes arrive over time, and the model has to work out itself which task a question belongs to.

The scenario shapes the result. In their tests on handwritten digits (split and permuted MNIST), regularisation methods such as EWC failed in the class-incremental scenario, and some form of replay seemed necessary.

What about large language models?

The survey by Wu et al. (2024) sums up the tension for large language models: they are too expensive to retrain often, yet they need updates to gain new skills and keep up with new knowledge. Model makers release new versions trained on newer data; Anthropic's model overview lists a training data cutoff for each Claude model. Between versions, there are cheaper routes:

• RAG keeps changing facts in documents the model searches. Wu et al. see it as a way to add factual knowledge, while continual learning aims at the model's language and reasoning skills.
• Adapters such as LoRA leave the base model's weights unchanged. Biderman et al. (2024) found that LoRA learns less of a new domain (programming, mathematics) than full fine-tuning, and keeps more of what the base model could do outside it.

Example

Suppose the webshop from the opening updates its email sorter every month by training it further on that month's emails. After a quiet month for returns, the model may handle return questions worse, because the new examples overwrote part of what it had learned. A simple fix is replay: keep a sample of older, labelled emails and add it to every training round. That costs storage, and because customer emails contain personal data, the privacy point from De Lange et al. applies here too.