Self-supervised learning

Self-supervised learning is a way to train AI on raw data, such as text, photos or recordings, without people labelling the examples first. The model hides part of the data from itself and learns by predicting the missing part, for example the next word of a sentence. This is how the language models behind chatbots learn language, and it avoids the slow and costly work of labelling by hand.

How does it work?

Yann LeCun and Ishan Misra of Meta describe the general technique (March 2021) as predicting any hidden part of the input from the part that is visible. The task the model trains on is called a pretext task. It is made from the data itself, so the right answer is always known (IBM). Common pretext tasks:

• Predict the next token from the ones before it. GPT-4 was pre-trained this way, according to its technical report.
• Fill in hidden words. BERT (Google, 2018) masks 15% of the tokens in each sequence at random and predicts them from the words on both sides.
• Rebuild part of an image. Masked autoencoders (He et al., 2021) hide 75% of the patches of a photo and reconstruct the missing pixels.
• Match two versions of the same example. SimCLR (Chen et al., 2020) learns that two edited copies of one photo belong together and copies of different photos do not.

Text was the easier case. For a missing word the model can give a probability to every word in its vocabulary, LeCun and Misra explain, but there are endless possible video frames or image patches. In March 2021 they wrote that self-supervised learning had not yet improved computer vision as much as language. Later that year the masked-autoencoder paper reported better transfer results than supervised pre-training.

What is it for?

The pretext task itself is rarely the goal. As Lilian Weng puts it, what matters is the representation of the data that the model learns along the way. That representation is then reused for a real task with few labels, a form of transfer learning. wav2vec 2.0 (Facebook AI, 2020) pre-trained speech recognition on 53,000 hours of unlabelled audio; fine-tuned on only ten minutes of transcribed speech, it reached a word error rate of 4.8% on the clean LibriSpeech test set.

Is it supervised or unsupervised?

Both names are used. IBM's machine learning explainer calls it essentially supervised learning on unlabelled data, and IBM's explainer on self-supervised learning says it uses unsupervised learning for tasks that normally need supervised learning. LeCun and Misra prefer "self-supervised" to "unsupervised", because it uses far more feedback signals than standard supervised learning does. What sets it apart is the source of the labels: the data itself.

Where does it go wrong?

Predicting text is a different goal from helping a user. OpenAI's InstructGPT paper (Ouyang et al., 2022) notes that predicting the next token on a web page differs from following instructions helpfully and safely, and that such models make up facts, produce biased or toxic text or ignore instructions. That is why chat models get further training after pre-training, such as RLHF.

Example

Take the sentence "The customer paid the invoice on time". If each word counts as one token, next-token prediction turns it into six training examples without any human work: "The" → "customer", "The customer" → "paid", and so on up to "The customer paid the invoice on" → "time". A masked model gets the same sentence with, say, "invoice" hidden and fills it in from both sides.