Embedding

An embedding is a list of numbers that an AI model makes from a piece of text, such as a sentence or a product description. Texts with a similar meaning get similar lists, even when they share no words. That lets the search on a help page find the article "Cancel your subscription" for a customer who types "how do I stop paying?".

What does an embedding look like?

An embedding is a vector: a fixed-length list of decimal numbers. The length depends on the model: 384 numbers for the open model all-MiniLM-L6-v2, and by default 1,536 for OpenAI's text-embedding-3-small and 3,072 for text-embedding-3-large. What counts is the distance between two vectors. According to OpenAI's embeddings guide, a small distance suggests that two texts are closely related and a large distance that they are not. The usual measure is cosine similarity.

Where does the idea come from?

One of the best-known word embeddings is word2vec. Mikolov et al. (Google, 2013) computed word vectors from very large text collections and reported that simple arithmetic works on them: vector("King") minus vector("Man") plus vector("Woman") lands closest to vector("Queen"). Jay Alammar's Illustrated Word2vec shows how that works.

Later models embed whole sentences. Sentence-BERT (Reimers and Gurevych, 2019) produces sentence embeddings that can be compared with cosine similarity. Finding the most similar pair among 10,000 sentences took about 65 hours with BERT, which has to read every pair together, and about 5 seconds with Sentence-BERT.

How is an embedding model trained?

Modern embedding models learn from pairs of texts that belong together. For its embedding research, OpenAI (Neelakantan et al., 2022) used neighbouring pieces of text on the internet as pairs. Training pulls the vectors of each pair together and pushes them away from the other texts in the same batch (contrastive learning). The resulting embeddings worked for both classification and semantic search.

How do you use them?

OpenAI lists search, clustering, recommendations, anomaly detection, diversity measurement and classification as common uses. For semantic search and RAG, the steps are:

1. Embed every document or passage once and store the vectors in a vector database.
2. Embed each question with the same model. Vectors from another model, or from a retrained version of the same model, are not compatible (Hu et al., 2022); the models above do not even produce the same number of values.
3. Return the passages whose vectors lie closest to the question.

Where does it go wrong?

• Embeddings copy the biases in their training text. Bolukbasi et al. (2016) found that word2vec vectors trained on Google News completed "man is to computer programmer as woman is to x" with "homemaker".
• A similarity score is relative. What counts as a high score differs per model, so test any cut-off on your own data (see cosine similarity).

Example

Suppose a webshop's help centre has an article titled "Cancel your subscription", and a customer types "how do I stop paying?". Keyword search finds nothing, because the two share no words: search engineers call this problem synonymy (Manning, Raghavan and Schütze). With embeddings, the shop embeds each help article once and each question as it comes in, and the search returns the articles with the closest vectors. A chatbot can then answer from those articles. Whether the right article comes out on top depends on the model, so the shop tests it with real customer questions.