Semantic search

Semantic search finds results by meaning instead of by matching words. A customer who types "something to keep me dry on the bike" in a webshop gets the waterproof cycling jacket, although its title shares none of those words. For a business, it means that customers find things in their own words. An AI chatbot uses the same step to find the right passage in your documents.

How does it differ from keyword search?

Keyword search matches the words of the query against the words in the documents. A widely used formula, BM25, ranks a document higher when rare query words appear in it often, and corrects for document length (Manning, Raghavan and Schütze). Its blind spot is what the same textbook calls synonymy: one concept can be described with different words, so a search for "aircraft" misses a page about a "plane" (Introduction to Information Retrieval).

Semantic search compares meaning instead. Web search moved this way too: in October 2019 Google started using the language model BERT to interpret queries, which it said would help Search better understand one in ten English searches in the US, especially longer queries in which words such as "for" and "to" matter.

How does it work?

Semantic search uses an embedding model, which turns a text into an embedding: a list of numbers that captures its meaning.

1. Embed every document or passage once and store the vectors in a vector database.
2. Embed the query with the same model.
3. Return the documents whose vectors lie closest to the query vector, usually measured with cosine similarity.

Dense Passage Retrieval (Karpukhin et al., 2020) showed that this works for question answering: a model trained on pairs of questions and passages found a passage with the answer in its top 20 results 9 to 19 percentage points more often than a strong BM25 system. A multilingual model even lets a Dutch question find an English document: Reimers and Gurevych (2020) trained models that place a sentence and its translation at the same point, in more than 50 languages.

In RAG, semantic search is the retrieval step: the passages it finds go into the prompt, and the language model answers from them.

Where does it go wrong?

• Exact terms. Embedding models can miss exact matches that matter, Anthropic writes (September 2024): a search for error code "TS-999" may return pages about error codes in general. Keyword search looks for that exact string.
• Unfamiliar material. The BEIR benchmark (Thakur et al., 2021) tested ten retrieval systems on 18 datasets: BM25 held up well as a baseline, and dense retrieval models often did worse than other approaches on data unlike their training data.
• Rare names. On simple questions such as "Where was Arve Furset born?", dense retrievers did far worse than keyword search: they generalise to common names only, unless they saw that type of question in training (Sciavolino et al., 2021).
• There is always a nearest match. The search returns the closest documents even when none of them is relevant, so a cut-off score has to be tested per model.

What is hybrid search?

Hybrid search runs both, BM25 for the exact words and semantic search for the meaning, and merges the two result lists; Anthropic's retrieval setup does exactly this. A simple way to merge is reciprocal rank fusion (Cormack, Clarke and Büttcher, 2009): a document scores 1/(60 + its rank) in each list, and the scores are added up.

Example

Suppose an accountancy firm makes its internal knowledge base searchable. An employee asks "What's our policy on working from home?", and semantic search returns the document "Remote work guidelines", which never uses the words "working from home". Then she searches for form "HR-12": an embedding model may return forms in general, while keyword search finds the exact code. Hybrid search can find both.