Natural language processing (NLP)

Natural language processing (NLP) is the part of AI that works with human language: software that reads, sorts, translates or writes text, and sometimes listens to speech. A customer service team meets it when incoming emails are sorted by topic automatically, and a webshop when its chatbot understands a question typed in everyday words. Translation apps and the spam filter in your mailbox use it too.

Where does it come from?

The field goes by many names, according to the textbook by Jurafsky and Martin (draft of August 2026), including computational linguistics and, since the rise of language models, simply language modelling. Its roots lie in the late 1940s to early 1960s, when Claude Shannon worked on predicting the next word (1951). From about 1965 to the early 1990s, symbolic approaches with explicit rules and structures dominated. ELIZA (1966), Joseph Weizenbaum's program that imitated a therapist with simple text substitutions, dates from that time. From the late 1980s, statistical methods spread from speech recognition to text. Until 2019 the standard approach was to label training data and train a classifier for each task.

How does it work now?

The transformer (Vaswani et al., 2017) was invented for machine translation and quickly became a general-purpose architecture. Google's BERT (2018) was first pre-trained on unlabelled text and then adapted to a task with one extra output layer; it set new best results on eleven NLP tasks. Jurafsky and Martin see prompting, from 2019, as the next turning point: instead of training a classifier per task, you type a question and a language model predicts the answer. Since then NLP has largely been about generating text, with large language models.

A language model predicts the next word from the words before it. The text is first split into tokens, which the model turns into embeddings: lists of numbers that represent meaning. To write, it picks a next token according to its probability, adds it to the text and repeats.

Which tasks does it cover?

• Text classification: giving a text a category, such as spam or not spam, or the positive or negative tone of a product review (sentiment analysis).
• Named entity recognition: finding the names of people, places and organisations in a text.
• Machine translation, summaries and answers to questions. With prompting, one language model can do many of these.
• Speech recognition and speech synthesis, which the textbook also counts as part of the field.

How well does it handle Dutch?

Joshi et al. (Microsoft Research India, 2020) sorted 2,485 languages into six classes by the amount of labelled and unlabelled data available. Dutch is in class 4, one class below English and German: plenty of unlabelled text, less labelled data. Models trained on Dutch alone help: BERTje (University of Groningen, 2019), trained on 2.4 billion tokens, beat the multilingual BERT model of the same size on tasks such as named entity recognition and sentiment analysis. RobBERT (KU Leuven, 2020) followed.

Where does it go wrong?

• Language is ambiguous. In Jurafsky and Martin's example, "JFK" can be a person, the airport in New York or one of many schools, bridges and streets.
• Translation misses context. When Google put a neural network into its translation service in 2016, it noted that the system could still drop words, mistranslate names or rare terms, and translate sentences one at a time without the context of the page.
• Language models can hallucinate: write text that is factually wrong, such as people or facts that do not exist.

Example

Google Translate launched in 2006 with phrase-based translation, which cuts a sentence into words and phrases and translates them largely independently. In September 2016 Google moved Chinese to English, about 18 million translations a day, to a neural network that handles the whole sentence as one unit. Bilingual raters found 55% to 85% fewer translation errors on several major language pairs. In June 2020 it switched to a hybrid: a transformer encoder that reads the sentence and a faster decoder of the older recurrent type (RNN) that writes the translation.