Latent space

Latent space is the internal map an AI model builds of the things it processes. Every photo, sentence or product becomes a point on that map, a list of numbers, and things that are alike end up close together. You never see it, but a webshop that suggests similar products or an image generator that turns a text prompt into a picture can both work with such a map.

How does it work?

A neural network turns its input, layer by layer, into lists of numbers. The numbers in a hidden layer form a point in a space with one dimension per number, and that space is the latent space. Latent means hidden: the training data never says what these numbers should be, so the model works them out itself.

An autoencoder makes this explicit. The textbook Deep Learning by Goodfellow, Bengio and Courville describes it as a network trained to copy its input to its output through a hidden code: an encoder turns the input into the code, and a decoder rebuilds the input from it. When the code is smaller than the input, the network is forced to keep the most salient features of the training data. A variational autoencoder (VAE) by Kingma and Welling (2013) also shapes the latent space into a known distribution, so you can pick a new point and decode it into new data.

Where is it used?

• Search by meaning. An embedding is a point in the latent space of a text model, and texts with a similar meaning get nearby points (semantic search). For word vectors, Mikolov et al. (2013) reported that vector("King") minus vector("Man") plus vector("Woman") lands closest to vector("Queen").
• Image generation. Stable Diffusion runs its diffusion process in the latent space of an autoencoder, which takes much less computing than working on pixels (Rombach et al., 2022). A text encoder turns the prompt into numbers that steer each denoising step.
• Walking between points. Decoding points on the line between two latent points gives a gradual transition between two images. White (2016) showed for VAEs and GANs that following a curved path (spherical interpolation) gives sharper images than a straight line.

Where does it go wrong?

The dimensions have no labels. Olah et al. (2017) found that random directions in a network's activation space often look interpretable, but less often than single neurons, and wrote that researchers have almost no clue how to select meaningful directions. Famous examples can also mislead. Nissim, van Noord and van der Goot (University of Groningen, 2019) point out that the standard word2vec analogy code never returns one of the input words; allow that, and the analogy often returns the input word itself. And every model has its own latent space: points from two different models, or from a retrained version of the same model, cannot be compared directly (Hu et al., 2022).

Example

Stable Diffusion v1 works on images of 512 by 512 pixels. Its autoencoder shrinks each side by a factor of 8 and stores 4 numbers per position, so an image of 512 × 512 × 3 colour values (786,432 numbers) becomes a latent of 64 × 64 × 4 (16,384 numbers), 48 times fewer (model card). The prompt enters through a CLIP text encoder and cross-attention. The model removes noise from that small latent step by step, and the decoder then turns the result back into a full-size image.