Diffusion model

A diffusion model is a type of AI that creates images, and now also video, from a text description. During training it sees photos buried in random noise, step by step, and learns to undo that. To make a new picture, it starts from pure noise and removes it in steps, steered by your prompt. Stable Diffusion works this way; a webshop could use it for banners.

How does it work?

The idea comes from Sohl-Dickstein et al. (2015), inspired by physics: slowly destroy the structure in data with a forward diffusion process, then learn the reverse process that restores it. Ho, Jain and Abbeel (2020) turned it into high-quality image generation with DDPM (denoising diffusion probabilistic models):

1. Forward process: take a training image and add a little Gaussian noise at each of 1,000 steps, until only noise remains.
2. Training: a neural network, a U-Net in DDPM, learns to predict the noise that was added at a given step.
3. Reverse process: to generate, start from pure noise and let the network remove its estimate of the noise, step by step, until an image appears.

Each step is a full pass through the network, which makes generation slow. Samplers such as DDIM (2020) produce good images 10 to 50 times faster than DDPM; the diffusers library uses 50 steps by default.

How does a text prompt steer the image?

A text encoder turns the prompt into embeddings, and the denoising network reads them through cross-attention at every step. Stable Diffusion v1 uses OpenAI's CLIP ViT-L/14 as its text encoder (model card); DALL-E 3 uses Google's T5 XXL (DALL-E 3 paper).

Classifier-free guidance (Ho and Salimans, 2022) sets how closely the image follows the prompt. The model is trained both with and without the prompt. At each step it makes one prediction of each kind and pushes the result further in the direction of the prompt; a higher guidance scale means a stronger push and less variety. ControlNet (2023) adds spatial control to a trained model, such as edges, a depth map or a human pose.

Why work in a latent space?

Diffusion on raw pixels often took hundreds of GPU days to train. Rombach et al. (December 2021) moved the process into the compressed latent space of a pretrained autoencoder. In Stable Diffusion, released in August 2022, that autoencoder shrinks each side of the image by a factor of 8: a 512 × 512 image becomes a 64 × 64 grid with four channels. DALL-E 3 works in the latent space of the same autoencoder.

How does diffusion compare with GANs?

Before diffusion, GANs made the best images, but they are often hard to train and can collapse without carefully chosen settings. In 2021, Dhariwal and Nichol showed that diffusion models beat the best GANs on image quality on ImageNet, while covering more of the variety in the data. The price is speed: every image takes many passes through the network.

Where does it go wrong?

The model card of Stable Diffusion v1 (2022) lists its limits: it cannot render legible text, struggles with compositions such as "a red cube on top of a blue sphere", may get faces and people wrong, works worse outside English, reproduces some images that appear many times in its training data, and often treats white and Western as the default. Later models improved on some of this; DALL-E 3 can put text in an image, but unreliably.

Is diffusion only for images?

OpenAI describes its video model Sora (February 2024) as a diffusion transformer (technical report). The technique is also used for text: NVIDIA's Nemotron-Labs Diffusion language models (2026) can generate several tokens at once by diffusion (model card).