GAN (generative adversarial network)

A GAN (generative adversarial network) is an AI technique in which two programs train each other. The generator makes fake examples, such as photos of faces; the discriminator tries to tell them apart from real ones. Each round the forger gets better at fooling the checker, until its fakes look real. GANs produce realistic photos of people who do not exist, which also turn up on fake online profiles.

How does it work?

Ian Goodfellow and colleagues introduced GANs in June 2014 (Generative Adversarial Networks). They compare the generator to a team of counterfeiters trying to produce fake currency, and the discriminator to the police trying to detect it. The competition drives both to improve until the counterfeits are indistinguishable from the real thing.

• The generator takes random noise as input and turns it into a sample, such as an image.
• The discriminator receives real samples from the training data and samples from the generator, and estimates for each one the probability that it is real.
• Both neural networks are trained in turns with backpropagation. The discriminator learns to catch fakes; the generator learns from the discriminator's feedback how to make better ones.

In theory the game ends when the generator reproduces the distribution of the training data and the discriminator can only guess: it gives every sample a probability of 1/2. A simplified training step in PyTorch, after the DCGAN tutorial, which trains a GAN on photos of celebrities:

1
z = torch.randn(batch_size, latent_dim)        # random noise
2
fake = G(z)
3
# 1. discriminator: real images count as 1, fakes as 0
4
d_loss = bce(D(real), ones) + bce(D(fake.detach()), zeros)
5
d_opt.zero_grad(); d_loss.backward(); d_opt.step()
6
# 2. generator: try to make D call the fakes real
7
g_loss = bce(D(fake), ones)
8
g_opt.zero_grad(); g_loss.backward(); g_opt.step()

Step 2 follows a tip from the original paper: the generator maximises the chance that its fake is taken for real, instead of minimising the chance that it gets caught, which gives stronger gradients early in training. In GAN Lab you can watch a small GAN train in the browser.

Which GANs are well known?

• StyleGAN (NVIDIA, December 2018) generates faces and separates features such as pose and identity from details such as freckles, so each can be controlled. It came with FFHQ, a dataset of 70,000 face photos at 1024×1024 pixels.
• pix2pix (2016) learns from example pairs to turn one image into another, such as a label map into a photo or black and white into colour.
• CycleGAN (2017) does this without pairs, for example turning horses into zebras.
• SRGAN (2016) enlarges photos four times with realistic detail.

Where does it go wrong?

• Mode collapse: the generator maps many different inputs to the same output and produces only a few kinds of samples. Goodfellow's 2016 tutorial calls complete collapse rare and partial collapse common.
• Non-convergence: training looks for a balance between two players instead of the lowest value of one loss, and an update that helps one player can undo the other's progress. The tutorial calls this the largest open problem.
• No likelihood: a GAN produces samples, but cannot tell you how likely a given sample is.

In 2021 Dhariwal and Nichol (OpenAI) showed that diffusion models produce better images than the best GANs on ImageNet (Diffusion Models Beat GANs on Image Synthesis). GANs capture less of the variety in the data and are often hard to train, they wrote, but diffusion models are slower at generating because they need many denoising steps. Well-known image generators since then, such as Stable Diffusion, use diffusion. Stability AI's Adversarial Diffusion Distillation (November 2023) adds an adversarial loss to a diffusion model so it can make an image in one to four steps.

Example

In February 2019 software engineer Philip Wang launched ThisPersonDoesNotExist.com, which showed a new face made by NVIDIA's StyleGAN each time the page loaded (The Verge). In 2022 researchers at the Stanford Internet Observatory found more than 1,000 LinkedIn profiles with what appeared to be GAN-generated faces, used to send sales messages; more than 70 businesses were listed as their employers (NPR, March 2022). In the first one they spotted, the woman wore one earring, strands of hair disappeared and reappeared, and her eyes sat exactly in the middle of the image.