How AI Image Generation Works: Diffusion Models Explained

From GANs to Diffusion Models

The field of AI image generation has transformed dramatically in the span of a few years. Before 2022, the dominant approach was Generative Adversarial Networks (GANs) — a framework in which two neural networks (a generator and a discriminator) compete in a game that drives the generator to produce increasingly realistic images. GANs achieved impressive results for specific image types (faces, landscapes) but were notoriously difficult to train stably and struggled with diversity, often collapsing to generate a narrow range of images.

Diffusion models have largely supplanted GANs as the leading paradigm for high-quality image generation. Introduced in their modern form by Ho et al. in the 2020 paper Denoising Diffusion Probabilistic Models (DDPM), diffusion models work on a fundamentally different principle: they learn to reverse a gradual noise corruption process, starting from random noise and progressively refining it into a coherent image. Combined with text conditioning through systems like CLIP and transformer-based text encoders, diffusion models produce images of remarkable quality and diversity from natural language descriptions — the technology powering DALL-E 2, DALL-E 3, Stable Diffusion, Midjourney, and Adobe Firefly.

The Core Idea: Learning to Denoise

The mathematical foundation of diffusion models is a two-process framework. The forward diffusion process (also called the noising process) is fixed and simple: take a real training image and gradually add Gaussian noise over many small steps (typically 1000 steps), until the image is completely destroyed and indistinguishable from pure random noise. Each step adds a small, mathematically defined amount of noise, so the process is entirely deterministic and requires no learning.

The reverse diffusion process is what the model learns: given a noisy image at any step in the corruption sequence, predict and subtract the noise to recover a slightly cleaner version of the image. If the model can learn to perfectly denoise at each step, it can generate new images by starting from pure random noise and running the reverse process for all 1000 steps, progressively converting noise into a coherent image. The model is trained on a massive dataset of images by randomly sampling a training image, randomly sampling a noise level, applying the corresponding forward noise, and then training the model to predict the noise that was added — a denoising objective that turns out to be mathematically equivalent to maximizing the likelihood of the training data.

The U-Net Architecture

The neural network architecture used as the denoising function in most diffusion models is a U-Net — a convolutional neural network with an encoder-decoder structure originally developed for medical image segmentation. The U-Net processes the noisy image through a series of downsampling layers (the encoder, which compresses the image to a lower-resolution feature representation) followed by upsampling layers (the decoder, which reconstructs a full-resolution output). Skip connections between corresponding encoder and decoder layers preserve spatial detail, enabling the network to be precise about both high-level semantic content and low-level texture.

Modern diffusion model U-Nets incorporate transformer-based attention layers (particularly cross-attention) within the architecture, which allow text conditioning to be integrated at multiple scales of the feature representation. The noise level (timestep) is injected into the network at each layer through time embeddings, telling the model at which point in the denoising trajectory it is operating — since the appropriate denoising behavior varies dramatically depending on whether the image is mostly noise (early steps, where the model focuses on global structure) or nearly clean (late steps, where fine details are refined).

Text Conditioning: How Words Become Images

The ability to generate images from text descriptions — text-to-image generation — requires a mechanism for conditioning the denoising process on text input. The most influential approach was introduced in OpenAI's CLIP (Contrastive Language-Image Pre-training), a model trained on 400 million image-text pairs to produce aligned representations in a shared embedding space where semantically related text and images are mapped to nearby points.

In modern text-to-image diffusion models, text prompts are encoded by a powerful text encoder — typically a large language model such as OpenAI's CLIP text encoder, Google's T5, or a custom transformer. The resulting text embeddings are injected into the U-Net through cross-attention layers, where the image features attend to the text embeddings at each denoising step. This allows the text to guide the denoising process at every scale, steering the generation toward image content, composition, style, and attributes described in the prompt. Classifier-Free Guidance (CFG) is a technique that amplifies the text conditioning by training the model both with and without text conditioning and then interpolating between the two at inference time — higher guidance scales produce images more faithful to the text prompt but can sacrifice diversity and photorealism at extreme values.

Latent Diffusion Models and Stable Diffusion

Running diffusion in pixel space for high-resolution images is computationally prohibitive — 1000 denoising steps of a 512x512 image with a large U-Net would require impractical amounts of memory and compute. Latent diffusion models (LDMs), the innovation introduced by Rombach et al. at Stability AI and foundational to Stable Diffusion, solve this by moving the diffusion process into a compressed latent space rather than pixel space.

An LDM uses a separately trained Variational Autoencoder (VAE) to compress images into a lower-dimensional latent representation (typically 8 times smaller in each spatial dimension, so a 512x512 image becomes a 64x64 latent) and reconstruct them from that representation. The diffusion model operates entirely in this latent space, learning to denoise latent vectors rather than pixel arrays. Because the latent space is dramatically smaller than pixel space, LDMs are far more computationally efficient — enabling generation of high-resolution images on consumer hardware, democratizing access to image generation and enabling the open-source Stable Diffusion ecosystem. At generation time, the denoised latent is decoded back to pixel space by the VAE decoder.

Improvements: SDXL, DALL-E 3, and Flow Matching

Diffusion model research has advanced rapidly. Stable Diffusion XL (SDXL) improved on the original by using a larger U-Net, a two-stage generation pipeline (base + refiner model), and a larger text encoder ensemble (CLIP + OpenCLIP) to better handle complex prompts. DALL-E 3 (OpenAI, 2023) achieved dramatically improved prompt following by training on a dataset of images with highly detailed, accurate captions generated by GPT-4V, addressing the chronic problem of diffusion models failing to correctly render text, count objects, or follow complex spatial instructions.

Flow matching (also called rectified flow or continuous normalizing flows) is an alternative to the standard DDPM denoising objective that offers straighter denoising trajectories and more efficient sampling with fewer steps. Models trained with flow matching (such as Stable Diffusion 3 and Flux) can produce high-quality images in as few as 4-20 denoising steps versus the 50-1000 steps required by early diffusion models, dramatically reducing generation time. DiT (Diffusion Transformers), replacing the U-Net backbone with a pure transformer architecture, shows strong scaling properties and underpins several frontier generation models. These advances are collectively driving image generation quality and efficiency toward levels indistinguishable from professional photography for many use cases.

Practical Considerations and Ethics

• Prompt engineering: The quality of generated images is highly sensitive to prompt wording. Detailed descriptions of subject, style, lighting, composition, and medium dramatically improve outputs.
• Negative prompts: Many systems allow specifying what to exclude from an image, helping avoid common artifacts.
• Training data concerns: Diffusion models are trained on web-scraped data including copyrighted images, raising ongoing legal and ethical debates about artistic copyright and compensation for training data contributors.
• Deepfakes and misuse: The ability to generate photorealistic images of real people raises serious concerns about non-consensual imagery and political disinformation. Responsible deployment requires watermarking, provenance standards like C2PA, and platform-level misuse controls.
• Hardware requirements: Running Stable Diffusion locally requires a GPU with at least 6 GB of VRAM for standard resolutions; cloud-based APIs lower the barrier but incur per-generation costs.