What Is Generative AI: LLMs, Diffusion Models, and How They Create

A New Kind of AI

For most of AI's history, the dominant paradigm was discriminative: systems were trained to classify inputs, detect objects, predict values, or recommend items. These are powerful capabilities, but they all involve mapping an existing input to a label or score. Generative AI represents a different ambition — teaching machines not just to recognize patterns but to create new content that has never existed before, content that looks, reads, or sounds like something a human might have made.

The term "generative AI" covers a broad family of models, but the two architectures that have defined the current era are large language models (LLMs), which generate text token by token, and diffusion models, which generate images (and increasingly video and audio) by iteratively refining a field of noise. Both emerged from decades of research in deep learning and became commercially viable through a combination of architectural innovation, massive datasets, and unprecedented compute investment in the early 2020s.

Large Language Models: Next-Token Prediction at Scale

At their core, large language models are trained on a deceptively simple objective: given a sequence of text, predict the next token. A token is roughly a word or word-fragment — the text "unforgettable" might be split into the tokens "un", "forget", and "table." During training, the model sees trillions of tokens drawn from books, websites, code repositories, and other text sources. For every position in the sequence, it must output a probability distribution over the vocabulary, and it is penalized when the true next token is not assigned high probability. Through this process, the model learns grammar, facts, reasoning patterns, social conventions, and much more — because all of these are implicit in the statistical regularities of human-generated text.

The architecture underlying virtually all modern LLMs is the Transformer, introduced in 2017. Transformers use self-attention mechanisms to allow every position in the sequence to attend to every other position, weighting the relevance of each context token dynamically. This gives them far better long-range context handling than the recurrent architectures that preceded them. GPT-3, released by OpenAI in 2020 with 175 billion parameters, demonstrated that scaling up Transformers dramatically improved not just performance but also the emergence of qualitatively new capabilities — solving math problems, writing code, translating between languages the model was never explicitly trained on.

How LLMs Generate Text

At inference time, an LLM generates text autoregressively: it samples the next token from its output probability distribution, appends that token to the context, and feeds the extended context back into the model to generate the token after that. This continues until the model generates a special end-of-sequence token or a length limit is reached. The sampling strategy matters significantly: greedy decoding always picks the highest-probability token, producing deterministic but sometimes repetitive output; temperature sampling introduces randomness, with higher temperatures producing more varied and creative output at the cost of coherence; top-p (nucleus) sampling restricts sampling to the smallest set of tokens whose cumulative probability exceeds a threshold, balancing diversity and quality.

Modern LLMs are typically not deployed as raw next-token predictors. They are further refined through instruction tuning — fine-tuning on examples of instructions paired with desired outputs — and reinforcement learning from human feedback (RLHF), which trains the model to produce outputs that human raters rate as helpful, harmless, and honest. These post-training steps are what transform a raw language model into a useful assistant that follows instructions, declines harmful requests, and maintains a coherent conversational persona.

Diffusion Models: Creating Images from Noise

Diffusion models take a fundamentally different approach to generation. During training, the model learns to reverse a process that progressively adds Gaussian noise to an image. Starting with a real image, the forward process adds a small amount of noise at each of hundreds or thousands of steps until the image is indistinguishable from pure noise. The model is trained to predict the noise added at each step, which is equivalent to learning to denoise. At inference time, generation runs in reverse: start with pure noise and iteratively apply the learned denoising function, gradually revealing a coherent image over many steps.

The key insight, formalized in denoising diffusion probabilistic models (DDPMs) and score-based generative models, is that the model never needs to generate the entire image at once. Instead, it makes a large number of small, local corrections to a noisy intermediate state. This divide-and-conquer approach makes high-resolution, high-fidelity image generation tractable. Systems like DALL-E 2, Stable Diffusion, and Midjourney add text conditioning by training the denoising model to take a text embedding — produced by a pre-trained language encoder — as additional input, allowing users to guide generation with natural language prompts.

Other Generative Architectures

LLMs and diffusion models dominate the current landscape, but several other architectures contribute to the generative AI ecosystem. Variational autoencoders (VAEs) encode inputs into a compact latent space and decode latent vectors back to the data domain, enabling controlled interpolation and editing. Diffusion models themselves often work in the latent space of a VAE for computational efficiency — this is the "latent diffusion" approach used by Stable Diffusion. Generative adversarial networks (GANs), the dominant image generation method before diffusion models, use a game between a generator and a discriminator to produce realistic images; while largely superseded for pure generation quality, GANs remain useful in applications requiring fast inference.

For audio and music, specialized architectures like AudioLM, MusicLM, and Meta's AudioCraft combine Transformer-based language models operating on discrete audio tokens with neural audio codecs that convert between waveforms and tokens. Video generation, one of the most computationally demanding generative tasks, is tackled by models like Sora and Runway Gen that extend diffusion and Transformer approaches to temporal sequences. Code generation is handled by LLMs fine-tuned on code (GitHub Copilot, Code Llama), which treat programs as a specialized form of text.

Capabilities and Limitations

Generative AI systems exhibit a remarkable range of capabilities. Modern LLMs can write persuasive essays, debug software, explain scientific concepts, compose poetry, translate between dozens of languages, and engage in sustained multi-turn conversation. Image generation models can render photorealistic scenes, produce artistic illustrations in virtually any style, and edit existing images with surgical precision. These capabilities have genuine economic value: creative professionals, software engineers, researchers, and businesses across every sector are integrating generative AI tools into their workflows.

Yet these systems have well-documented limitations. LLMs hallucinate — confidently asserting false facts with the same fluency they use for true ones — because their training objective optimizes for plausible text, not factual accuracy. They are sensitive to prompt phrasing, can be led astray by adversarial inputs, and struggle with tasks requiring precise numerical reasoning or logical deduction beyond their training distribution. Diffusion models can produce anatomically incorrect images (notoriously, distorted hands), struggle with precise text rendering, and have difficulty with spatial relationships that require genuine scene understanding. Both families of models can reproduce harmful content from their training data if not carefully constrained.

Social and Economic Impact

The rapid deployment of generative AI has sparked intense debate about its broader implications. Copyright and intellectual property questions are actively contested in courts around the world: training on copyrighted text and images without licensing raises unresolved legal questions. The ease of generating realistic synthetic media has amplified concerns about deepfakes and AI-generated misinformation. Labor market economists study which professions are most exposed to automation by generative tools — early evidence suggests white-collar knowledge work faces greater disruption than predicted by earlier automation research.

At the same time, generative AI is accelerating scientific research, enabling medical image synthesis for training diagnostic models, generating candidate drug molecules, and helping researchers write and review code for large-scale data analysis. In education, it is redefining how students draft, revise, and learn from written work. In software development, AI-assisted coding tools increase developer productivity measurably. The governance challenge — how to ensure these powerful tools are deployed safely, equitably, and accountably — is one of the defining policy questions of the decade.