Part 8/8 of the series (final) ← Previous | Glossary

It is recommended to read Parts 1-7 first, especially Part 7 on sampling methods and latent variable models. This part covers four generative models: GANs, normalizing flows, autoencoders, and diffusion models.

Chapter 17 Generative Adversarial Networks (GAN)

You can check out kaggle - I’m also a painter↗ (you

Adversarial Training

In plain words: The core idea of a GAN can be understood through the analogy of a “forger vs. authenticator.” The Generator is like someone forging paintings; its goal is to produce fakes that are indistinguishable from the real thing. The Discriminator is like an appraiser; its goal is to tell which paintings are real and which are fakes. The two continuously play against each other—the forger’s skill keeps improving, and the appraiser’s eye keeps sharpening—until eventually the forger can produce works that pass as genuine.

The goal of a generative model is to learn the probability distribution (i.e., the data’s “generative law”) $p_{\text{data}}(\mathbf{x})$ from which samples in the training dataset are drawn. Once the model has learned this distribution, it can generate new samples similar to the training data.

• Parametric Model: We introduce a generative model with learnable parameters $p(\mathbf{x} | \mathbf{w})$, where $\mathbf{x}$ is a vector in the data space (such as pixel values of an image) and $\mathbf{w}$ is the model’s learnable parameter.
• Conditional Generative Model: Sometimes we want to generate samples of a specific type, such as generating an image of a cat. This can be achieved by introducing a condition variable $\mathbf{c}$, turning the model into $p(\mathbf{x} | \mathbf{c}, \mathbf{w})$. $\mathbf{c}$ can be a class label (such as “cat” or “dog”) or a more complex description.

The first class of generative models discussed in this chapter is the Nonlinear Latent Variable Model. Its core idea is to map a simple low-dimensional latent distribution (the distribution followed by the latent variable $\mathbf{z}$) to a complex high-dimensional data distribution through a nonlinear transformation.

Remember the latent variable model from Part 7? GANs use the same framework: a simple latent variable $\mathbf{z}$ is transformed by a neural network into a complex data sample $\mathbf{x}$.

• Latent Space: We introduce a low-dimensional latent variable $\mathbf{z}$ that follows a simple prior distribution $p(\mathbf{z})$, typically chosen to be a standard normal distribution: $$p(\mathbf{z}) = \mathcal{N}(\mathbf{z} | \mathbf{0}, \mathbf{I})$$
• Generator Network: We define a nonlinear function $\mathbf{x} = g(\mathbf{z}, \mathbf{w})$ implemented by a deep neural network, where $\mathbf{w}$ are the network’s weight parameters. This network is called the Generator.
• Implicit Distribution: Together, the generator network $g(\mathbf{z}, \mathbf{w})$ and the prior distribution $p(\mathbf{z})$ implicitly define a distribution $p_G(\mathbf{x})$ over the data space. We cannot write down the analytical form of $p_G(\mathbf{x})$, but we can generate samples that follow $p_G(\mathbf{x})$ by sampling $\mathbf{z}$ from $p(\mathbf{z})$ and computing $g(\mathbf{z}, \mathbf{w})$.

Adversarial network: the generator produces fake images from latent variable z, and the discriminator judges whether the input is real or fake

The figure above shows the overall architecture of a GAN:

• Left: a “Latent Variable” $\mathbf{z}$ is fed into the “Generator” network $g(\mathbf{z}, \mathbf{w})$, producing “Synthetic Images” $\mathbf{x}_{\text{synth}}$.
• Middle: a “Discriminator” network $d(\mathbf{x}, \mathbf{\phi})$. The discriminator receives two types of input: “Real Images” $\mathbf{x}_{\text{real}}$ from the training set and the synthetic images from the generator.
• The discriminator outputs a scalar $t \in [0, 1]$, representing the probability that the input image is a real image.

Adversarial Training

Adversarial training is the core mechanism of a GAN. It introduces a second network—the Discriminator—which plays a game against the generator.

• Discriminator Network: This is a classification neural network $d(\mathbf{x}, \mathbf{\phi})$ with parameters $\mathbf{\phi}$. Its task is to distinguish whether an input image comes from the real dataset $\{ \mathbf{x}_n \}$ or is a synthetic image produced by the generator. The discriminator’s output is a scalar between 0 and 1, representing the probability that the input image is real.
• The Game Process (imagine a showdown between a forger and an appraiser):
  • Generator’s objective: Maximize the discriminator’s error rate. It wants the generated image $\mathbf{x}_{\text{synth}} = g(\mathbf{z}, \mathbf{w})$ to “fool” the discriminator into thinking it is a real image, i.e., $d(g(\mathbf{z}, \mathbf{w}), \mathbf{\phi}) \approx 1$.
  • Discriminator’s objective: Minimize its own error rate. It wants to accurately distinguish real images from synthetic ones: for a real image $\mathbf{x}_{\text{real}}$, it outputs $d(\mathbf{x}_{\text{real}}, \mathbf{\phi}) \approx 1$; for a synthetic image $\mathbf{x}_{\text{synth}}$, it outputs $d(\mathbf{x}_{\text{synth}}, \mathbf{\phi}) \approx 0$.

Training Process

You may ask: how do the forger and the appraiser “train together”? The answer is alternating training—first fix one and train the other, then swap.

1. Training the discriminator (with the generator fixed):

• Sample a batch of real samples $\{ \mathbf{x}_n \}$ from the real dataset.
• Sample a batch of $\mathbf{z}$ from the latent distribution $p(\mathbf{z})$, and obtain a batch of synthetic samples $\{ g(\mathbf{z}_n, \mathbf{w}) \}$ through the generator.
• Compute the discriminator’s loss $L_D$, so that it learns to tell real from fake: $$L_D = -\left[ \frac{1}{N_{\text{real}}} \sum_{n \in \text{real}} \log d(\mathbf{x}_n, \mathbf{\phi}) + \frac{1}{N_{\text{synth}}} \sum_{n \in \text{synth}} \log(1 - d(g(\mathbf{z}_n, \mathbf{w}), \mathbf{\phi})) \right]$$
• Use gradient descent to update the discriminator parameters $\mathbf{\phi}$ to minimize $L_D$.

2. Training the generator (with the discriminator fixed):

• Sample a batch of $\mathbf{z}$ from the latent distribution $p(\mathbf{z})$.
• Obtain synthetic samples $\{ g(\mathbf{z}_n, \mathbf{w}) \}$ through the generator.
• Compute the generator’s loss $L_G$.

• Use gradient descent to update the generator parameters $\mathbf{w}$ to minimize $L_G$.

Gradient comparison of the two loss functions: why use -log(d) instead of log(1-d)

In short: the horizontal axis $d$ is the discriminator’s output (0 = fake, 1 = real). When the generator is poor ($d \approx 0$), the gradient of $\log(1-d)$ is nearly zero (“cannot learn”), whereas the gradient of $-\log(d)$ is large (“learns fast”).

Mode Collapse

Imagine this: You ask someone to draw all kinds of animals, but they find that drawing cats is especially easy to fool the appraiser, so no matter how much you push them, they only ever draw cats. This is mode collapse—the generator “slacks off” and only produces a few kinds of outputs.

Mode collapse is a common and serious problem in GAN training.

• Definition: During training, the generator’s weights may adapt to a state where all latent variable samples $\mathbf{z}$ are mapped onto a very small, valid subset of outputs in the data distribution. In the extreme case, the generator may only produce one or a few fixed outputs $\mathbf{x}$.
• Example: A GAN trained on handwritten digits might learn to only generate the digit “3” while completely ignoring the other digits.
• Consequences: Although the discriminator cannot tell these “3”s apart from real “3”s, it cannot detect that the generator fails to produce the full range of digits. At this point, the discriminator’s output on these “3” samples stabilizes around 0.5 (meaning it is completely unable to distinguish), causing training to stop and the generator to be unable to improve further.
• Cause: This usually occurs when the discriminator becomes too powerful, or when the generator finds a shortcut that can stably “fool” the discriminator.

The Zero-Gradient Problem

In plain words: At the very beginning of training, the forger is so bad that the appraiser can tell real from fake at a glance. The feedback the forger gets is “you’re too fake,” but there is no specific direction for improvement—like a student who scores zero on an exam but the teacher only says “wrong” without telling them where they went wrong. This is the vanishing gradient problem.

A fundamental difficulty in training a GAN is that when the generated distribution $p_G(\mathbf{x})$ and the real data distribution $p_{\text{data}}(\mathbf{x})$ differ greatly, the discriminator is easy to train, and its gradient signal becomes extremely weak.

When the two distributions are far apart, the discriminator is nearly zero in the region of generated samples, causing the gradient to vanish

The figure above shows the problem:

• $p_{\text{data}}(x)$ is the distribution of the real data (e.g., a Gaussian distribution).
• $p_G(x)$ is the generator’s initial distribution (e.g., another Gaussian far away).
• The discriminator function $d(x)$ outputs close to 1 near the real data and close to 0 near the generated samples.

Improvements

To address training difficulties, researchers have proposed various improvement methods.

Least-Squares GAN (LSGAN)

In short: Change the discriminator’s criterion from a “real/fake probability” to a “score,” using the score gap to provide gradients to the generator.

• Use a smoother discriminator function $\tilde{d}(x)$ so that it has stronger gradients in the region of generated samples, thereby providing more effective learning signals to the generator.
• Least-Squares GAN (Least-Squares GAN, LSGAN):
  • Modify the discriminator so that it outputs a real-valued number $d(\mathbf{x}, \mathbf{\phi}) \in \mathbb{R}$ rather than a probability $[0, 1]$.
  • Replace the cross-entropy loss with a least-squares loss: $$\min_{\mathbf{w}} \max_{\mathbf{\phi}} V_{\text{LSGAN}} = \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}}[(d(\mathbf{x}, \mathbf{\phi}) - b)^2] + \mathbb{E}_{\mathbf{z} \sim p(\mathbf{z})}[(d(g(\mathbf{z}, \mathbf{w}), \mathbf{\phi}) - a)^2]$$ where $a$ and $b$ are target values (e.g., $a=0, b=1$). This loss function produces larger gradients when far from the target value, helping to alleviate the vanishing gradient problem.

Input Noise

• Add Gaussian noise to both the real data and the synthetic samples.
• Effect: This makes the data points no longer concentrate at discrete locations but spread out over a small region. This forces the discriminator to learn a smoother decision boundary, thereby avoiding steep, zero-gradient cliffs in the region of generated samples.

Wasserstein GAN (WGAN)

Imagine this: Suppose $p_G$ is a pile of earth and $p_{\text{data}}$ is the target terrain. The Wasserstein distance is the minimum “work” (volume of earth × distance) needed to move that pile of earth to the target location. This distance can tell you “how far apart” even when the two piles of earth do not overlap at all, whereas the original GAN’s loss has already “gone on strike” in this situation (gradient is zero).

• Directly measure the distance (the Wasserstein distance, also known as the Earth Mover’s Distance) between the generated distribution $p_G(\mathbf{x})$ and the real distribution $p_{\text{data}}(\mathbf{x})$, and let the generator minimize this distance.
• Implementation (WGAN):
  • Use a discriminator that outputs real values (now called the Critic).
  • Use Gradient Penalty to constrain the norm of the discriminator function’s gradient $\nabla_{\mathbf{x}} d(\mathbf{x}, \mathbf{\phi})$ (to keep it close to 1), which guarantees that the discriminator is 1-Lipschitz continuous (i.e., its rate of change is bounded above), so that it can approximate the Wasserstein distance.

Image Generative Adversarial Networks

Deep Convolutional GAN (DCGAN)

Remember the CNN from Part 5? DCGAN is simply applying the convolutional structure of a CNN to a GAN.

DCGAN is an important milestone in the history of GANs, proving that GANs can generate high-quality images.

• Architecture: Uses a standard convolutional neural network (CNN) as the generator and discriminator.
  • Generator: Starts from a low-dimensional latent vector $\mathbf{z}$ and gradually generates a high-resolution image through a series of Transposed Convolution (a kind of “reverse convolution” used to upscale a small image) layers.
  • Discriminator: A standard CNN classifier that takes an image as input and outputs a scalar probability.
• Training stability: Through techniques such as batch normalization and appropriate nonlinear activation functions (such as ReLU and Leaky ReLU), DCGAN trains more stably than early GANs.

CycleGAN

Imagine this: You have many photos of horses and photos of zebras, but no paired photos of “the same horse turned into a zebra.” CycleGAN can still learn to turn horses into zebras! Its trick is “cycle consistency”—turn a horse into a zebra and then back into a horse, and you should get the original horse.

CycleGAN is a model for Unpaired Image-to-Image Translation.

• Problem: We have two image domains $X$ (e.g., photos of horses) and $Y$ (e.g., photos of zebras), but no paired samples (i.e., no paired images of “the same horse turned into a zebra”).
• Goal: Learn a mapping $G_{X \to Y}$ that can transform an image from domain $X$ into domain $Y$, and vice versa $G_{Y \to X}$.
• Adversarial loss:
  • Introduce two discriminators $D_X$ and $D_Y$.
  • $D_Y$ tries to distinguish real $Y$-domain images from images generated by $G_{X \to Y}(X)$.
  • $D_X$ tries to distinguish real $X$-domain images from images generated by $G_{Y \to X}(Y)$.
• Cycle Consistency Loss:
  • To ensure the translation is reasonable, a cycle consistency constraint is introduced. An image $\mathbf{x}$ from domain $X$ transformed into domain $Y$ and then back into domain $X$ should yield an image very similar to the original image $\mathbf{x}$ (“after a round trip, it is still the same”).

Information flow through CycleGAN. The total error of data points xn and yn is the sum of the errors of their 4 components

Semantic Structure of the Latent Space

A surprising finding is that the latent space $\mathbf{z}$ of a trained GAN spontaneously organizes itself into a semantically meaningful structure.

• Smooth interpolation: If you walk along a smooth path in the latent space from $\mathbf{z}_1$ to $\mathbf{z}_2$, the generated images will smoothly transition from one scene to another, and the intermediate images all look reasonable.

Obtained by smoothly moving between randomly generated positions inside the latent space

• Disentangled Representation: One can find specific directions in the latent space that correspond to semantically meaningful transformations—such as changing the degree of smiling, changing hair color, changing age, etc.
  • Example (human faces):

Chapter 17 Summary

One-sentence version: A forger (generator) and an authenticator (discriminator) keep playing against each other, until eventually the forger can produce works that pass as genuine.

Knowledge map:

• Core idea: The minimax game between generator and discriminator
• Key problems: Mode collapse, vanishing gradients, training instability
• Improvement path: LSGAN (smooth loss) -> WGAN (Wasserstein distance) -> CycleGAN (unpaired translation)
• Interesting property: The latent space has a semantic structure and allows “arithmetic” (e.g., smiling man - man + woman = smiling woman)

Chapter 18 Normalizing Flows

Imagine this: You have a ball of clay with a very simple shape (say, a sphere). A normalizing flow is a series of reversible “pinch” and “stretch” operations that turn this simple ball of clay into any complex shape you want (say, a horse). The key point is that every step is reversible—you can also pinch the horse back into a ball. This is the core idea of normalizing flows: through a series of reversible transformations, “morph” a simple distribution into a complex distribution.

Remember the change-of-variables formula from Part 7? The mathematical foundation of normalizing flows is exactly the change of variables in probability theory: when you apply a reversible transformation to a random variable, the probability density of the new variable can be computed via the Jacobian determinant (which measures how much the transformation scales volume).

The goal of this chapter is to design a class of invertible functions that can serve as transformation layers in a generative model. These models are called Normalizing Flows. The key requirements are:

• Invertibility: Given the output $\mathbf{x}$, the input $\mathbf{z}$ must be computable uniquely.
• Easily computable Jacobian determinant: To compute the density transformation, we need to know the determinant $|\det \mathbf{J}|$ of the transformation’s Jacobian matrix $\mathbf{J}$. This computation must be efficient.

Consider a simple linear transformation:

Its inverse transformation is:

Although it is invertible and the Jacobian determinant is easy to compute ($|\det \mathbf{J}| = |a|$), it has a fundamental problem: linear transformations are closed under composition. This means that no matter how many linear transformation layers you stack, the overall effect is still a linear transformation. Just as no matter how you stack stretching and rotation, the shape of the clay is essentially still simple. Therefore, even with multiple layers, you can only generate Gaussian distributions and cannot capture complex, non-Gaussian data distributions.

Coupling Flow

Imagine this: You split the ball of clay into left and right halves. The left half stays put first, and the right half is “pinched” according to the shape of the left half—if the left is round, the right is stretched; if the left is flat, the right is compressed. Then you swap roles: the right stays put, and the left is “pinched” according to the right. After alternating a few times, the ball of clay becomes a complex shape!

To solve the insufficient expressiveness of linear transformations, we introduce the Coupling Flow, using Real NVP (Real-valued Non-Volume Preserving) as an example.

• Core idea: Split the latent variable vector $\mathbf{z}$ into two parts $\mathbf{z} = (\mathbf{z}_A, \mathbf{z}_B)$, with dimensions $d$ and $D-d$. Apply different transformations to the two parts.
• Transformation rules:
  1. The first part $\mathbf{z}_A$ is copied directly to the output (the “unchanged half”): $$\mathbf{x}_A = \mathbf{z}_A$$
  2. The second part $\mathbf{z}_B$ undergoes an affine transformation (scaling + translation), but the transformation’s parameters (translation $b$ and scaling $s$) are nonlinear functions of $\mathbf{z}_A$: $$\mathbf{x}_B = \exp(s(\mathbf{z}_A, \mathbf{w})) \odot \mathbf{z}_B + b(\mathbf{z}_A, \mathbf{w})$$ where $\odot$ denotes the Hadamard Product (element-wise multiplication). $s(\mathbf{z}_A, \mathbf{w})$ and $b(\mathbf{z}_A, \mathbf{w})$ are scaling and translation vectors computed by a neural network. The exponential function $\exp(\cdot)$ ensures the scaling factor is positive, guaranteeing the transformation is invertible.

Single-layer structure of the Real NVP normalizing flow model

Stacking Multiple Layers

A single layer of Real NVP has an obvious limitation: $\mathbf{z}_A$ is left unchanged in the transformation (“half the clay is not pinched”). This limits the model’s expressiveness.

• Solution: Stack multiple coupling layers and swap the roles of $\mathbf{z}_A$ and $\mathbf{z}_B$ at each layer.
  • First layer: $\mathbf{z}_A$ unchanged, $\mathbf{z}_B$ transformed.
  • Second layer: $\mathbf{z}_B$ unchanged, $\mathbf{z}_A$ transformed.
• Effect: Through this alternation, each part of the variables will be transformed at different layers. After stacking enough layers, the model can learn very complex, nonlinear invertible transformations.

Stacked coupling flows: through alternately swapping roles, all variables get transformed

Autoregressive Flow

Imagine this: When painting, you first paint the sky ($x_1$), then decide what color to paint the grass ($x_2$) based on the color of the sky, then decide what color to paint the house ($x_3$) based on the sky and the grass… The value of each pixel depends on the pixels already painted before it. This is the idea of “autoregression.”

The inspiration for autoregressive flows comes from a basic fact in probability theory: any multidimensional joint distribution can be decomposed into a product of a series of conditional distributions.

• Joint distribution decomposition: For a $D$-dimensional vector $\mathbf{x} = (x_1, x_2, ..., x_D)$, its joint probability can be written as: $$p(\mathbf{x}) = p(x_1, x_2, ..., x_D) = \prod_{i=1}^D p(x_i | x_1, x_2, ..., x_{i-1}) = \prod_{i=1}^D p(x_i | \mathbf{x}_{1:i-1})$$ where $\mathbf{x}_{1:i-1}$ denotes the subvector from $x_1$ to $x_{i-1}$. This requires us to impose a fixed ordering on the variables.

Masked Autoregressive Flow (MAF)

The Masked Autoregressive Flow (MAF) directly uses the above decomposition to construct an invertible generative model.

• Transformation rule: MAF defines a transformation from the latent variable $\mathbf{z}$ to the data $\mathbf{x}$: $$x_i = h(z_i, g_i(\mathbf{x}_{1:i-1}, \mathbf{w}_i))$$
  • $z_i$ is the $i$-th component of the latent variable $\mathbf{z}$.
  • $g_i(\mathbf{x}_{1:i-1}, \mathbf{w}_i)$ is a conditioner, usually a neural network, that predicts the transformation’s parameters based on the preceding $x_1, ..., x_{i-1}$.
  • $h(\cdot, \cdot)$ is a coupling function that combines $z_i$ and the conditioner’s output to produce $x_i$. This function must be invertible with respect to $z_i$.
  • Key point: $x_i$ depends only on $z_i$ and the preceding $x_1, ..., x_{i-1}$, not on the following $x_{i+1}, ..., x_D$.
• Masking: To implement this “depends only on preceding variables” constraint in a neural network, we can use a mask. By setting specific values to zero in the neural network’s weight matrix, we force the network to ignore the information of $x_j (j \geq i)$ when computing $x_i$.

MAF vs IAF: A Trade-off in Efficiency

In plain words: MAF and IAF are like two sides of the same coin. MAF is “fast at computing probability, slow at generating,” while IAF is “fast at generating, slow at computing probability.” Which one you choose depends on your task.

Masked Autoregressive Flow (MAF)—efficient likelihood computation, inefficient sampling:

• Given a data point $\mathbf{x}$, we can efficiently compute its log-likelihood $\log p(\mathbf{x})$—because all $z_i$ can be computed in parallel ($\mathbf{x}$ is known).
• But sampling from MAF is very slow, because the generation process must follow the strict order $x_1 \to x_2 \to ... \to x_D$, with each step having to wait for the previous step to complete, giving a time complexity of $O(D)$.

Inverse Autoregressive Flow (IAF)—efficient sampling, inefficient likelihood computation:

• The transformation rule becomes $x_i = h(z_i, g_i(\mathbf{z}_{1:i-1}, \mathbf{w}_i))$—the conditioner depends on the latent variables $\mathbf{z}_{1:i-1}$ rather than the data variables $\mathbf{x}_{1:i-1}$.
• Since $\mathbf{z}$ can be sampled all at once, all $x_i$ can be computed in parallel, making sampling very fast.
• But computing the likelihood (the inverse transformation) requires serial solving, because computing $z_i$ requires knowing $\mathbf{z}_{1:i-1}$ first.

Two autoregressive normalizing flow structures, (a) masked autoregressive flow allows efficient evaluation of the likelihood function, (b) inverse autoregressive flow allows efficient sampling

Basis for choosing:

• If you need to quickly evaluate the probability of a data point (e.g., density estimation, anomaly detection), choose MAF.
• If you need to quickly generate large numbers of samples (e.g., image generation), choose IAF.

Relationship to coupling flows: Autoregressive flows and coupling flows are closely related. A coupling flow can be seen as a special case of an autoregressive flow—it splits the variables into two groups ($A$ and $B$) rather than $D$ groups. This sacrifices some expressiveness, but greatly improves computational efficiency (since the transformations within each group can be parallelized).

Continuous Flows

Neural ODE

Imagine this: A normal neural network is like walking up stairs—one step at a time, jumping discretely. A Neural ODE, on the other hand, is like taking an elevator—rising smoothly and continuously.

Traditional neural networks and normalizing flow models are all composed of discrete layers. The Neural ODE (Neural Ordinary Differential Equation) offers a brand-new perspective: viewing the network as a continuous dynamical system.

• Core idea: Instead of defining a series of discrete transformations $\mathbf{z}_0 \to \mathbf{z}_1 \to \dots \to \mathbf{z}_L$, we define a vector field (Vector Field, describing the direction and speed of motion at each point) $\mathbf{f}(\mathbf{z}(t), \mathbf{w})$ over a continuous time $t$, where $\mathbf{z}(t)$ is the state vector at time $t$.
• Differential equation: The evolution of the state $\mathbf{z}(t)$ is described by an ordinary differential equation (ODE): $$\frac{d\mathbf{z}(t)}{dt} = \mathbf{f}(\mathbf{z}(t), \mathbf{w})$$ where $\mathbf{f}(\cdot)$ is a function implemented by a deep neural network, and $\mathbf{w}$ are its parameters.
• Initial and final states: Given the initial state $\mathbf{z}(0)$, we can integrate this ODE from $t=0$ to $t=T$ to obtain the final state $\mathbf{z}(T)$: $$\mathbf{z}(T) = \mathbf{z}(0) + \int_0^T \mathbf{f}(\mathbf{z}(t), \mathbf{w}) dt$$
• Analogy with traditional networks: This is similar to a residual network (ResNet): $$\mathbf{z}_{l+1} = \mathbf{z}_l + \mathbf{f}(\mathbf{z}_l, \mathbf{w}_l) \Delta t$$ When the step size $\Delta t \to 0$, the discrete residual connection converges to a continuous ODE.

Since $\mathbf{f}(\mathbf{z}(t), \mathbf{w})$ is a complex neural network, the ODE usually cannot be solved analytically. We need to use a numerical ODE solver (such as the Runge-Kutta method).

• Process:
  1. Define the ODE function $\mathbf{f}(\mathbf{z}(t), \mathbf{w})$.
  2. Provide the initial condition $\mathbf{z}(0)$.
  3. Specify the integration interval $[0, T]$.
  4. Call a black-box ODE solver (e.g., scipy.integrate.solve_ivp) to compute $\mathbf{z}(T)$.
• Adaptivity of the solver: Advanced solvers (such as dopri5) are adaptive. They automatically choose the integration step size and time points $t$ based on the complexity of the function $\mathbf{f}$. This makes the solving process more efficient and more precise.

Backpropagation (Adjoint Method)

The main challenge in training a Neural ODE is computing the gradient of the loss function $L$ with respect to the parameters $\mathbf{w}$. Chen et al. (2018) proposed an efficient Adjoint Method, whose memory cost is independent of network depth—a huge advantage.

Neural ODE Flow

A Neural ODE can be used to build a new type of normalizing flow model, called a continuous normalizing flow.

• Transformation: The transformation from input $\mathbf{z}(0)$ to output $\mathbf{z}(T)$ is defined by the ODE: $$\frac{d\mathbf{z}(t)}{dt} = \mathbf{f}(\mathbf{z}(t), \mathbf{w})$$
• Density transformation: If we define a simple prior distribution $p(\mathbf{z}(0))$ in the input space (such as a standard normal distribution), then through the ODE’s evolution, this distribution propagates to the output space $p(\mathbf{z}(T))$.
• Evolution of the log density: Chen et al. (2018) proved that the log probability density $\log p(\mathbf{z}(t))$ evolves according to another ODE: $$\frac{d \log p(\mathbf{z}(t))}{dt} = -\text{Tr}\left( \frac{\partial \mathbf{f}}{\partial \mathbf{z}(t)} \right)$$ where $\partial \mathbf{f} / \partial \mathbf{z}$ is the Jacobian matrix of $\mathbf{f}$ with respect to $\mathbf{z}$, and $\text{Tr}(\cdot)$ denotes the trace of a matrix.
• Training: To compute the log-likelihood $\log p(\mathbf{z}(T))$ of the output $\mathbf{z}(T)$, we need to solve two ODEs simultaneously:
  1. $d\mathbf{z}/dt = \mathbf{f}(\mathbf{z}, \mathbf{w})$, from $\mathbf{z}(0)$ to $\mathbf{z}(T)$.
  2. $d\log p/dt = -\text{Tr}(\partial \mathbf{f} / \partial \mathbf{z})$, from $\log p(\mathbf{z}(0))$ to $\log p(\mathbf{z}(T))$. This can be achieved by concatenating the two equations into an augmented system, which is then solved by a single ODE solver.
• Sampling: When sampling from $p(\mathbf{z}(T))$, first sample $\mathbf{z}(0)$ from $p(\mathbf{z}(0))$, then obtain $\mathbf{z}(T)$ by solving the ODE.

How a simple Gaussian distribution becomes a multi-modal distribution at time t=T through a continuous transformation. When flow lines diverge, density decreases; when flow lines converge, density increases

Chapter 18 Summary

One-sentence version: Through a series of invertible transformations, “morph” a simple distribution (such as a Gaussian) into a complex data distribution, just like pinching clay into any shape.

Knowledge map:

• Core idea: Invertible transformation + change-of-variables formula (Jacobian determinant)
• Three architectures: Coupling flow (Real NVP, split into two halves and alternate transformations), autoregressive flow (MAF/IAF, transform dimension by dimension conditionally), continuous flow (Neural ODE, continuous transformation)
• Key trade-off: MAF is “fast at probability, slow at generating,” IAF is “fast at generating, slow at probability”
• Comparison with GAN: Normalizing flows can compute likelihood exactly (GANs cannot), but expressiveness is limited by the invertibility constraint

Chapter 19 Autoencoder

Imagine this: An autoencoder is like “lossy compression.” You compress a high-resolution photo into a small file (encoding), then restore a photo from that small file (decoding). The compression process is “extracting the essence,” and the restoration process is “rebuilding from the essence.” If the restored photo after compression and restoration looks very much like the original, it means the encoder has learned the core features of the data.

Deterministic Autoencoder

An autoencoder is a neural network model aimed at learning effective representations of data. Its core idea is to reconstruct the input data through an “encode-decode” process.

• Network structure: A typical autoencoder consists of two parts:
  1. Encoder: Maps the input vector $\mathbf{x}$ to a latent representation (Latent Representation, also called hidden representation) $\mathbf{z}(\mathbf{x})$.
  2. Decoder: Maps the latent representation $\mathbf{z}$ back to the output vector $\mathbf{y}(\mathbf{z})$.
• Training objective: The number of output units equals the number of input units. The training objective is to make the network’s output $\mathbf{y}$ as close as possible to the original input $\mathbf{x}$, thereby learning an identity mapping.
• Latent representation: After training, the hidden layer inside the network (i.e., $\mathbf{z}$) provides a compressed or abstract representation of the input data, which can be used for downstream tasks (such as classification, clustering).

If no constraints are imposed on the network, the simplest solution is for the network to learn an identity function (copying the input directly to the output), which is like “compressing” but the file size does not change—completely useless. To force the network to learn meaningful representations, some form of constraint must be introduced.

• Introducing constraints: Common constraint methods are:
  1. Dimensionality constraint: Limit the dimension $M$ of the latent representation $\mathbf{z}$ to be smaller than the dimension $D$ of the input $\mathbf{x}$ (i.e., $M < D$). This forces the network to perform dimensionality reduction (Dimensionality Reduction), learning the most important features of the data—like compressing 100-dimensional data into 10 dimensions.
  2. Sparsity Constraint: Even if $M \geq D$, regularization (such as L1 regularization) can be used to encourage most elements of $\mathbf{z}$ to be zero, thereby learning a sparse representation.
  3. Denoising constraint: By adding noise to the input, train the network to recover the original input from the corrupted input. This forces the network to learn the intrinsic structure and robustness of the data.

Linear Autoencoder

Consider the simplest autoencoder: an input layer, one hidden layer ($M < D$), and an output layer, with all activation functions being linear.

• Error function: Usually the squared error is used to measure the reconstruction error: $$E(\mathbf{w}) = \frac{1}{2} \sum_{n=1}^N \| \mathbf{y}(\mathbf{x}_n, \mathbf{w}) - \mathbf{x}_n \|^2$$
• Equivalence to PCA: When the hidden layer uses a linear activation function, the optimal solution of this autoencoder is equivalent to Principal Component Analysis.
  • At the global minimum of the error function, the network projects the input data onto the subspace spanned by the first $M$ principal components of the data.
  • The weight vectors connected to the hidden units form a basis of the principal subspace (although they are not necessarily orthogonal or normalized).
• Limitation of nonlinear activation functions: Even if a nonlinear activation function (such as sigmoid) is used in the hidden layer, for this two-layer network structure, the optimal solution is still equivalent to linear PCA. This shows that relying solely on a nonlinear activation function cannot break through the limitation of linear dimensionality reduction.

Deep Autoencoder

To achieve true nonlinear dimensionality reduction, deeper network structures need to be introduced.

• Nonlinear mapping: The network can be viewed as two consecutive mappings $F_1$ and $F_2$:
  • $F_1$: Maps the $D$-dimensional input space to the $M$-dimensional latent space $S$. Due to the presence of nonlinear layers, this mapping can be very complex and is no longer limited to linear transformations.
  • $F_2$: Maps the $M$-dimensional latent space $S$ back to the $D$-dimensional output space.
• Geometric interpretation: As shown in the figure below, $F_2$ defines how the latent manifold $S$ is embedded into the high-dimensional data space. Since $F_2$ is nonlinear, this embedding can be non-planar, thereby capturing complex nonlinear structures in the data.
• Training challenges: Unlike the linear autoencoder, the error function $E(\mathbf{w})$ of a deep autoencoder is no longer a quadratic function of the parameters $\mathbf{w}$, so the optimization process is nonlinear. This requires computationally expensive nonlinear optimization techniques, and there is a risk of falling into local optima.

Geometric interpretation

Sparse Autoencoder

Another way to constrain is to encourage the sparsity of the latent representation $\mathbf{z}$ through regularization.

• L1 regularization: Add an L1 norm term to the error function, penalizing the sum of the absolute values of the latent unit activations: $$E(\mathbf{w}) = E(\mathbf{w}) + \lambda \sum_{k=1}^K |z_k|$$ where $E(\mathbf{w})$ is the original unregularized error (such as squared error), $K$ is the number of hidden units, and $\lambda$ is the regularization coefficient.
• Effect: L1 regularization tends to produce sparse solutions, i.e., most $z_k$ values are zero or close to zero. This forces the network to use only a few hidden units to represent the input, thereby learning more interpretable features.
• Gradient computation: Although the regularization term acts on unit activations rather than network parameters, gradients can still be computed efficiently for training through automatic differentiation.

Denoising Autoencoder

Remember the diffusion model later on? The core idea of the denoising autoencoder is very similar to that of the diffusion model—both are “add noise to the data, then learn to remove the noise.” It can be said that the denoising autoencoder is the intellectual predecessor of the diffusion model.

A denoising autoencoder learns robust representations of data by learning to recover the original input from a corrupted input.

• Training process:
  1. Take an original input vector $\mathbf{x}_n$.
  2. Apply noise to it (such as randomly zeroing some inputs, or adding Gaussian noise), obtaining a corrupted version $\tilde{\mathbf{x}}_n$.
  3. Feed $\tilde{\mathbf{x}}_n$ as input to the autoencoder, obtaining the output $\mathbf{y}(\tilde{\mathbf{x}}_n, \mathbf{w})$.
  4. The training objective is to minimize the error between the output and the original, uncorrupted input $\mathbf{x}_n$: $$E(\mathbf{w}) = \sum_{n=1}^N \| \mathbf{y}(\tilde{\mathbf{x}}_n, \mathbf{w}) - \mathbf{x}_n \|^2$$
• Learning mechanism: By learning to “denoise,” the network is forced to discover the statistical regularities and intrinsic structure in the data. For example, in image data, it learns that adjacent pixels are highly correlated, so it can use the information of surrounding pixels to repair the corrupted pixels.
• Relationship with score matching: The training of a denoising autoencoder is closely related to Score Matching. The score function $s(\mathbf{x}) = \nabla_{\mathbf{x}} \ln p(\mathbf{x})$ is an “arrow” pointing toward regions of high data density. The reconstruction direction $\mathbf{y}(\tilde{\mathbf{x}}) - \tilde{\mathbf{x}}$ learned by the denoising autoencoder also points toward the data manifold, similar to the score vector. This connection will reappear in the diffusion model chapter.

In a denoising autoencoder, data points are assumed to lie on a low-dimensional manifold in the data space, and are corrupted by additive noise. The autoencoder learns to map the corrupted data points back to their original values, so the autoencoder learns a vector pointing toward the manifold for every point in the data space

Masked Autoencoder (MAE)

Remember the Transformer from Part 6? MAE applies the Transformer to images—it splits the image into small patches (patch), just like splitting text into tokens, then randomly masks out most of the patches and lets the model learn to recover them.

The Masked Autoencoder is a special form of the denoising autoencoder, especially suitable for images and Transformer-based architectures.

• Masking operation: The input image is divided into multiple “patches.” During training, a portion of the patches is randomly selected and “masked” or discarded (e.g., 75%). Unlike BERT, instead of replacing them with a fixed “mask token,” these patches are directly omitted.
• Architecture: Usually combined with a vision Transformer.
  • Encoder: Receives the unmasked patches as input and computes their representations.
  • Decoder: Needs to reconstruct the entire image (including the masked patches). To achieve this, between the encoder and the decoder, the masked patches must be re-inserted using a fixed mask token vector (with positional encoding attached) to restore the original sequence length.
• Training objective: The loss function only computes the reconstruction error on the masked patches (such as mean squared error). The decoder is trained to predict the original pixel values of these missing patches.
• Advantages:
  1. Computational efficiency: Since the encoder only processes the unmasked patches (e.g., 25%), the computation is greatly reduced.
  2. Powerful representation learning: By predicting large missing regions of the image, the encoder is forced to learn the global semantics and structural information of the image.
• Downstream tasks: After training, the decoder is discarded. The encoder (possibly with new output layers attached) is used for downstream tasks such as image classification and detection.

Architecture of the masked autoencoder in the training phase. After training, the decoder is discarded, and the encoder is used to map images to internal representations for use in subsequent tasks

The masked image is on the left (80% of input patches masked), the reconstructed image is in the middle, and the original image is on the right

Variational Autoencoder (VAE)

Imagine this: A normal autoencoder is “lossy compression”—it compresses a picture into a fixed vector. A VAE goes further: instead of compressing it into a fixed vector, it compresses it into a probability distribution (such as a Gaussian distribution with “mean here, and variance this large”). To generate a new picture, you only need to randomly sample a point from this distribution, then decode it.

Remember variational inference and ELBO from Part 7? The mathematical foundation of the VAE is exactly variational inference—using a simple distribution $q(\mathbf{z})$ to approximate the complex posterior distribution $p(\mathbf{z}|\mathbf{x})$, and maximizing the ELBO.

The Variational Autoencoder (VAE) is a probabilistic generative model that approximates a complex posterior distribution through variational inference.

• Generative model: A VAE defines a latent variable model—the way to generate data is: first sample $\mathbf{z}$ from a simple distribution, then generate $\mathbf{x}$ through the decoder: $$p(\mathbf{x}|\mathbf{w}) = \int p(\mathbf{x}|\mathbf{z}, \mathbf{w}) p(\mathbf{z}) d\mathbf{z}$$ where $p(\mathbf{z})$ is the prior distribution (usually a standard normal distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$), and $p(\mathbf{x}|\mathbf{z}, \mathbf{w})$ is the likelihood function defined by the decoder neural network.
• Difficulty of posterior inference: Directly computing the posterior distribution $p(\mathbf{z}|\mathbf{x}, \mathbf{w})$ is difficult, because the marginal likelihood $p(\mathbf{x}|\mathbf{w})$ cannot be computed analytically (it requires integrating over all possible $\mathbf{z}$).

To solve the difficulty of posterior inference, the VAE uses the Evidence Lower Bound (ELBO) to approximately maximize the log-likelihood.

• Optimization objective: Maximize the ELBO $\mathcal{L}(\mathbf{w})$, which is equivalent to minimizing the KL divergence (a measure of the difference between two distributions) between the approximate posterior $q(\mathbf{z})$ and the true posterior $p(\mathbf{z}|\mathbf{x}, \mathbf{w})$.

Amortized Inference

For efficient learning, the VAE uses a neural network (the encoder) to approximate the posterior distribution for all data points.

• Encoder network: Introduce a parameterized encoder network $q(\mathbf{z}|\mathbf{x}, \mathbf{\phi})$, which maps the input $\mathbf{x}$ to a distribution (usually a Gaussian) of the latent variable $\mathbf{z}$.
• Parameterization: A typical encoder outputs a diagonal Gaussian distribution’s mean $\mathbf{\mu}(\mathbf{x}, \mathbf{\phi})$ and variance $\mathbf{\sigma}^2(\mathbf{x}, \mathbf{\phi})$: $$q(\mathbf{z}|\mathbf{x}, \mathbf{\phi}) = \prod_{j=1}^M \mathcal{N}(z_j | \mu_j(\mathbf{x}, \mathbf{\phi}), \sigma_j^2(\mathbf{x}, \mathbf{\phi}))$$
• Joint optimization: Now, the ELBO depends on both the decoder parameters $\mathbf{w}$ and the encoder parameters $\mathbf{\phi}$. The objective is to jointly optimize $\mathcal{L}(\mathbf{w}, \mathbf{\phi})$.

Reparameterization Trick

In plain words: The problem is this—a neural network needs to learn through “backpropagation,” but the “sampling” operation is like a wall that gradients cannot pass through. The clever trick of the reparameterization trick is: separate the “random sampling” from the parameter computation. It is like saying “I don’t sample directly from the distribution, but first sample a fixed random number, then compute the sampling result with a formula”—this way the gradient can flow back through the formula.

Direct gradient descent on the ELBO is difficult, because the sampling operation blocks backpropagation.

• Problem: The ELBO contains an expectation term, which needs to be approximated by Monte Carlo methods (Monte Carlo, i.e., stochastic sampling approximation): $$\frac{1}{L} \sum_{l=1}^L \ln p(\mathbf{x}|\mathbf{z}^{(l)}, \mathbf{w}), \quad \mathbf{z}^{(l)} \sim q(\mathbf{z}|\mathbf{x}, \mathbf{\phi})$$ But $\mathbf{z}^{(l)}$ is sampled from $q$, and $q$ depends on $\mathbf{\phi}$, so the gradient cannot pass through the sampling operation to the encoder.
• Solution: The reparameterization trick separates the randomness from the parameter dependence.
  • For a Gaussian distribution, the sampling can be expressed as: $$\mathbf{z} = \mathbf{\mu} + \mathbf{\sigma} \odot \mathbf{\epsilon}, \quad \mathbf{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$$
  • Now, $\mathbf{z}$ explicitly depends on $\mathbf{\mu}$ and $\mathbf{\sigma}$ (i.e., $\mathbf{\phi}$), while $\mathbf{\epsilon}$ is an independent noise source unrelated to the parameters.
  • This allows the gradient to backpropagate through $\mathbf{\mu}$ and $\mathbf{\sigma}$ to the encoder network.

Fixing the latent variable z to a specific sampled value prevents the error signal from backpropagating to the encoder network

The reparameterization trick allows the error signal to backpropagate to the encoder network by computing from a sample of an independent random variable e instead of sampling z directly

Combining all the above parts, the complete ELBO objective function of the VAE can be written as:

Chapter 19 Summary

One-sentence version: An autoencoder learns to “compress-decompress” data; the VAE upgrades it to a probabilistic version—compressing into a distribution rather than a point, so that it can generate new data.

Knowledge map:

• Deterministic autoencoder: Encoder + decoder, learning meaningful representations through constraints
  • Linear autoencoder ≈ PCA
  • Deep autoencoder: true nonlinear dimensionality reduction
  • Sparse autoencoder: L1 regularization encourages sparsity
  • Denoising autoencoder: recovering original data from noise (connected in idea to the diffusion model)
  • Masked autoencoder: Transformer + large-area occlusion
• Variational Autoencoder (VAE): Encode into a distribution, train with ELBO, reparameterization trick lets gradients propagate
• Comparison with GAN: VAE training is stable but generated images tend to be blurry; GAN training is difficult but generation quality is high

Chapter 20 Diffusion Models

Imagine this: You are calling a friend, and the signal is clear at first (the original image). Then someone gradually adds more and more noise to the phone line (forward noising), until you can no longer hear what the other person is saying (it becomes pure noise). What a diffusion model does is: learn how to step by step restore a clear signal from a pile of noise (reverse denoising).

You may ask: “Anyone can add noise? Removing noise is the hard part, right?” Exactly! The cleverness of the diffusion model lies in: although the forward noising is fixed and does not need to be learned, each step of the reverse denoising is very small, so small that a neural network can easily learn it.

The Diffusion Model is currently a generative model framework that achieves SOTA (State-of-the-Art) results in fields such as image generation. Its core logic can be summarized as “forward noising -> reverse denoising”:

1. Forward Process (Forward Process, encoder): Gradually “corrupt” the training data (such as images) through multiple steps of additive Gaussian Noise, eventually turning the data into samples from a standard Gaussian distribution (completely devoid of original data information);
2. Reverse Process (Reverse Process, decoder): Train a deep neural network to learn to reverse the above noising process—starting from a standard Gaussian sample, denoise step by step, and finally generate new samples consistent with the training data distribution.

Comparison with GAN and VAE: A diffusion model can be viewed as a kind of “Hierarchical VAE”—the distribution of the forward noising is fixed (no need to learn), and only the generative distribution of the reverse denoising needs to be learned. Compared to GANs, it avoids the instability of adversarial training, and training is simple and easy to parallelize; but generating samples requires multiple passes through the decoder (usually hundreds to thousands of steps), so the computational cost is higher.

Forward noising process: starting from the original image x, after T steps of noising we get z_T close to a Gaussian distribution, and the reverse process is to gradually denoise from z_T back to x

Forward Encoder

In plain words: The forward encoder is the “rule for adding noise.” At each step, you sprinkle a bit of Gaussian noise onto the picture (like sprinkling sand onto a photo). One sprinkle is not too blurry, but after hundreds of sprinkles, the photo becomes a field of random sand grains—you can no longer tell what it originally was.

The forward encoder defines “how to gradually add noise to the data.” The whole process is a Markov Chain (i.e., each step depends only on the result of the previous step, not on earlier history), as follows:

For an original image $x$ in the training set, the first noising step mixes $x$ with independent Gaussian noise to obtain the first noisy image $z_1$:

where:

• $\epsilon_1 \sim \mathcal{N}(\epsilon_1 | 0, I)$: standard Gaussian noise (mean 0, identity covariance matrix);
• $\beta_1 \in (0,1)$: Noise Variance, controlling the strength of the first noising step. The smaller $\beta$ is, the weaker the noising, and the smaller the change to the image;
• The role of the coefficients $\sqrt{1-\beta_1}$ and $\sqrt{\beta_1}$: ensures that the mean of $z_1$ is closer to 0 than $x$, and the covariance is closer to the identity matrix $I$ than $x$ (paving the way for the subsequent multi-step noising to eventually converge to a standard Gaussian).

From the perspective of probability distribution, the conditional distribution of $z_1$ given $x$ is a Gaussian distribution:

Repeating the single-step noising $T$ times gives the noisy sequence $z_1, z_2, ..., z_T$, and the noising rule at each step is consistent with the first step:

where $\epsilon_t \sim \mathcal{N}(\epsilon_t | 0, I)$, and each $\epsilon_t$ is an independent new noise.

The corresponding conditional distribution is:

Parameter setting: $\beta_t$ needs to be set manually (called the noise schedule), usually following $\beta_1 < \beta_2 < ... < \beta_T$ (weak noising early, strong noising later), ensuring that when $T$ is large enough, $z_T$ approaches a standard Gaussian.

The forward noising process can be represented as a probabilistic graphical model:

• Shaded node $x$: observed variable (original image, known);
• Blank nodes $z_1, ..., z_T$: latent variables (noisy images, unknown);
• Forward distribution $q(z_t | z_{t-1})$: defines the noising direction (encoder, known);
• Target-to-learn reverse distribution $p(z_{t-1} | z_t, w)$: defines the denoising direction (decoder, $w$ are the network parameters, to be learned);
• Conditional distribution $q(z_{t-1} | z_t, x)$: given the original image $x$, the distribution of inferring $z_{t-1}$ from $z_t$—this is the key to deriving the training objective.

Probabilistic graphical model of the diffusion model: x is the known original image, and z_1 to z_T are the gradually noised latent variables

Diffusion Kernel (Marginal Distribution of $z_t$)

In plain words: The diffusion kernel tells you some good news—during training you don’t need to actually add noise step by step all the way to step $t$; you can get there in one shot by directly generating the step-$t$ noisy image $z_t$ from the original image $x$. This greatly speeds up training!

In the forward process, if you want to directly obtain $z_t$ (without computing the intermediate $z_1,...,z_{t-1}$), you can do so through the “marginal distribution.” The conditional distribution of $z_t$ given $x$ (called the Diffusion Kernel) is a Gaussian distribution:

where $\alpha_t$ is the cumulative product of $\beta$ (the proportion of signal retained):

Training significance: The diffusion kernel lets you jump directly to any step $t$ during training, without running the full forward chain, greatly improving training efficiency.

The diffusion kernel can also be rewritten in noising form (which is very useful when deriving the loss function later):

Note: here $\epsilon_t$ is the “accumulated noise” (the total noise from $x$ to $z_t$), rather than the single-step noising $\epsilon_\tau$.

Limit case: When the number of noising steps $T \to \infty$, $\alpha_T \to 0$, and the diffusion kernel becomes $q(z_T | x) = \mathcal{N}(z_T | 0, I)$—$z_T$ is completely standard Gaussian noise, with nothing to do with the original image $x$. This is what we want: the forward process eventually “erases” all the original information.

Conditional Distribution (the “Ground Truth” for the Reverse Process)

In plain words: We want to train a neural network to “denoise,” but it needs a “ground truth” to learn from. The conditional distribution $q(z_{t-1} | z_t, x)$ is exactly this ground truth—it tells you “if you know what the original image $x$ is, then how should you recover $z_{t-1}$ from $z_t$.”

Our goal is to learn the reverse denoising (from $z_t$ to $z_{t-1}$), but directly inverting $q(z_t | z_{t-1})$ gives $q(z_{t-1} | z_t)$, which requires knowing $p(x)$ (the data distribution) that we don’t know, so it cannot be computed. Therefore, we consider the “reverse conditional distribution given $x$” $q(z_{t-1} | z_t, x)$—the good news is that this distribution is a simple Gaussian!

Significance of this distribution: It provides a “target distribution” for training the neural network—during training, make the reverse distribution $p(z_{t-1} | z_t, w)$ predicted by the network as close as possible to this “ground truth” $q(z_{t-1} | z_t, x)$.

Reverse Decoder

In plain words: The reverse decoder is that “denoiser”—a neural network that takes the noisy image $z_t$ and the current step number $t$ as input and outputs the denoised mean. It is the core part of the diffusion model that needs to be trained.

The core of the reverse decoder is “learning a deep neural network that approximates the reverse denoising process corresponding to $q(z_{t-1} | z_t, x)$,” and through this network generate $x$ (a clear data sample) from $z_T$ (pure noise).

Directly computing $q(z_{t-1} | z_t)$ is infeasible (it requires integrating over all possible $x$), so we model the reverse distribution $p(z_{t-1} | z_t, w)$ with a neural network, in the form of a Gaussian distribution:

where:

• $\mu(z_t, w, t)$: the mean output by the deep neural network ($w$ are the network parameters, and $t$ is input to adapt to different noise intensities);
• Variance set to $\beta_t I$: referencing the variance of the forward noising, and when $\beta_t$ is small, the reverse distribution approximates a Gaussian, with variance close to $\beta_t I$.

On network architecture: The output dimension of the neural network must match the input ($z_t$)—in image data, this is the image size. The commonly used architecture is U-Net—an encoder-decoder structure with skip connections, suitable for capturing both local and global features of images. In recent years, Transformer architectures have also been used.

Choice of $\beta_t$: Why “Small Steps, Slow Walk”?

Imagine this: If you add a lot of noise at once ($\beta_t$ large), it’s like blurring the photo all at once—when denoising, it’s hard to guess what the original was. But if you add only a little noise each time ($\beta_t$ small), the change at each step is small, and the neural network can easily learn to “remove this step’s blur.”

It is recommended to set $\beta_t \ll 1$ (weak noising at each step), for the following reasons:

1. In the forward noising, a small $\beta_t$ means $z_t$ and $z_{t-1}$ differ little, so the “correction magnitude” in reverse denoising is small, making it easier for the network to learn;
2. The reverse distribution $q(z_{t-1} | z_t)$ will approximate a Gaussian:

• Top figure: when $\beta_t$ is large, $q(z_{t-1} | z_t)$ is a multi-modal distribution (complex, hard for the network to model);
• Bottom figure: when $\beta_t$ is small, $q(z_{t-1} | z_t)$ is close to Gaussian (simple, easy to model).

The cost: a large number of steps $T$ (usually hundreds to thousands) is needed for $z_T$ to approach a standard Gaussian, causing high computational cost when generating samples. This is the fundamental reason why diffusion models are “fast to train, slow to generate.”

Joint Distribution of the Reverse Process

The entire reverse denoising process is a Markov chain, with joint distribution:

where:

• $p(z_T) = \mathcal{N}(z_T | 0, I)$ (consistent with the forward process’s $q(z_T)$);
• $p(x | z_1, w)$: the distribution of finally generating $x$ (no noise) from $z_1$ (weak noise), in a form similar to $p(z_{t-1} | z_t, w)$ (Gaussian distribution).

Training Objective: Evidence Lower Bound (ELBO)

Remember the VAE’s ELBO? The diffusion model uses exactly the same idea—directly maximizing the log-likelihood is too hard (it requires integrating over all possible noise paths), so we maximize a lower bound of it, the ELBO.

The ideal training objective of a generative model is to maximize the log-likelihood $\ln p(x | w)$ of the data, but this likelihood requires integrating over all latent variables $z_1,...,z_T$, involving complex neural networks, and cannot be computed directly. Therefore, borrowing the idea of the VAE—maximize the lower bound (ELBO) of the likelihood.

For any latent variable distribution $q(z)$, we have:

where $\mathcal{L}(w)$ is the ELBO, and $KL(\cdot \| \cdot) \geq 0$. Therefore $\ln p(x | w) \geq \mathcal{L}(w)$, and maximizing $\mathcal{L}(w)$ indirectly maximizes the likelihood.

Choose $q(z) = q(z_1,...,z_T | x)$ (the fixed distribution of the forward process), substitute into the ELBO and simplify, and finally decompose it into a “reconstruction term” and a “consistency term”: