Deep Learning Notes: Glossary

This is the companion glossary for the Deep Learning Notes series. Each term comes with a plain-language explanation and a link to the article where it first appears, for easy reference anytime.

It’s recommended to use this alongside the series articles — the glossary helps you recall quickly, while the articles help you understand deeply.

Part 1 | Covers Ch1-2

Term	English	Plain-Language Explanation
Error Function	Error Function	A function that measures the gap between the model’s predictions and the true values; the smaller it is, the closer the line drawn by the model is to the data points
Overfitting	Overfitting	The model is too complex and learns the random noise in the data as if it were a pattern, performing poorly on new data
Underfitting	Underfitting	The model is too simple and hasn’t even learned the basic patterns of the data
Regularization	Regularization	Adding a penalty term to the model parameters to prevent overfitting caused by overly large parameters — like putting a “tightening crown” (restraint) on the model
Weight Decay	Weight Decay	Another name for regularization, because the penalty term keeps “shrinking” the weights during training
Cross-Validation	Cross-Validation	A method of splitting the data into multiple parts and taking turns training and validating, to fairly evaluate model performance
Hyperparameter	Hyperparameter	Parameters that need to be set manually before training (e.g., learning rate, number of network layers), not learned by the model itself
Activation Function	Activation Function	A function that applies a nonlinear transformation to the weighted sum of a neuron’s inputs; common ones are ReLU, sigmoid, and tanh
Pre-activation	Pre-activation	The weighted sum of a neuron’s inputs — each input multiplied by its weight and added up, but before passing through the activation function
Multi-Layer Perceptron (MLP)	Multi-Layer Perceptron (MLP)	A neural network composed of multiple layers of neurons; the more layers, the more complex patterns it can learn
Probability Density Function (PDF)	Probability Density Function (PDF)	A function describing the probability magnitude near a point for a continuous variable; the area under the curve represents probability
Prior	Prior	The belief about the probability of an event before observing data, e.g., “according to the weather forecast, there’s a 30% chance of rain”
Posterior	Posterior	The updated probability of an event after observing data, e.g., “after seeing the ground is wet, the chance of rain becomes 73%”
Likelihood	Likelihood	The probability of observing the data given the parameters, used to judge which set of parameters best explains the existing data
Maximum Likelihood Estimation (MLE)	Maximum Likelihood Estimation (MLE)	Finding the parameter values most likely to have produced the observed data — “which set of parameters makes the data most probable”
Entropy	Entropy	A measure of the uncertainty or “disorder” of a random variable; the more uncertain, the greater the entropy
Kullback-Leibler Divergence	Kullback-Leibler Divergence	A measure of the difference between two probability distributions; the larger the value, the more different the two distributions are
Mutual Information	Mutual Information	A measure of the amount of shared information between two variables; the larger the value, the more correlated the two variables are
Frequentist	Frequentist	Holds that probability is “the frequency with which something happens over many repeated experiments,” and that parameters are fixed but unknown
Bayesian	Bayesian	Holds that probability is “our degree of belief that something will happen,” and updates beliefs via Bayes’ theorem
Aleatoric Uncertainty	Aleatoric Uncertainty	Noise inherent in the data itself (e.g., measurement error), which cannot be eliminated by adding more data
Epistemic Uncertainty	Epistemic Uncertainty	Uncertainty caused by insufficient data; it decreases as more data is collected
Cumulative Distribution Function (CDF)	Cumulative Distribution Function (CDF)	$P(z) = \int_{-\infty}^z p(x)dx$ , representing “the probability that $x$ is less than or equal to $z$ ”
Empirical Distribution	Empirical Distribution	The practice of placing a “spike” of height $1/N$ at each observed data point to approximate the true distribution
Precision	Precision	The reciprocal of variance, $1/\sigma^2$ ; the greater the precision, the more concentrated the distribution
Jacobian Matrix	Jacobian Matrix	A matrix composed of all first-order partial derivatives of a vector function; the absolute value of its determinant represents the volume scaling factor before and after the transformation
Jacobian Factor	Jacobian Factor	The scaling factor $\\|dg/dy\\|$ when transforming probability density variables, ensuring the total probability is conserved after transformation
Jensen’s Inequality	Jensen’s Inequality	For a convex function $f$ : $f(E[x]) \leq E[f(x)]$ , used to prove that KL divergence is non-negative

Part 2 | Covers Ch3-4

Term	English	Plain-Language Explanation
Bernoulli Distribution	Bernoulli Distribution	A probability distribution describing two-outcome events like a coin toss (heads/tails)
Multinomial Distribution	Multinomial Distribution	A probability distribution with multiple outcomes like rolling a die; it’s a generalization of the Bernoulli distribution
Gaussian / Normal Distribution	Gaussian / Normal Distribution	The most common “bell curve” distribution; many phenomena in nature approximately follow it
Covariance	Covariance	Measures the degree to which two variables change together — positive covariance means they increase and decrease together, negative means one increases as the other decreases
Moments	Moments	Statistics describing the shape of a distribution: the first moment is the mean (center of mass), the second moment relates to variance (degree of dispersion)
Bias-Variance Tradeoff	Bias-Variance Tradeoff	A simple model has high bias (misses the target) and low variance (very stable); a complex model has low bias (hits accurately) and high variance (wobbles left and right)
Linear Regression	Linear Regression	The simplest regression model that fits data with a straight line (or hyperplane)
Gaussian Mixture Model (GMM)	Gaussian Mixture Model (GMM)	A weighted mixture of K Gaussian distributions, which can approximate any continuous density function
Exponential Family	Exponential Family	A mathematical framework that expresses various common distributions (Bernoulli, Gaussian, Poisson, etc.) using a unified “template”
Sufficient Statistic	Sufficient Statistic	A function that summarizes all the information in the data; knowing it is enough, and the raw data can be discarded
Natural Parameter	Natural Parameter	The parameter vector in the canonical form of exponential family distributions; it’s a transformation of the original parameters
Kernel Density Estimation (KDE)	Kernel Density Estimation (KDE)	A non-parametric method that places a “hill” at each data point and superimposes them to estimate the density
Basis Function	Basis Function	Applies a fixed nonlinear transformation to input features (e.g., $x \to x^2$ ), extending a linear model into a nonlinear one
Normal Equation	Normal Equation	The closed-form solution for linear regression: $\mathbf{w} = (\mathbf{\Phi}^T\mathbf{\Phi})^{-1}\mathbf{\Phi}^T\mathbf{t}$
Design Matrix	Design Matrix	A matrix $\mathbf{\Phi}$ whose rows are the basis function values of samples; it’s the core of linear regression
Mahalanobis Distance	Mahalanobis Distance	A “distance” measure that accounts for correlations between features; it degrades to Euclidean distance when the covariance matrix is the identity matrix
Decision Theory	Decision Theory	A framework that uses a loss function to quantify the cost of predictions and makes optimal decisions from a probabilistic model

Part 3 | Covers Ch5-6

Term	English	Plain-Language Explanation
Discriminant Function	Discriminant Function	A function that directly outputs classification decisions — input data, output “which category it belongs to”
Decision Boundary	Decision Boundary	The dividing line a classifier uses to separate different categories, like provincial borders on a map
1-of-K / One-Hot Encoding	1-of-K / One-Hot Encoding	Using a vector of length K to represent K categories, with only the corresponding position set to 1 and the rest 0, e.g., category 2 → [0,1,0,0,0]
Logistic Regression	Logistic Regression	A model that uses the sigmoid function for binary classification; although its name contains “regression,” it’s actually a classification method
Sigmoid Function	Sigmoid Function	An S-shaped curve function that compresses any real number into the range 0~1; the output can be interpreted as a probability
Softmax Function	Softmax Function	A function that converts multiple real numbers into a probability distribution, ensuring all outputs sum to 1
Cross-Entropy	Cross-Entropy	A loss function measuring the difference between classification predictions and true labels; the more accurate the prediction, the smaller the cross-entropy
Generative Model	Generative Model	First learns what the data “looks like” for each category, then classifies using Bayes’ theorem
Discriminative Model	Discriminative Model	Directly learns “how to distinguish different categories,” without caring what each category specifically looks like
Confusion Matrix	Confusion Matrix	A table showing a classifier’s prediction results across categories, revealing which classes are easily confused
ROC Curve	ROC Curve	A curve plotting the true positive rate and false positive rate at different thresholds; the closer to the top-left corner, the better
Curse of Dimensionality	Curse of Dimensionality	In high-dimensional space, data becomes extremely sparse, and low-dimensional intuitions completely fail
Data Manifold	Data Manifold	The low-dimensional structure of where high-dimensional data actually distributes — e.g., although handwritten digit images are 784-dimensional, their actual variation is governed by only a few factors
Residual Connection	Residual Connection	A connection style that lets information bypass certain layers and pass through directly, helping train very deep networks
Transfer Learning	Transfer Learning	Applying knowledge learned from one task to another related task, e.g., using an ImageNet-pretrained model for medical image classification
Tensor	Tensor	A general term for multi-dimensional arrays — a scalar is 0-dimensional, a vector is 1-dimensional, a matrix is 2-dimensional, and higher dimensions are called tensors
Mixture Density Network (MDN)	Mixture Density Network (MDN)	A network that can output multiple possible results (rather than a single prediction), suitable for cases where one input corresponds to multiple reasonable outputs
Naive Bayes Classifier	Naive Bayes Classifier	A generative classifier that assumes all features are conditionally independent given the class
Linear Discriminant Analysis (LDA)	Linear Discriminant Analysis (LDA)	A generative classifier that assumes all categories share the same covariance matrix, producing a linear decision boundary
Quadratic Discriminant Analysis (QDA)	Quadratic Discriminant Analysis (QDA)	A generative classifier that allows each category its own covariance matrix, producing a quadratic decision boundary
Radial Basis Function (RBF) Network	Radial Basis Function (RBF) Network	A network that places local basis functions centered on training samples
Representation Learning	Representation Learning	Automatically learning effective representations of input data, rather than relying on manually designed features
Contrastive Learning	Contrastive Learning	A self-supervised method that learns representations by pulling similar samples closer and pushing dissimilar ones apart
Residual Network (ResNet)	Residual Network (ResNet)	A deep network that uses skip connections to bypass certain layers, making it possible to train very deep networks
Forward Propagation	Forward Propagation	The process of computing outputs by passing inputs through the network layer by layer
Data Augmentation	Data Augmentation	Artificially expanding training data by applying transformations (rotation, cropping, flipping) to existing samples
Area Under ROC Curve (AUC)	Area Under ROC Curve (AUC)	The area under the ROC curve; 0.5 = random guessing, 1.0 = perfect classification
F-score (F1)	F-score (F1)	The harmonic mean of precision and recall, comprehensively measuring classification performance

Part 4 | Covers Ch7-9

Term	English	Plain-Language Explanation
Gradient	Gradient	The direction and rate of fastest change of a function at a point — standing on a mountain, the gradient points in the steepest uphill direction
Learning Rate	Learning Rate	A hyperparameter controlling the step size of each parameter update — too large a step easily overshoots the optimum, too small makes training too slow
Stochastic Gradient Descent (SGD)	Stochastic Gradient Descent (SGD)	Estimates the gradient using only a portion of the data each time; much faster than using all data but noisier
Momentum	Momentum	Gives the optimizer “inertia,” helping it push through local minima and saddle points
Adam Optimizer	Adam Optimizer	An optimization algorithm combining momentum and adaptive learning rates; one of the most commonly used optimizers today
Backpropagation	Backpropagation	An algorithm that computes gradients layer by layer from the output layer to the input layer; it’s the core of training neural networks
Chain Rule	Chain Rule	The rule for differentiating composite functions — the mathematical foundation of backpropagation
Dropout	Dropout	A regularization technique that randomly “turns off” a portion of neurons during training to prevent overfitting
Batch Normalization	Batch Normalization	Standardizes the input of each layer, making training faster and more stable
Weight Initialization	Weight Initialization	The method of assigning initial values to parameters before training begins; good initialization makes training smoother
Vanishing/Exploding Gradients	Vanishing/Exploding Gradients	The problem in deep networks where gradients become extremely small or large during propagation
Early Stopping	Early Stopping	A regularization method that stops training early when the validation error stops decreasing
Hessian Matrix	Hessian Matrix	A square matrix of second-order partial derivatives of the error function with respect to all parameters, describing the local curvature of the error surface
Adaptive Gradient	Adaptive Gradient	An optimizer that adaptively adjusts each parameter’s learning rate by accumulating squared gradients; suitable for sparse features
Root Mean Square Propagation	Root Mean Square Propagation	An optimizer that improves AdaGrad by using an exponential weighted moving average, able to “forget” early gradients
Exponential Moving Average (EMA)	Exponential Moving Average (EMA)	An averaging method where recent data has greater weight and distant data is gradually “forgotten”
Layer Normalization	Layer Normalization	Normalizes all features of a single sample (unlike batch normalization which normalizes across samples); commonly used in Transformers
Automatic Differentiation	Automatic Differentiation	A technique that decomposes any differentiable program into elementary operations to compute gradients precisely
Inductive Bias	Inductive Bias	The set of assumptions a learning algorithm makes about the target function, determining how the model generalizes from limited data
Double Descent	Double Descent	A modern phenomenon where test error first decreases, then rises (the classic bias-variance regime), then decreases again (the over-parameterized regime)
Ensemble Method	Ensemble Method	Combining the predictions of multiple models to obtain better generalization performance than a single model

Part 5 | Covers Ch10-11

Term	English	Plain-Language Explanation
Convolution	Convolution	Sliding a small filter over the input to extract local features — like using a magnifying glass to scan an image block by block
Filter / Kernel	Filter / Kernel	A small matrix used in convolution to extract features; different kernels extract different features (edges, textures, etc.)
Pooling	Pooling	Downsampling the feature map, reducing its size while preserving the main features — like shrinking a photo but still seeing the content clearly
Feature Map	Feature Map	The output of a convolution operation, representing features detected at different positions
Receptive Field	Receptive Field	The size of the input image region corresponding to a single point on the output feature map
Probabilistic Graphical Model	Probabilistic Graphical Model	A model that uses a graph structure (nodes and edges) to represent probabilistic relationships between variables
Bayesian Network	Bayesian Network	A probabilistic graphical model using a directed graph to represent causal relationships — arrows mean “because of A, therefore B”
Markov Random Field	Markov Random Field	A probabilistic graphical model using an undirected graph to represent correlations — edges mean “A and B are correlated”
d-separation	d-separation	A method for determining whether two variables in a Bayesian network are conditionally independent
Translation Equivariance	Translation Equivariance	When the input is shifted, the output shifts by the same distance — a CNN can detect features regardless of where they are in the image
Fully Convolutional Network (FCN)	Fully Convolutional Network (FCN)	A network that replaces fully connected layers with 1×1 convolutions, able to handle inputs of arbitrary size
Transposed Convolution	Transposed Convolution	A learnable upsampling operation that maps a low-resolution feature map back to high resolution
Up-sampling	Up-sampling	An operation that increases the spatial resolution of a feature map, used in segmentation and generation tasks
U-Net	U-Net	A symmetric encoder-decoder architecture that preserves spatial details via skip connections, used for semantic segmentation
Skip Connection	Skip Connection	Directly connecting an encoder layer to the corresponding decoder layer, preserving spatial details lost during downsampling
Saliency Map	Saliency Map	A visualization technique showing which regions of the input image have the greatest influence on the classification decision
Adversarial Attack	Adversarial Attack	Adding tiny perturbations to the input that are imperceptible to the human eye, causing the network to make completely wrong predictions
Style Transfer	Style Transfer	Using CNN features to combine the “content” of one image with the “style” of another
Markov Chain	Markov Chain	A stochastic process where the next step depends only on the current state (memoryless)
Hidden Markov Model (HMM)	Hidden Markov Model (HMM)	A model where observed data is generated by hidden states forming a Markov chain
State-Space Model	State-Space Model	A general framework where observed data depends on latent variables evolving over time

Part 6 | Covers Ch12-13

Term	English	Plain-Language Explanation
Attention Mechanism	Attention Mechanism	Lets the model dynamically assign different importance weights to different inputs — like automatically focusing on key paragraphs when reading an article
Self-Attention	Self-Attention	Attention where Query, Key, and Value all come from the same input sequence, allowing every element in the sequence to “see” all other elements
Query / Key / Value	Query / Key / Value	The three roles in attention: Query is “what I’m looking for,” Key is “what label I have,” Value is “my actual content”
Multi-Head Attention	Multi-Head Attention	Multiple parallel attention heads, each focusing on a different type of dependency (syntax, semantics, etc.)
Positional Encoding	Positional Encoding	Adds positional information to the input sequence, because self-attention itself doesn’t distinguish order
Transformer	Transformer	A deep learning architecture based on the self-attention mechanism; the foundation of modern NLP and multimodal models
Word Embedding	Word Embedding	A technique that maps words to low-dimensional dense vectors; semantically similar words are also close in vector space
Tokenization	Tokenization	The process of splitting text into tokens (words, subwords, or characters)
BERT	BERT	A bidirectional encoder that learns language understanding through “fill-in-the-blank” style pretraining
GPT	GPT	A generative pretrained Transformer that learns text generation through “predict the next word” style pretraining
Graph Neural Network (GNN)	Graph Neural Network (GNN)	A neural network for processing graph-structured data (molecules, social networks, etc.)
Message Passing	Message Passing	The mechanism by which nodes in a GNN aggregate neighbor information — each node “gathers intelligence” from its neighbors and then updates itself
Over-Smoothing	Over-Smoothing	The problem in GNNs where, after too many layers, node representations tend to become identical — “those near vermilion turn red, those near ink turn black,” until everyone ends up the same
Adjacency Matrix	Adjacency Matrix	A matrix representing which nodes in a graph are connected by edges
Permutation Invariance	Permutation Invariance	The property that the output doesn’t change with the permutation order of the input nodes
Beam Search	Beam Search	A decoding strategy that keeps the B best candidate sequences at each step, balancing search quality and computation cost
Temperature Parameter	Temperature Parameter	A parameter controlling the randomness of softmax sampling — lower is more deterministic, higher is more random
Masked Attention	Masked Attention	An attention mechanism that prevents tokens from attending to future positions, used in autoregressive decoders
Cross-Attention	Cross-Attention	Attention where Q comes from the decoder and K and V come from the encoder output, bridging the encoder and decoder
Large Language Model (LLM)	Large Language Model (LLM)	A super-large-scale Transformer model pretrained on massive text, capable of performing various language tasks
Low-Rank Adaptation (LoRA)	Low-Rank Adaptation (LoRA)	A fine-tuning method that freezes pretrained weights and trains only low-rank matrices, reducing trainable parameters by orders of magnitude
Prompt Engineering	Prompt Engineering	Guiding a large language model to complete tasks by designing input prompts, without updating weights
Vision Transformer (ViT)	Vision Transformer (ViT)	An architecture that cuts images into patches treated as tokens and processes them with a Transformer
Mel Spectrogram	Mel Spectrogram	Converts an audio waveform into a time-frequency matrix using a perceptually uniform frequency scale
Graph Attention Network (GAT)	Graph Attention Network (GAT)	A GNN variant that uses attention to dynamically weight the importance of different neighbors
Aggregation Operator	Aggregation Operator	A function used in GNN message passing to combine neighbor information (sum, mean, max)
Permutation Equivariance	Permutation Equivariance	The property that node-level predictions change consistently with the permutation order of nodes; a GNN must satisfy this
Byte Pair Encoding (BPE)	Byte Pair Encoding (BPE)	A tokenization method that iteratively merges the most frequent adjacent character pairs until the target vocabulary size is reached

Part 7 | Covers Ch14-16

Term	English	Plain-Language Explanation
Monte Carlo Methods	Monte Carlo Methods	A method of approximating computations using a large number of random experiments — e.g., estimating pi by randomly throwing points
Markov Chain Monte Carlo (MCMC)	Markov Chain Monte Carlo (MCMC)	A sampling method that builds a “random walk” process; after enough steps, the distribution converges to the target distribution
Latent Variable	Latent Variable	Hidden factors that exist in the data but cannot be observed — e.g., you see a person’s expression (observation) and infer their mood (latent variable)
K-Means Clustering	K-Means Clustering	An unsupervised learning algorithm that divides data into K clusters — “things of a kind come together”
Expectation-Maximization	Expectation-Maximization	An iterative optimization algorithm for models with latent variables — first guess the latent variables (E-step), then optimize parameters (M-step), repeating the cycle
Principal Component Analysis (PCA)	Principal Component Analysis (PCA)	A dimensionality reduction method that finds the direction of maximum variance in the data — like finding the best 2D projection angle for a 3D object
Evidence Lower Bound (ELBO)	Evidence Lower Bound (ELBO)	An optimizable “guaranteed” lower bound on the log-likelihood; the core objective function of VAE training
Variational Inference	Variational Inference	An inference method that approximates a complex posterior distribution with a simple distribution
Rejection Sampling	Rejection Sampling	Sampling from an “envelope” distribution, accepting or rejecting based on whether it falls below the target density
Proposal Distribution	Proposal Distribution	A “stand-in” distribution used in rejection sampling and MCMC to propose candidate samples
Acceptance Probability	Acceptance Probability	The probability of accepting a proposed sample in rejection sampling and MCMC, directly affecting sampling efficiency
Energy-Based Model	Energy-Based Model	Defines probability via an energy function: $p(x) \propto e^{-E(x)}$ ; the lower the energy, the higher the probability
Partition Function	Partition Function	The normalization constant $Z$ in an energy model, requiring integration over the entire space, usually difficult to compute
Whitening	Whitening	A preprocessing step that transforms data to zero mean, unit covariance, and uncorrelated dimensions
Factor Analysis	Factor Analysis	Similar to probabilistic PCA but allows each observed dimension its own noise variance (diagonal covariance)
Independent Component Analysis (ICA)	Independent Component Analysis (ICA)	Assumes latent variables are statistically independent non-Gaussian variables; used for blind source separation
Kalman Filter	Kalman Filter	A state-space model for sequential data where latent variables form a Markov chain, used for tracking and time series
Dequantization	Dequantization	Converting discrete observations into continuous variables by adding uniform noise, preventing density from collapsing onto integer points

Part 8 | Covers Ch17-20

Term	English	Plain-Language Explanation
Generative Adversarial Network (GAN)	Generative Adversarial Network (GAN)	A generative model where a generator and discriminator are trained adversarially — like a forger and an appraiser playing a game, until the forger can produce works indistinguishable from the real thing
Generator	Generator	The network in a GAN responsible for generating fake data, aiming to fool the discriminator
Discriminator	Discriminator	The network in a GAN responsible for judging whether data is real or fake, aiming to see through the generator
Normalizing Flows	Normalizing Flows	A generative model that turns a simple distribution into a complex one through a series of invertible transformations — like kneading a ball of clay into a complex shape
Autoencoder	Autoencoder	A network that compresses input into a low-dimensional representation and then reconstructs it, learning the “essential summary” of the data
Variational Autoencoder (VAE)	Variational Autoencoder (VAE)	A generative model that introduces probabilistic modeling into an autoencoder, able to generate new data rather than just reconstruct
Diffusion Model	Diffusion Model	A model that generates data by gradually denoising — first turn the image completely into noise, then learn how to recover it step by step
Forward Process	Forward Process	The process in a diffusion model of gradually adding noise to the data until it becomes pure noise
Reverse Process	Reverse Process	The process in a diffusion model of gradually denoising to recover the data; this is the process of generating new data
Noise Prediction	Noise Prediction	The core idea of diffusion models — instead of directly predicting the image, predict “what noise was added,” then subtract it
Vector Quantization	Vector Quantization	Mapping a continuous vector to the nearest codeword in a discrete codebook, used in models like VQ-VAE
Score Function	Score Function	A vector pointing in the direction of increasing probability density — telling you which way to go to find higher-probability regions
Mode Collapse	Mode Collapse	GAN training failure — the generator only produces a few kinds of outputs, ignoring the full data distribution
Wasserstein Distance	Wasserstein Distance	Measures the minimum “effort” to “transport” one distribution into another; meaningful even when distributions don’t overlap
Cycle Consistency Loss	Cycle Consistency Loss	A loss in CycleGAN that ensures A→B→A recovers the original image, enabling unpaired image translation
Coupling Flow	Coupling Flow	A controllable normalizing flow design (e.g., Real NVP) where part of the input is transformed based on another part
Masked Autoregressive Flow (MAF)	Masked Autoregressive Flow (MAF)	An autoregressive flow with fast likelihood computation but slow sampling
Inverse Autoregressive Flow (IAF)	Inverse Autoregressive Flow (IAF)	An autoregressive flow with fast sampling but slow likelihood computation; the computational properties of MAF reversed
Reparameterization Trick	Reparameterization Trick	Separates randomness from the parameters ( $\mathbf{z} = \mathbf{\mu} + \mathbf{\sigma} \cdot \mathbf{\epsilon}$ ), allowing gradients to backpropagate through sampling in a VAE
Denoising Autoencoder	Denoising Autoencoder	Learns robust representations by recovering clean input from corrupted input; the conceptual predecessor of diffusion models
Masked Autoencoder (MAE)	Masked Autoencoder (MAE)	An image denoising autoencoder using large-scale masking + Transformer
Guided Diffusion	Guided Diffusion	Adding guidance signals to diffusion generation to steer it in a desired direction (e.g., a specific class or text description)
Classifier-Free Guidance	Classifier-Free Guidance	Training conditional and unconditional generation simultaneously, controlling the mixing ratio at inference; the mainstream method in Stable Diffusion
Latent Diffusion Model	Latent Diffusion Model	Runs the diffusion process in a compressed latent space rather than pixel space, greatly reducing the computation for high-resolution images
Contrastive Divergence	Contrastive Divergence	A training algorithm for energy models that approximates the likelihood gradient using short-chain sampling, avoiding computation of the partition function
Langevin Dynamics	Langevin Dynamics	A sampling method combining score-function gradient ascent with random noise perturbation
Denoising Score Matching	Denoising Score Matching	Trains a score function estimator by learning to denoise, equivalent to the training objective of diffusion models
Diffusion Kernel	Diffusion Kernel	The conditional distribution $q(z_t
Stochastic Differential Equation (SDE)	Stochastic Differential Equation (SDE)	The continuous-time limit of discrete diffusion processes, unifying diffusion models, score matching, and Langevin dynamics
Neural ODE	Neural ODE	Treats a neural network as a continuous dynamical system rather than discrete layers, defining the network via an ODE solver