Learn AIGenerate with a variational autoencoder.
Train a tiny probabilistic encoder and decoder, trace a noisy latent sample and compare reconstruction with KL regularization.
By the end: Explain the encoder distribution, likelihood, reparameterization and ELBO; distinguish their expected quantities from finite sampled estimates.
Start with something concrete.
An ordinary autoencoder assigns each point one code. What changes if each point receives a range of plausible codes? Move a noise value while holding the encoder fixed and trace the result through the decoder.
Try the interactive model ↓You sample a latent from the standard-normal prior and decode its mean. Have you sampled the full model when the decoder has nonzero Gaussian observation noise?
Change one thing. Watch what follows.
- Inspect an input’s encoder μ and σ and move ε. Explain which numbers change without training.
- Train 25-update batches at β = 1 and compare reconstruction NLL and KL separately.
- Compare equally long runs at β = 0 and 4 from the same initialization. Capture the trade-off without comparing differently weighted totals alone.
- Select a development point and inspect its encoder distribution; it never enters training gradients.
- Expand the gradient trace and use ε = 0 to predict the reconstruction derivative through log variance.
- Inspect prior-generated observations and export a report naming failures and a new evaluation design.
Loading this browser experiment…
Replace one code with a distribution
Start with a point on a sheet of graph paper. An ordinary autoencoder assigns that point a code and tries to reconstruct it. A variational autoencoder, or VAE, instead assigns a distribution of possible codes. In this lab the code is a single number. The encoder predicts its center, μ (“mu”), and its standard deviation, σ (“sigma”), which describes spread.
The authored data contain two small clusters in two dimensions. Sixteen points train the model; sixteen interleaved but different points are development examples. These are controlled synthetic coordinates, not measurements from people or images. They let you inspect the mechanism without large downloads. The lab’s one-dimensional latent variable is a chosen modeling constraint, not a claim that every two-dimensional dataset can be represented faithfully in one dimension.
The encoder distribution is written q(z | x): our approximation to plausible latent codes z given observed input x. A Gaussian is a bell-shaped density specified by μ and σ. The solid curve shows that approximation for the selected point. The dashed curve is the chosen prior p(z), a standard Gaussian centered at zero with standard deviation one. The prior describes how generation starts before seeing an input.
Separate the noise from what the model learns
Suppose an encoder produces μ = 0.8 and σ = 0.5. Take a standard-normal noise value ε = −1 (“epsilon”). The sampled code is 0.8 + 0.5 × (−1) = 0.3. With ε = 0, it is exactly 0.8. Moving the lab’s noise slider changes the inspected sample without changing any weight.
The symbol ∼ means “is sampled from”; N(0, 1) denotes a Gaussian with mean zero and variance one. This is reparameterization: randomness lives in ε, while the sample is a differentiable calculation involving the learned μ and σ. We can hold the noise draw fixed while asking how a small parameter change would affect reconstruction. We do not differentiate through a random-number generator or claim that noise disappears.
The encoder uses two affine calculations from the two input coordinates: one for μ, one for a raw log-variance value. Its actual log variance is ℓ = 3 tanh(raw/3), so ℓ remains between −3 and 3. Then σ = exp(ℓ/2). This smooth bound is a teaching stability choice; it also restricts the distributions the encoder can represent. It is not a required feature of every VAE.
The decoder describes a likelihood
The sampled code enters four learned tanh units, which produce two decoder output means. Around these means the decoder specifies a Gaussian with independent coordinate noise and a fixed standard deviation s = 0.35. This conditional distribution is p(x | z), the likelihood of an observation given a latent code.
The reconstruction term is negative log likelihood, or NLL, under that conditional distribution. If an observed point is close to the predicted mean, its density is larger and its NLL smaller. For two coordinates with the same fixed standard deviation:
Here x₁, x₂ are the observed coordinates, m₁(z), m₂(z) are the decoder’s two means, s is its fixed observation standard deviation and π is approximately 3.14159. We use natural logarithms, so the units are nats per point. The two coordinate contributions are summed, not averaged over pixels.
For x = [1, 0] and mean [0.8, 0.1], the squared-distance sum is 0.04 + 0.01 = 0.05. Dividing by 2 × 0.35² gives about 0.2041. The constant log(2π × 0.35²) is about −0.2618, so NLL is about −0.0577. A negative value is possible because a continuous density can exceed one; it is not the probability of landing at one exact point. Omitting the constant would change the reported NLL even though, with fixed s, it would not change the learned gradients.
An input can have many plausible latent codes, so reconstruction loss is averaged over samples from its encoder distribution. During training the lab uses four seeded noise draws per point per update. The monitoring panel uses sixteen fixed draws per point, making repeated comparisons reproducible. Neither is an exact integral or the marginal likelihood p(x).
Why add a KL term?
Excellent reconstruction alone does not teach a decoder what to do with codes drawn from the prior. The Kullback–Leibler divergence, or KL, measures how this input’s encoder distribution differs from the prior. For a one-dimensional Gaussian encoder and a standard Gaussian prior, we can calculate it exactly:
At μ = 0 and σ = 1, KL is zero. At μ = 2 and σ = 1, it is 2 nats. For μ = 0.8 and σ = 0.5, it is about 0.6381 nats. Changing the center or spread can make an input easier to reconstruct but increases this mismatch. KL is nonnegative; it is not a percentage or a symmetric distance between samples.
The training objective averages reconstruction NLL plus β (“beta”) times KL over the training inputs. At β = 1, its expectation is the negative evidence lower bound, or negative ELBO. “Evidence” here means the marginal likelihood of the observed input under the generative model.
Read this as: average conditional log likelihood under the encoder, minus its mismatch with the prior, gives a lower bound on log evidence. The expectation symbol E means an average over possible encoder samples. A finite Monte Carlo estimate of that expectation can fluctuate above or below its expected value; a single reported estimate is not itself guaranteed to obey the bound. Exact KL does not remove reconstruction sampling noise.
At β = 0 the model ignores the KL penalty; at β = 4 it weights that penalty more strongly. These are useful controlled comparisons, but their totals optimize different trade-offs. Do not declare one model better solely because its differently weighted total is smaller. Compare the components, reconstruction behavior and generated distribution under the same evaluation protocol. A strong KL penalty can encourage an encoder to carry little information about the input; inspect this possibility rather than assuming regularization is always beneficial.
Optional: follow a reconstruction derivative
Hold ε fixed. Increasing μ by a small amount increases z by the same amount, so ∂z/∂μ = 1. With ℓ = log variance, σ = exp(ℓ/2), giving ∂z/∂ℓ = σ ε / 2. The decoder’s reconstruction derivative with respect to z is multiplied by these factors to reach μ and ℓ. At ε = 0 this draw’s reconstruction derivative through ℓ is zero, but KL can still contribute a log-variance derivative.
The lab displays this one-draw path. The actual update additionally averages all training points and noise draws, adds the KL derivatives, differentiates the bounded log-variance transformation and clips the whole gradient vector to norm 5 when necessary. All 24 parameters update from gradients of the same preceding model. Finite-difference tests hold the noise table fixed when checking every derivative; changing noise between perturbed evaluations would confound the comparison.
Generate from the complete model
For a new observation, sample z from the prior, decode its two means, then draw independent observation noise around those means. No training point is encoded during this process. The right-hand plot includes that observation noise. The left-hand plot instead decodes each input’s encoder mean as a deterministic diagnostic. Decoding a mean is generally not the same as averaging the decoder over random codes.
Both plots use identical fixed axes. Coordinates outside the plotted window are counted and retained in the numerical listing, rather than silently clamped onto its border. Inspect overlap, missing cluster coverage and points between clusters. A smooth model density, a small KL or an attractive picture does not independently establish useful generation.
Make a claim you can support
Use the same initialization and number of updates to compare β = 0, 1 and 4. Export the parameters, objective components and notes. Guided learners should explain a sample calculation and one trade-off. Engineers can reproduce the fixed-noise gradient check and verify that evaluation never changes parameters. Decision learners can identify which observed failure would matter for a proposed task and specify new evaluation data before examining them.
These small public development examples cannot qualify a real-image generator. The lesson’s practice checks are separate from independent project assessment, human artifact review and learner pilots, which remain pending.
At β = 1, a finite set of sampled reconstruction losses plus analytic KL gives one reported objective. Which interpretation is correct?
Take this idea with you.
A VAE learns an approximate posterior and a conditional likelihood while regularizing toward a chosen prior. Trace the sampled objective, its assumptions and its estimation limits before interpreting generated results.