Infer what cannot be directly observed.

A measurement does not arrive with its hidden explanation

Imagine measuring a scalar quantity produced by one of two operating modes. You see the number, but the mode is not recorded. One approach is to force a label immediately. Another is to keep uncertainty about which component could have produced the observation.

The lab uses twelve original synthetic training measurements and six separate development measurements. There are no known true component labels. The two-component Gaussian mixture is a model we choose, not a discovery that the world contains exactly two populations. Its hidden variable z can be A or B; its observed value x is the measurement.

Each component has a mean, μA or μB, and both share a standard deviation s. The mixing weight πA is the prior probability of selecting A; πB = 1 − πA. Generation would first choose a component, then draw a measurement from its Gaussian. Inference runs in the other direction: given x, how much support does the model assign to each possible component?

Enumerate the two explanations

For each component, multiply its prior weight by the density it assigns to x. Add the two contributions to get the mixture’s density at x, also called the evidence density here. Normalize each contribution by that sum to obtain the posterior probability of the component.

The vertical bar means “given.” p(x | A) is the Gaussian measurement density under A; p(A | x) is a probability over the two latent states after observing x. These are different quantities. A continuous density is not the probability of landing at a single exact real number.

For a small arithmetic example, suppose πA = 0.6 and πB = 0.4. At the observation, the two component densities are 0.2 and 0.3. The weighted contributions are 0.12 and 0.12, so posterior probabilities are each 0.5. A had the larger prior weight, but B assigned more density to this measurement. Both pieces of evidence matter.

The lab computes in logarithms and uses a stable log-sum-exp calculation before normalizing. Its results are exact for this two-state model up to floating-point rounding. They are not a guarantee that the Gaussian assumptions or the chosen number of components are correct. Swapping the names A and B can leave the entire observable density unchanged.

Fit with fractional responsibilities

We do not initially know the means or mixing weight. Expectation–maximization, or EM, alternates two steps. In the E-step, hold the model fixed and compute each training measurement’s posterior over A/B. These posterior probabilities are called responsibilities. A point with responsibility 0.7 for A contributes a fractional count of 0.7 to A and 0.3 to B.

Click only the E-step. The table appears, but the model parameters have not changed. Every row still represents one observation, and its responsibilities sum to one. The word “expectation” refers to expected latent counts under the model, not observing a hidden label.

In the M-step, keep those responsibilities fixed and refit the mixing weight and means. Let rᵢA be A’s responsibility for training measurement xᵢ, and let N be the number of training measurements. Their sum N_A is A’s effective count.

Apply the same calculation to B. For example, two measurements −1 and 2 with A-responsibilities 0.8 and 0.2 give N_A = 1.0 and weighted mean (−0.8 + 0.4)/1.0 = −0.4. This small worked example illustrates one component update; the actual lab uses all twelve rows.

Both means update using responsibilities from the same preceding model. Recomputing some rows after changing only one mean would be a different procedure. The lab stages the E-step and rejects responsibilities belonging to another model. Changing an initialization or standard deviation clears the staged work. Development measurements never enter the effective counts or mean updates.

Be precise about what EM improves

The shared component standard deviation is fixed in this experiment. The M-step maximizes the expected complete-data log likelihood with respect to the two means and mixing weight under that constraint. Standard-deviation fitting is deliberately omitted, so this is not the full unconstrained Gaussian-mixture algorithm.

Training negative log likelihood, or NLL, averages −log p(x) over the training measurements. Lower is better for the fitted density. Each exact E/M iteration here cannot increase training NLL, apart from numerical rounding. This does not guarantee improved development NLL, a global optimum, useful assignments or the correct number of components.

Try identical starting centers. The two components receive the same relative responsibilities at every measurement and can keep identical means after updating. Symmetry prevents them from spontaneously separating in this deterministic calculation. The run can satisfy the EM update rules while remaining a poor representation of the visible two-cluster pattern. Restarting from separated centers asks a different optimization question; selecting an initialization using development results makes those results development evidence.

The density plot keeps its measurement and density scales fixed. It shows weighted component contributions and their total, plus both data partitions. Gaussian tails extend outside the plotted window. Neither a narrow curve nor a low training NLL establishes that a measurement belongs to a real-world category.

Approximate inference is a separate choice

Now freeze the fitted model mentally and inspect one probe measurement. Exact inference is cheap because there are only two latent states. We deliberately restrict an approximation q to five choices for q(A): 0, 0.25, 0.5, 0.75 or 1. In each case q(B) = 1 − q(A). This tiny variational family makes approximation error visible; it is not a neural encoder or a scalable inference system.

If the exact posterior assigns A probability 0.62, none of those five choices represents it exactly. We can choose the member with the largest evidence lower bound, or ELBO, but its best achievable bound can still be below the exact log evidence. The lab enumerates all five allowed members instead of hiding an optimizer’s behavior.

Here p(x,z) is the joint density: mixing weight times component density. The sum averages its log density under q and adds q’s entropy through the negative log q term. At q(z) = 0, that state’s contribution is defined as zero. The implementation skips the term rather than evaluating log(0).

This finite example gives an exact check on the approximation:

Read it as: exact log evidence minus the bound equals the KL divergence from the chosen approximation to the exact posterior. The gap is zero when q matches that posterior. In this lab both sides are two-state sums, so the bound is evaluated directly. This differs from the preceding VAE lesson’s finite random estimate of a continuous expectation.

A larger family could remove this particular gap: any Bernoulli probability between zero and one can represent the exact two-state posterior. In larger latent models, enumerating all states or using an unrestricted family may be computationally impractical. That is why approximation matters. This lab isolates the restriction without implying that real inference is always a five-choice search.

Keep two kinds of failure separate

A zero approximation gap means q matches the posterior of the selected model. It does not mean the model fits reality. Conversely, improving a mixture’s training likelihood does not guarantee that a restricted q accurately represents its posterior for every input. Keep the fitted parameters fixed when comparing inference approximations, and keep the inference calculation fixed when comparing model fits.

For guided practice, trace one E/M update and one posterior calculation, then explain the symmetry trap. For implementation practice, reproduce the update, check the exact ELBO identity at q = 0 and q = 1, and reject stale responsibilities. For a decision brief, compare separated and merged fits, describe uncertainty near the overlap and state why component assignments are not verified group memberships.

Export up to three comparisons with a note naming the model assumptions, fitting outcome, approximation gap and proposed new evaluation. These exercises use public development data and do not substitute for independent project assessment, human review or learner pilots, which remain pending.