Trade off reward and exploration with entropy.

A dial, not a menu

Earlier policies chose between a few actions. A continuous policy can choose a dial position, a steering amount or a motor command. A probability for each item in a short menu is no longer enough: we need a distribution over a range.

Entering midway? A policy chooses actions. A critic estimates a future return for an action. A replay buffer stores past action/reward receipts. A gradient describes how a small parameter change changes an objective. You can start with the experiment and reveal the equations later.

This lesson uses one continuing context. Every action lies between −1 and +1, earns reward −(action−0.6)², and returns to the same context. The best reward is zero, at action 0.6. An action of 0.1 earns −0.25 because it is 0.5 from the target. There is no episode or terminal reward. The target is visible to you as a diagnostic; the actor learns through fitted critics and never receives the derivative of this reward formula.

This is a small SAC-style mechanism: a stochastic continuous actor, replay, two critics, delayed targets and an entropy term. It is not a deep robotics agent. The larger algorithms and engineering challenges belong in implementation studio work.

Sample first, then squash

The actor learns two numbers: a latent mean μ and a log spread ℓ. Exponentiating ℓ gives a positive standard deviation σ. Draw a standard Gaussian noise value ε, which is usually near zero but sometimes farther away, and construct:

$$\sigma=e^\ell,\qquad u=\mu+\sigma\epsilon,\qquad a=\tanh(u)$$

Think of μ as the center of a cloud and σ as its width before squashing. The tanh function bends that unbounded cloud into the action range (−1,1). It is nearly linear near zero and flattens near either edge. Its derivative is 1−a², so saturated actions respond less to latent changes.

At zero noise, the action is tanh(μ). That generally differs from the average sampled action because tanh is nonlinear. The lab displays both. A manual noise slider lets you trace u and a without training or consuming the collection random sequence.

Entropy measures spread, in the chosen units

For a continuous action, π(a) is a density, not a point probability. Probabilities are areas under the density over intervals. A uniform density of 2 on an interval of width 0.5 has total area one. Density values above one are valid.

Differential entropy is the expected negative log density:

$$H(\pi)=\mathbb E_{a\sim\pi}[-\log\pi(a)]$$

Natural logarithms give units called nats. Unlike discrete entropy, differential entropy can be negative: the uniform width-0.5 example has entropy log(0.5), about −0.693. Rescaling actions also changes it. Compare policies on the same action scale; do not treat the sign as a validity check.

The lab uses 256 fixed, seeded noise draws to estimate entropy and mean reward. Its histogram shows sample counts, not exact density. Reusing these draws makes changes easier to inspect but does not create fresh independent evidence.

Temperature sets a trade-off

The actor minimizes the sample average of:

$$L_{\mathrm{actor}}=\alpha\log\pi(a)-\min(Q_1(a),Q_2(a))$$

The second term encourages high critic values. The first rewards entropy because expected log density is negative entropy. The nonnegative temperature α chooses the relative weight. Zero removes this explicit entropy bonus, but the actor still samples actions while it learns. A larger temperature can favor more spread even when some sampled actions earn lower rewards.

Temperature here is a fixed training-objective weight. It is not the decoder sampling-temperature control from the language lessons, and it is not automatically learned. Different α values define different objectives, so comparing their objective values as if they were the same task score is misleading. Report environment reward and entropy separately.

Fit two critics to a soft target

Each critic is a learned quadratic curve:

$$Q(a)=q_0+q_1a+q_2a^2$$

Two separately initialized critics supply a minimum. This can reduce some optimistic estimates; it does not prove the smaller estimate is correct. Delayed copies move slowly to reduce abrupt target changes.

For each replay receipt, sample a next action from the preceding actor and form a stopped target:

$$y=r+0.8\left[\min(\bar Q_1(a'),\bar Q_2(a'))-\alpha\log\pi(a')\right]$$

A bar marks a delayed critic. The factor 0.8 discounts future soft return. The environment has only one context, so the next action comes from that same policy. “Stopped” means the target is treated as a constant when differentiating the current critic loss; its components are not updated through that loss.

Each critic minimizes average half-squared error, (Q(a)−y)²/2, on eight replay receipts. Its three derivatives are error multiplied by 1, a and a² respectively. All coefficients update together from the preceding model. Afterwards each delayed coefficient becomes 95% of its old value plus 5% of the newly fitted coefficient.

A soft Q value includes future entropy bonuses. It is not just the immediate reward or an ordinary reward-only return. The lab keeps those numbers distinct.

Follow the actor gradient through the sample

Writing a sampled action as tanh(μ+σ ε) makes the random noise explicit. During a gradient calculation, hold ε fixed and differentiate through the deterministic transformations. This is called reparameterization. It lets the actor follow the fitted critic’s slope with respect to its action.

Optional math: transformed density and both actor derivatives

Squashing changes how much probability mass fits into an action interval. Its log density needs a change-of-variables correction:

$$\log\pi(a)=-\frac{\epsilon^2}{2}-\ell-\frac12\log(2\pi)-\log(1-a^2)$$

The last term corrects for tanh compressing the latent axis. The implementation evaluates the log Jacobian with stable softplus arithmetic near saturation, rather than subtracting two nearly equal floating-point values.

Let Q* be whichever current critic is smaller at the sampled action; a tie selects the first critic. Its action derivative is q₁+2q₂a. For a fixed noise draw, the actor-loss derivatives are:

$$\frac{\partial L}{\partial\mu}=2\alpha a-Q^{*\prime}(a)(1-a^2)$$ $$\frac{\partial L}{\partial\ell}=\alpha(-1+2a\sigma\epsilon)-Q^{*\prime}(a)(1-a^2)\sigma\epsilon$$

The entropy part includes both the Gaussian spread and the squashing correction. The critic part follows the chain from parameters to latent sample to action to value. Average eight contributions, then subtract learning rate times each derivative. The critic coefficients and noise stay fixed during this differentiation. No derivative of the environment reward appears.

Keep the protocol visible

Each update collects one new action/reward receipt, retains the latest 64, samples eight replay receipts with replacement, and draws separate noises for targets and actor fitting. Both critics and the actor use the same preceding checkpoint. Delayed critics move after these simultaneous proposals succeed. This explicit teaching ordering differs from implementations that update critics before the actor, use Adam, learn a temperature or share deep representations.

Runs cap at 500 updates. The actor’s latent mean stays within ±3 and log spread within −4 to 0.5; critic coefficients stay within ±100. An invalid proposal rejects the requested work and preserves the previous checkpoint. These are numerical bounds, not a learned safety policy. They are separate from tanh’s action bounds.

Worker requests publish at most 100 updates atomically. Direct CPU controls perform at most ten. Cancellation and hidden/offscreen/navigation cleanup retain the preceding complete checkpoint. Inspection never trains.

Build an entropy comparison

Use seed 7, actor rate 0.01 and critic rate 0.05. Predict the ordering of action spread and reward for temperatures 0, 0.1 and 0.5. Train each for 500 updates and capture it before restarting. Keep the same action scale and update/interaction budget. Explain any mismatch between prediction and observation using the learned critic, sample noise and finite training budget rather than treating entropy as a guarantee.

Students can describe the histogram and reproduce one target with a calculator. Software engineers can independently implement the transformed log density, check gradients with fixed noise and replay the exported last update. Managers can write a decision brief separating reward, exploratory variation, operating constraints and evidence still needed. All learners can try the deeper branch or skip familiar explanations.

Three temporary captures and bounded notes download as JSON. Matching seeds shares random draws, but different policies collect different actions. Repeat other seeds before claiming a robust trend. These public experiments do not replace the later project assessment, authenticate an implementation, or establish deployment readiness. Human artifact review and learner pilots remain pending.