Choose continuous actions.

An action can be a number

Some agents choose from a menu; others output a quantity such as force. A continuous action can vary within a numerical interval. A discrete action selects one of finitely many choices. This lab compares a force in −4 to +4 newtons with the three choices −4, 0 and +4 N.

The comparison keeps the learned model fixed. The discrete version rounds the same model’s output to the nearest allowed choice, with exact halfway ties going to zero. It is not a separately trained discrete-action policy, so the result should not be generalized to all discrete controllers.

Learn from a known controller

An authored feedback rule supplies training labels: clip(4 × position error − 4 × velocity, −4, 4) N. We deliberately know how the teacher works. That makes it possible to inspect the data and test the training implementation without claiming that the teacher is optimal.

The learner is a small neural network: two inputs, four hidden tanh neurons and one tanh output. Each hidden neuron has two weights and a bias; the output has four weights and a bias. That totals 17 parameters. Multiplying its final tanh by 4 bounds the predicted force to ±4 N.

Training compares the normalized output with the teacher force divided by four. The half-squared errors are averaged over the training rows. All parameters change simultaneously using those gradients. Displayed mean absolute error, or MAE, converts predictions back to newtons for interpretation.

Coverage matters more than a reassuring average

The full training set contains 25 error/velocity pairs. A deliberately restricted alternative contains only five rows where velocity is zero. A learner may fit these resting examples well without learning how to brake a moving cart.

Sixteen offset coordinate pairs form a disjoint development grid. They do not supply gradients, but repeated inspection makes them development evidence, not an independent final test. These are authored synthetic states and labels, not a measured robot dataset.

A rollout can also visit observed states outside the training region. The lab counts those samples. With restricted training, nearly every moving state is outside the observed velocity line. A bounded output remains possible there, but boundedness does not establish a correct action.

Separate policy, representation and actuator

The inspector exposes three stages. First the policy computes a raw force. Second the chosen action representation either keeps it continuous or quantizes it to three choices. Third the actuator clips that request to its own force limit.

For example, a continuous request of 3 N and a quantized request of 4 N both become 1 N when the actuator limit is ±1 N. Their applied forces coincide for that instant even though their policies differ. The same limit applies to the authored reference controller.

Saturation restricts force, not position, speed or collision risk. Those quantities require separate operating checks. This lesson stops only at broad numerical bounds and does not claim those bounds make a controller safe.

Evaluate the loop, not only a table of labels

All controllers start the same one-kilogram cart at rest, with a one-metre target. Each applied force changes velocity and position; the next observation therefore depends on earlier actions. Small prediction errors can change the future states being visited and compound over time.

Noise is added independently to measured position and velocity. Each controller receives the same seeded sequence of offsets, but its measurements still differ because its true trajectory differs. This is a matched comparison under one reused noise realization, not a broad robustness study. The true state remains a diagnostic; it does not silently replace the noisy controller input.

Freeze a model before running the comparison. Later training does not change that historical result. Replacing it requires another explicit comparison, which is still repeated development practice. No rollout trains the network or adds teacher labels.

Smoothness is only one outcome

The lab reports total variation of applied force: sum the absolute changes between successive applied forces. This measures how much the command changes at the sample rate. It excludes the initial jump and is not mechanical jerk, energy use or a complete wear model.

Inspect force variation alongside target error, final velocity, saturation, coverage and run status. A controller that does nothing can look perfectly smooth. A run that stops early can accumulate less variation simply because it has fewer actions. Neither should win on that number alone.

The last-half-second criterion checks sampled position and velocity near the target. It does not establish infinite-horizon stability. Fixed plots and clipping counts retain visible failure evidence.

Report the experiment’s limits

For guided practice, compare the same trained model with continuous and quantized outputs, then lower the force limit. For implementation practice, check all parameter gradients and replay the cart’s dynamics from exported applied-force receipts. For a decision brief, state the data coverage, model checkpoint, noise sequence, actuator limit and failed outcomes.

Three temporary captures preserve bounded comparisons. They are practice artifacts, not authenticated training histories, independent implementation review or completed learner pilots. The next operating-limits lesson adds explicit runtime constraints and fallback decisions.