What can the agent observe?

A picture of position is not a picture of motion

Two carts are at the center of a track. The left-moving cart needs a push right to stop; the right-moving cart needs a push left. A camera reporting only position gives the same answer for both: zero. This is observation aliasing: different underlying situations look identical to the decision rule.

The lab’s environment state is a pair: position x in cells and velocity v in cells per tick. Velocity can be −1, 0 or +1. An action a is a change in velocity, also −1, 0 or +1 cells per tick. One tick first updates velocity, then position:

$$v_{t+1}=\operatorname{clip}(v_t+a_t,-1,1),\qquad x_{t+1}=x_t+v_{t+1}$$

Here t labels the current tick and t+1 the next. Clip means limit the value to the interval from −1 to +1. This discrete teaching model is not a realistic physics engine. With x=0, v=+1 and a=−1, the new velocity is zero and the new position remains zero. With the same action but v=−1, clipping keeps velocity at −1 and the cart moves to x=−1.

State, observation and the Markov assumption

A state contains the information the model needs to predict the next state and reward after an action. The Markov assumption says that, conditional on the current state and action, earlier history supplies no additional predictive information. It is an assumption about the chosen representation and environment model, not a claim that the world has no history.

Position and velocity suffice for this fixed simulator. Position alone does not: x=0 followed by action 0 can produce x=−1, 0 or +1, depending on the hidden velocity. In a partially observable decision process, the environment still has a state, while the agent receives a limited or noisy observation of it.

The lab keeps these roles separate. The environment updates its true state and checks collision and success. The programmed controller receives an observation object. It cannot read hidden velocity through that function’s arguments. The learner can inspect scenario setup and post-action diagnostics, but those are not controller inputs.

Recover velocity from a short history

Under this model, each movement obeys x at the current tick = previous x + current v. Therefore:

$$v_t=x_t-x_{t-1}$$

Read this as: current velocity equals the change in position across one tick. For previous position −1 and current position 0, velocity is +1. For previous position +1 and current position 0, velocity is −1. Those two histories disambiguate the same current position.

This formula works because observations are exact, adjacent and taken after the corresponding movement; velocity does not change again between movement and sensing. It is not a general theorem that two frames solve partial observability. Unknown time gaps, stale frames, noisy measurements or unobserved disturbances can leave several states plausible. The hidden-state filtering lesson maintains uncertainty instead of claiming exact reconstruction.

Optional: units and a different interval

If two fresh positions are 0.5 seconds apart, dividing their displacement by 0.5 gives average velocity over that interval. It need not equal the current velocity if the cart accelerated during the interval. Our lab uses one fixed tick and a particular update order, so the subtraction gives the simulator’s current discrete velocity exactly.

Missing information requires an explicit assumption

When velocity and the previous position are both absent, the lab lists −1, 0 and +1 as compatible velocities. The controller gives each equal probability. This is a declared uniform prior for this demonstration, not a learned estimate or a posterior derived from the entire trial.

A belief state is a probability distribution over possible hidden states. Maintaining a justified belief generally requires a model of transitions, observations, prior uncertainty and the actions already taken. This controller deliberately does not implement that full filtering process in position-only mode. It starts with the same stated prior each time.

Withholding history lets you test the consequence directly: the history mode loses its advantage over position-only sensing. Full-state sensing still supplies velocity. Never silently replace a missing previous reading with a copy of the current one; that would fabricate evidence for zero velocity.

Compare a rule on matched scenarios

The controller considers all three actions and averages their next-step loss over its compatible velocities. Loss is absolute next position plus twice absolute next velocity, with ten additional points for a collision. Reward is the negative of that loss. These are authored teaching points, not physical costs.

At x=0 with no velocity information, holding gives losses 3, 0 and 3 across the three equally weighted velocities, so its expected loss is 2. Action −1 gives losses 3, 3 and 0, also averaging 2. Action +1 gives 0, 3 and 3. The specified tie rule chooses holding. Only the stationary cart succeeds.

History or full-state sensing identifies each velocity and chooses its cancelling action, stopping all three at the center. The exact three-case comparison changes only the observation mode. It supports a claim about these scenarios and this controller; it does not measure real-world sensor reliability or learned-policy generalization.

Better observation does not fix every decision problem

Try initial position 1 and velocity 0 with full-state sensing. Holding costs one point next step. Moving left reaches position zero but leaves velocity −1, costing two points. The one-step rule therefore holds forever, never succeeding. A learner can instead apply −1 and then +1, arriving and stopping in two steps.

This separates three issues: knowing the state, choosing an objective and planning far enough ahead. A low immediate loss is not task success. A full-state sensor cannot repair a short-sighted planning rule. The trial ends after twelve steps, on collision, or on stopping at zero; budget exhaustion is never called success.

Apply the observation audit

For an experiment report, show the two aliased cases, their different required actions and the extra history that resolves them. For implementation practice, write the observation adapter separately from the state transition, test both missing and present history, and check that hidden state never reaches the controller. For a decision brief, specify sensor freshness and timestamp requirements, costs of missing observations and an evaluation with noisy or delayed measurements.

As a transfer problem, a room thermometer reads 20°C in two rooms: one heating and one cooling. Explain what the single reading leaves unknown, what an earlier timestamped reading adds, and why it still may not reveal heater state, outside temperature or future disturbances. Do not assume a two-reading rule makes this new process Markov.

Compare your explanation

The earlier reading estimates a trend only over its known time interval. Hidden heater state or changing external conditions can still alter the next temperature despite matching current temperature and trend. You need a model and evidence that your chosen state representation predicts the decision-relevant outcomes. A successful cart example does not establish that for a room.

Three temporary captures and a JSON export preserve practice evidence. They do not certify implementation quality, independent mastery or learner effectiveness. Human review and learner pilots remain pending.