Estimate depth and pose.

A photograph gives direction before distance

Hold one finger in front of another and view them with one eye. They can overlap even though one is farther away. Move your head sideways: the nearer finger usually moves farther across your view. That change supplies evidence that one photograph did not contain.

The workbench starts with three authored points measured in meters. Point A is (1,0,5) and B is (2,0,10): B is twice as far right and twice as far forward. From the original left camera, both project to image coordinate (120,80). A hides B in the lab’s point-sample visibility rule. Their equal image position does not mean equal 3D position.

Move the camera to x = 1 meter. A projects to horizontal coordinate 100, while B projects to 110. A moved 20 image pixels and B moved 10. This relative image movement is parallax. The paired-camera experiment measures it from two simultaneous positions rather than requiring motion over time.

The top-view diagram omits the vertical coordinate y. The image diagrams include it. Read the coordinate list whenever two points look coincident in a simplified view.

Project a 3D point into an image

A pinhole camera is a useful idealized model: rays pass through one center and intersect an image plane. In the lab, x points right, y down and z forward. Both cameras have these orientations. They can translate sideways but do not rotate. A real camera may require different axis conventions, rotation and lens-distortion correction.

Start with A. Its sideways displacement is 1 meter and its forward distance is 5 meters. Their ratio is 1/5. A focal length of 100 pixels turns that ratio into a 20-pixel displacement. Add the image center, 100 pixels, to obtain horizontal coordinate 120.

X, Y, Z are the point’s world coordinates in meters. Cₓ, Cᵧ, C_z are the camera center in the same frame and units. u, v are image coordinates in pixels. Lowercase cₓ, cᵧ are the image’s principal point, where the forward axis meets the image; this lab uses the center (100,80) of a 200 × 160 image. f is focal length in pixels, fixed at 100 with square pixels. The denominator is forward depth relative to the camera.

Read the horizontal equation as: image center plus focal length times sideways displacement divided by forward distance. It applies to points in front of the camera. A point behind the camera does not get a valid forward projection. A projected point can also lie outside the image.

Practice: a point 0.5 meters right and 5 meters forward is displaced 100 × 0.5 / 5 = 10 pixels from the center. Doubling both distances preserves that image coordinate. One view alone cannot select a unique point along the ray without additional assumptions.

Visibility is a separate question

Projection answers where a point would appear. Occlusion asks whether something nearer blocks it. The lab retains the nearest point sample in each raster cell; an exact depth tie uses the original point order. A dashed ring reveals a hidden point for explanation. A real image would not supply that hidden coordinate.

The diagram’s large markers help readability but are not the shapes used for occlusion. This is a sparse point-sample demonstration, not a renderer of full animal surfaces or walls. Distinguish behind-camera, outside-image and occluded states; they have different causes.

Select left B at the original camera position. The algebra can still use the authored coordinate, but the workbench labels that reconstruction hypothetical because B is hidden in the left image. A successful calculation from a revealed answer is not successful perception.

Recover depth from paired views

The two cameras have matching calibration and orientation. Their centers differ only along x by the known baseline B, measured in meters. Such parallel views are already rectified: corresponding points should share an image row. General camera pairs need calibration and rectification before this simple formula applies.

For A with baseline 0.2 meters, the left coordinate is 120 and the right is 116. Disparity d is left horizontal coordinate minus right horizontal coordinate: 4 pixels. The ray geometry then gives depth 5 meters.

u_L and u_R are corresponding horizontal image positions in pixels; D is forward depth from the camera plane in meters; f is focal length in pixels. Pixel units cancel: 100 px × 0.2 m / 4 px = 5 m. Larger disparity means smaller depth when calibration and baseline are fixed.

After finding depth, reverse the projection ratio to recover sideways displacement. With A, (120 − 100) × 5 / 100 = 1 meter. Add the camera center to express the result in world coordinates. The lab averages the two vertical readings only after checking their difference is at most 0.5 pixels.

Zero disparity does not give a finite depth in this model. Negative disparity is inconsistent with its forward-depth assumptions. A row mismatch is reported separately. Very distant reconstructed points beyond the bounded workspace are also rejected; rejection is not evidence that the actual scene is impossible.

Small correspondence errors can become large depth errors

Keep A correctly paired and add +1 pixel to the right coordinate. Disparity falls from 4 to 3. Estimated depth increases from 5 to about 6.667 meters. Adding a pixel to disparity instead would give 20 / 5 = 4 meters. State which measurement changed before interpreting its sign.

A larger baseline increases disparity for the same depth. In the idealized setup this reduces sensitivity to a fixed pixel error. It does not guarantee better results in every real scene: larger viewpoint differences can change occlusion and make matching harder.

Optional mathematics: local sensitivity

The derivative of depth with respect to disparity is negative focal length times baseline divided by squared disparity.

$$\frac{\partial D}{\partial d}=-\frac{fB}{d^2}$$

At disparity 4 pixels, this is −1.25 meters per pixel. It describes a very small change near that point. Multiplying it by a full one-pixel change gives an approximation, not the exact finite difference. The workbench recomputes the actual ratio whenever you change an input.

Now pair left A with right B. Their image rows agree, and the resulting disparity is positive. The estimate can look plausible even though the points are different. Matching a row is a necessary geometric check here, not a proof of identity. Repeated textures, reflections, changing illumination and occlusion complicate correspondence in real images. The lab supplies point identities for inspection rather than implementing image matching.

Pose connects two coordinate descriptions

A point cloud is a collection of 3D samples. A pose describes position and orientation. Given matched source and target samples, pose estimation seeks a transformation that aligns them. Correspondence says which sample matches which; transformation says how the coordinates relate. Mixing those questions hides a common source of failure.

This lesson fits yaw around the y axis and three translation components. It fixes roll, pitch and scale, so it is not unrestricted 3D registration. In the authored experiment, the translation is (1,0.5,1) meters and you choose the yaw. The estimators do not receive those answers.

At yaw 0°, translation simply adds (1,0.5,1) to each point. At yaw 90°, a point (2,3,4) first becomes (4,3,−2) under our axis convention, then (5,3.5,−1) after translation. Rotation and translation order matters.

Optional mathematics: yaw and translation

$$X'=\cos(\theta)X+\sin(\theta)Z+t_x,\quad Y'=Y+t_y,\quad Z'=-\sin(\theta)X+\cos(\theta)Z+t_z$$

θ is yaw in radians; the controls display degrees. Sine and cosine describe the coordinates of a rotated unit direction. The t values are translation in meters. Primed coordinates are transformed coordinates. At 90°, cosine is zero and sine is one, giving the concrete example above. The fit estimates these parameters from the supplied coordinate pairs.

Compare fitting methods without hiding failures

A residual is the difference between a transformed source point and its supplied target. The lab reports Euclidean distance using all three coordinates. Root mean squared error, or RMSE, squares each distance, averages those squares, then takes the square root. Its units remain meters; larger errors receive more influence before the square root.

Ordinary least squares fits all rows. Change one target to an incorrect location and inspect how it pulls the fitted pose away from the clean answer. The apparent compromise is a consequence of the objective, not evidence that every row is partly correct.

The consensus method fits each pair, counts inliers whose distance is within the chosen tolerance, and selects the largest agreeing group. It breaks support ties by inlier error and can refine the fit on the selected group. With five rows it considers ten pairs. This deterministic small enumeration illustrates consensus reasoning; it is not random RANSAC, learned matching or a guarantee of robust recovery.

With one bad target, the four correct rows can agree on the intended pose. The bad row remains visible in all-row error even when the inlier error is almost zero. Changing the tolerance changes what counts as agreement; it is a modeling choice, not automatically a measured uncertainty interval.

Choose three coherently wrong targets. Those rows receive the same erroneous shift. They can form a larger consensus than the remaining correct rows. A low inlier error now accompanies the wrong pose. The clean authored-target error reveals this, but it is used only to explain the result—not to choose the fit. Real tasks may lack those clean targets.

Finally, choose the vertical line. All source points share the same x/z position, so rotating them around y does not reveal yaw. The engine returns no determined pose. More copies of the same uninformative geometry would not solve that ambiguity.

Apply the evidence to your path

For guided practice, explain one shared image coordinate, one stereo calculation and one pose failure with the diagrams and numbers. For implementation practice, reproduce projection and triangulation, test coordinate frames and invalid disparity, then implement residual scoring and a known-transform check. A fuller project needs calibration, image correspondence and general rotation estimation beyond this constrained lab.

For a decision brief, specify when missing depth should trigger a retry or human review. Compare the cost of a plausible wrong estimate with an explicit rejection. Include calibration drift, unavailable points, repeated texture, outlier assumptions and a predeclared test on unfamiliar geometry.

Your exported report is temporary practice evidence. The visible cases and lesson check do not establish independent implementation proficiency. Independent perception project assessment, human review and learner pilots remain pending.

Continue with the perception project workbench to train a small convolutional classifier, freeze subgroup evaluation and save a failure report. Real-image project capture and separate objective cases are available; the P06 implementation bridge and human review remain pending.

Separate perception transfer cases check unfamiliar metric, subgroup and spatial-evidence questions. First submissions stay distinct from practice retries; human implementation review remains pending.