Locate and segment objects.

Four questions about the same picture

“Is there a pet?” is an image-classification question. “Where is it?” requires spatial output. “Which pixels belong to pets?” asks for a semantic mask. “Which pixels belong to this particular pet?” additionally requires instance identity.

• A class label names a category for the image or region. It does not provide an outline.
• A bounding box gives left, top, right and bottom edges enclosing a region. It can contain background.
• A semantic mask assigns each pixel a class such as background or pet. Two pets can share the same class value.
• An instance mask distinguishes individual objects as well as their class. Arbitrary instance IDs can be renamed without changing the answer.

Start with the small authored scene. Both objects belong to class 1, but the annotation uses instance IDs 7 and 9. Merge their predicted identities and inspect the scores. Every foreground pixel remains correct while the predicted object count becomes one. The real pet trimaps later in the lab describe foreground/background/uncertainty, not a complete multi-object instance annotation; do not infer instance ground truth that the dataset does not supply.

Measure overlap before interpreting a score

Intersection over union, or IoU, asks how much two regions overlap compared with everything either region covers. Count the overlap once. Count each region’s area, then subtract the overlap once to avoid double-counting it.

Suppose two 4 × 4 boxes overlap in a 2 × 2 square. Each area is 16; intersection is 4; union is 16 + 16 − 4 = 28. IoU is 4 / 28 ≈ 14.3%. Matching widths alone did not make these boxes agree spatially.

A and B are the two regions. Vertical bars mean area for continuous boxes or pixel count for masks. ∩ means intersection: the part covered by both. All areas in one calculation use the same coordinate system and units. IoU ranges from 0 for no overlap to 1 for exact overlap of nonempty regions.

The lab uses pixel-edge coordinates. A box from (1,1) to (3,3) covers two columns and two rows, for area 4. Right and bottom are exclusive. Boxes that only touch at an edge have zero intersection area. When you select two corner cells, the drawing tool includes both cells and converts them to these edge coordinates. Selecting one cell twice therefore creates a one-pixel box, not an empty rectangle.

Practice: if two masks each cover six pixels and share four, their union is 6 + 6 − 4 = 8, so IoU is 50%. The arithmetic is the same even when neither mask is rectangular.

Match objects one at a time

At an IoU threshold of 0.50, a predicted object can match a same-class annotation with overlap at least 0.50. Each prediction and annotation can participate in at most one match. The lab finds the largest possible number of eligible pairs. It does not choose matches by equal numerical IDs.

In the merge case, the predicted mask contains eight pixels. Each four-pixel annotated object has intersection 4 and union 8 with it, giving IoU 0.50. Only one of the two objects can match the single prediction. Instance recall is therefore 1/2 even though semantic IoU is 1.

In the split case, two two-pixel fragments cover the left four-pixel object. Each fragment has IoU 0.50. At threshold 0.50, one can match and the other is an unmatched prediction. At threshold 0.75, neither fragment matches. The intact right object still matches.

Precision is matched predictions divided by all predictions. Recall is matched annotations divided by all annotations. Duplicate boxes do not create extra true objects: add the duplicate box and inspect precision. If there are no predictions, precision is undefined; if there are no annotations, recall is undefined. Undefined is not silently replaced by 100%.

This is a bounded teaching matching protocol. It is not confidence-ranked average precision or COCO evaluation. Those protocols require additional choices and should not be claimed from this widget’s counts.

A mask and its box can disagree in different ways

The merged prediction’s rectangular envelope includes the blank column between the two objects. Its box area is 10 rather than the mask’s eight pixels. Against either four-pixel object box, intersection is 4 and union is 10, so box IoU is 0.4 while the corresponding mask IoU is 0.5. The two metrics answer different geometric questions.

Try this on a real photograph: draw a broad box around the pet, fill the box as a draft mask, then reveal the source. Background within the rectangle is acceptable for a box envelope but wrong as a pet pixel label. Paint a few obvious background cells out of the mask and check which score changes.

Keep source annotations and your drafts separate

The local subset contains eight Oxford-IIIT Pet photographs and their trimaps. The original JPEGs and trimaps are retained unchanged; display and model inputs preserve aspect ratio with a longest side of 48 pixels. RGB uses area resampling, while annotation labels use nearest-neighbor resampling so interpolation does not invent new class values. This coarsening can erase small details and change boundaries.

Source trimap values are 1 for pet foreground, 2 for background, and 3 for unclassified. Unclassified cells are excluded from training loss and scored-pixel denominators, and the excluded count stays visible. They are not automatically background. The reference box used by this lab is derived from the classified foreground of the resized mask; it is not the source dataset’s animal-head box or a guarantee of the complete animal extent.

Five images retain their official trainval assignment for training. Three official test images are deliberately displayed and reused as development examples. There is no untouched evaluation set here. Eight convenience examples do not support a claim about breed coverage or general pet segmentation. Keep every derivative of one source image with that image; a crop or resized copy is not independent evidence.

Your painting and box edits are separate temporary drafts for each image. They neither overwrite source annotations nor retrain the model. The source reveal gives feedback for practice. It does not turn the activity into an independent assessment.

Train a model small enough to inspect

The tiny segmenter is a logistic classifier applied separately to each pixel. In color-only mode it learns three RGB weights and a bias: four active parameters. In color-plus-position mode it also learns x/y weights: six active parameters. RGB values are mapped from 0–255 to −1–1. Pixel-center coordinates are normalized within the image. There are no texture, neighborhood, contour or object-shape features.

r, g, b are normalized red, green and blue values. x, y are normalized pixel-center coordinates. w values are learned weights and c is the bias. s is the score; p is the predicted pet probability. e is approximately 2.718. Color-only mode fixes the coordinate contributions to zero. All quantities here are dimensionless.

As a worked example, with normalized RGB [1,0,−1], weights [0.2,0.1,−0.3] and bias −0.1, the color-only score is 0.2 + 0 + 0.3 − 0.1 = 0.4, giving probability about 59.9%. A 50% threshold labels that pixel pet; a 75% threshold labels the same prediction background. Changing the threshold does not change the learned weights or the probability.

Training minimizes mean binary cross-entropy on classified training pixels. The gradient contribution is proportional to predicted probability − target; a pet target is 1 and a background target 0. Gradients are computed from the preceding weights and applied simultaneously. Larger images contribute more classified pixels. Many pixels from one photograph are not many independent photographs.

Start with rate 0.3 and 1,000 updates. Compare color-only with color-plus-position under the same update budget, exporting a report before restarting if you want to retain it. Position may encode where the photographer usually placed the animal rather than a transferable shape feature. Lower training loss does not establish shape understanding.

Inspect what the metric hides

A model that predicts background everywhere can achieve 99% pixel accuracy on an image with only 1% foreground. Its foreground IoU is zero: the annotated foreground makes the union nonempty, but the intersection is empty. A class absent from both masks has undefined IoU, not a made-up perfect result.

The real-image experiment has substantial failures. At the guided color-only settings, the boxer example has pet IoU around 0.6% despite reduced training loss across the dataset. Look at the animal, the wrong regions and the source boundary before summarizing the model with one average.

Candidate boxes come from programmed four-connected grouping of predicted pixels. Adjacent same-class pixels form a region; diagonal contact alone does not join them. This is not a separately learned detector and does not establish true instance identity. One animal can fragment into several regions, and touching animals can merge.

Increasing minimum region area removes small predictions. That may remove noise or a real small object. The interface reports components removed by area, those omitted by the 16-region cap, and discarded pixels. These operations affect candidate regions; the displayed semantic IoU still scores the unfiltered model mask. State the stage being measured rather than mixing filtered boxes with unfiltered pixel scores.

Build a report appropriate to your goal

For guided practice, explain one hand-drawn box, one corrected mask and one merge/split result. For implementation practice, reproduce area arithmetic, explicit ignored-region handling and one-to-one matching; test empty denominators, duplicate predictions and boxes that only touch. Then compare the tiny pixel classifier with a model that can use neighborhoods on a properly licensed image split. The current workbench does not supply that richer trained model.

For a decision brief, specify whether the product needs category presence, localization, pixel outlines or individual counts. Include small-object failures, annotation ambiguity and image-level evaluation limits. A useful next evaluation would use new source images with the relevant sizes, backgrounds and viewpoints, chosen before tuning. Exported practice reports and passing a visible multiple-choice check do not establish independently reviewed segmentation proficiency; learner pilots and human artifact review remain pending.

Continue with the perception project workbench to train a small convolutional classifier, freeze subgroup evaluation and save a failure report. Real-image project capture and separate objective cases are available; the P06 implementation bridge and human review remain pending.

Separate perception transfer cases check unfamiliar metric, subgroup and spatial-evidence questions. First submissions stay distinct from practice retries; human implementation review remains pending.