# Product-scan validation images — frozen benchmark set

This is the **actual Validation Set the accuracy harness scores**
(`accuracy-check-scan.mts`'s `getImagePath()` points here, not at
`../product-test-images/`). It exists so re-running the harness always grades
the exact same images — the live-intake folder can keep growing from new
`/manual-label-scan` drops without silently shifting the benchmark underfoot.

**Naming**: each file is `<index> <no_sku>.<ext>` (e.g. `1 11110059.jpeg`),
where `<index>` is just this file's stable position in the set — it carries
no other meaning. `product_manual_labels.json`'s ground-truth entries for
these images use this same filename.

**Do not hand-edit this folder.** It's fully generated by
`node scripts/freeze-validation-set.mjs` (run from `backend/`), which copies
every flat (non-training) entry out of `product_manual_labels.json` from
`../product-test-images/`, renames it, and rewrites those entries'
`filename` fields to match. To add a new SKU/photo to the benchmark:
label it in the live-intake folder first (see that folder's README), then
re-run the freeze script.

**79 images as of 2026-07-14.** 5 of them (SKUs 12010801, 12012504, 12130504,
13050101, 15040102) are flagged `TRAINED-ON` in `product_manual_labels.json`'s
`notes` field — their only available source photo (from an external
`research-sam3` segmentation project) was already used to train the
classifier (as a SAM3 crop + augmentations), so they are **not** a clean
held-out test. Their per-image scores will read as memorization, not real
generalization, until a fresh, never-trained-on photo is dropped for those
SKUs. The other 74 are genuinely held out.