feat(backend): diff-vs-previous-run reporting for product-scan accuracy harness
Ports the DO-harness's auto-diff-vs-previous-run reporting into accuracy-check-scan.mts: prints a per-field, per-split (Training/ Validation) delta against the last product_accuracy_history.jsonl entry and calls out regressions/improvements explicitly, plus classifier method distribution and average confidence as informational context. Also adds the real held-out validation photo set into sources/product-test-images/ (75 photos, one per current SKU class) with its README documenting the drop-photo -> label -> re-run workflow, so the harness's Validation Set split actually has images to score against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xsxk4ZkDQVVaLUcixDcqb5
No files matched your search
|
After Width: | Height: | Size: 3.4 MiB |
|
After Width: | Height: | Size: 2.3 MiB |
|
After Width: | Height: | Size: 3.1 MiB |
|
After Width: | Height: | Size: 164 KiB |
|
After Width: | Height: | Size: 285 KiB |
|
After Width: | Height: | Size: 2.4 MiB |
|
After Width: | Height: | Size: 2.5 MiB |
|
After Width: | Height: | Size: 215 KiB |
|
After Width: | Height: | Size: 3.1 MiB |
|
After Width: | Height: | Size: 140 KiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 257 KiB |
|
After Width: | Height: | Size: 3.7 MiB |
|
After Width: | Height: | Size: 229 KiB |
|
After Width: | Height: | Size: 254 KiB |
|
After Width: | Height: | Size: 268 KiB |
|
After Width: | Height: | Size: 4.2 MiB |
|
After Width: | Height: | Size: 184 KiB |
|
After Width: | Height: | Size: 220 KiB |
|
After Width: | Height: | Size: 227 KiB |
|
After Width: | Height: | Size: 170 KiB |
|
After Width: | Height: | Size: 217 KiB |
|
After Width: | Height: | Size: 217 KiB |
|
After Width: | Height: | Size: 201 KiB |
|
After Width: | Height: | Size: 3.9 MiB |
|
After Width: | Height: | Size: 239 KiB |
|
After Width: | Height: | Size: 194 KiB |
|
After Width: | Height: | Size: 200 KiB |
|
After Width: | Height: | Size: 207 KiB |
|
After Width: | Height: | Size: 275 KiB |
|
After Width: | Height: | Size: 231 KiB |
|
After Width: | Height: | Size: 304 KiB |
|
After Width: | Height: | Size: 3.3 MiB |
|
After Width: | Height: | Size: 278 KiB |
|
After Width: | Height: | Size: 2.0 MiB |
|
After Width: | Height: | Size: 178 KiB |
|
After Width: | Height: | Size: 293 KiB |
|
After Width: | Height: | Size: 284 KiB |
|
After Width: | Height: | Size: 223 KiB |
|
After Width: | Height: | Size: 3.4 MiB |
|
After Width: | Height: | Size: 3.4 MiB |
|
After Width: | Height: | Size: 1.8 MiB |
|
After Width: | Height: | Size: 225 KiB |
|
After Width: | Height: | Size: 4.5 MiB |
|
After Width: | Height: | Size: 232 KiB |
|
After Width: | Height: | Size: 3.6 MiB |
|
After Width: | Height: | Size: 235 KiB |
|
After Width: | Height: | Size: 2.4 MiB |
|
After Width: | Height: | Size: 149 KiB |
|
After Width: | Height: | Size: 3.6 MiB |
|
After Width: | Height: | Size: 3.8 MiB |
|
After Width: | Height: | Size: 3.7 MiB |
|
After Width: | Height: | Size: 106 KiB |
|
After Width: | Height: | Size: 2.5 MiB |
|
After Width: | Height: | Size: 175 KiB |
|
After Width: | Height: | Size: 2.9 MiB |
|
After Width: | Height: | Size: 208 KiB |
|
After Width: | Height: | Size: 143 KiB |
|
After Width: | Height: | Size: 251 KiB |
|
After Width: | Height: | Size: 176 KiB |
|
After Width: | Height: | Size: 230 KiB |
|
After Width: | Height: | Size: 2.3 MiB |
|
After Width: | Height: | Size: 302 KiB |
|
After Width: | Height: | Size: 114 KiB |
|
After Width: | Height: | Size: 217 KiB |
|
After Width: | Height: | Size: 208 KiB |
|
After Width: | Height: | Size: 129 KiB |
|
After Width: | Height: | Size: 288 KiB |
|
After Width: | Height: | Size: 4.1 MiB |
|
After Width: | Height: | Size: 232 KiB |
|
After Width: | Height: | Size: 286 KiB |
|
After Width: | Height: | Size: 3.8 MiB |
|
After Width: | Height: | Size: 171 KiB |
|
After Width: | Height: | Size: 265 KiB |
@@ -0,0 +1,21 @@
|
||||
# Product-scan validation images
|
||||
|
||||
This folder is the **validation/test set** for the product-scan accuracy
|
||||
harness (`backend/scripts/accuracy-check-scan.mts`) — real-world photos that
|
||||
are *not* part of the classifier's reference dataset, so scoring against them
|
||||
measures actual accuracy instead of memorization.
|
||||
|
||||
**Workflow:**
|
||||
1. Drop a photo here directly (flat, no subfolders — a filename with no `/`
|
||||
is what marks an image as "validation" instead of "training").
|
||||
2. Label it via the `/manual-label-scan` page (correct `no_sku`, `nama_item`,
|
||||
`expiry_date` by hand — don't just accept the AI-scan prefill, that would
|
||||
make the ground truth equal to the model's own prediction).
|
||||
3. Run `node scripts/accuracy-check-scan.mts` from `backend/` — the photo now
|
||||
scores under "Validation Set", separate from "Training Set".
|
||||
|
||||
**This is not where new training photos go.** To improve the classifier
|
||||
itself (DINOv2 index / YOLO fine-tune), add photos to
|
||||
`pfm-web-app/public/produk-pfm/foto-kemasan-v2/<SKU folder>/` instead, then
|
||||
reindex/retrain per `docs/scan-product.md`'s "Model artifacts & retraining"
|
||||
section.
|
||||