diff --git a/backend/config/scratch_test_ocr.py b/backend/config/scratch_test_ocr.py deleted file mode 100644 index 74b4003..0000000 --- a/backend/config/scratch_test_ocr.py +++ /dev/null @@ -1,20 +0,0 @@ -from PIL import Image -import numpy as np -from paddleocr import PaddleOCR - -ocr = PaddleOCR(use_textline_orientation=True, lang='en') -image = Image.open('/app/config/test_img.jpeg').convert('RGB') -img_arr = np.array(image) -res_list = list(ocr.predict(img_arr)) - -texts = res_list[0].get('rec_texts', []) -dt_polys = res_list[0].get('dt_polys', []) - -for idx, (text, poly) in enumerate(zip(texts, dt_polys)): - if 'BB05032027' in text or 'BB' in text: - print(f"Match: {text}") - print("Raw poly:") - print(poly) - print("Pts computed:") - pts = [(float(p[0]), float(p[1])) for p in poly] - print(pts) diff --git a/backend/plans/next-enhancements.md b/backend/plans/next-enhancements.md index 83e4026..7ea17f3 100644 --- a/backend/plans/next-enhancements.md +++ b/backend/plans/next-enhancements.md @@ -450,6 +450,61 @@ implemented** — no code for this feature exists in the codebase yet. *Suggested order: 12.1 → 12.2 (needs 12.1's tables/movement helper) and 12.3 (needs 12.1's summary route) — 12.2/12.3 are independent of each other.* +## 13. Backend — Per-Sale Expiry Resolution (Candidate Matching) +`src/utils/expiry-matcher.ts`, `src/app/api/parse/route.ts`, +`src/app/api/v1/scan-product/route.ts`, `src/app/api/v1/documents/`, +`config/classify_ocr_server.py` + +Added 2026-07-16 from a user-directed grilling session (ad-hoc feature per +`AGENTS.md` Part B7, like §12). **Read +[../../docs/expiry-tracking-plan.md](../../docs/expiry-tracking-plan.md) first** +— full design, confirmed decisions (full automation at cashier, no cloud ever, +auto-FEFO + `inferred` flag as the only fallback), matching algorithm spec, and +phase plan. Core idea: expiry is captured once per batch at DO intake +(staff-typed on the stock-entry page, §12/root §9), so the cashier scan only has +to **match** OCR fragments against 1–3 known candidate dates — never free-read a +damaged dot-matrix print under time pressure. **Blocked on 12.1 + 12.2** (needs +`stock_batches` + the in-stock candidate filter). Flutter counterpart: root +`plans/next-enhancements.md` §10. + +- **13.1** [TODO] **`resolveExpiryFromEvidence()` matcher util + offline tuning + harness.** Pure TS util implementing the 4-stage resolution + (`single_batch` → `matched_exact` → `matched_fragment` → `inferred_fefo`) + with digit-confusion-aware fuzzy scoring of candidate date print-forms (and + batch codes) against the scan's OCR `text_lines`; thresholds + (`SCORE_MIN`/`MARGIN_MIN`) tuned offline by replaying the 79 frozen-benchmark + line-sets against synthetic candidate sets built from ground-truth labels — + tune for **zero wrong-candidate picks** (flagged FEFO beats a confident wrong + match). Includes verifying the classify server response actually carries + `text_lines` to the gateway (add to payload if not — small + `classify_ocr_server.py` change). Unit tests + harness script committed. +- **13.2** [TODO] **Wire resolution into both scan routes + persist + provenance.** `documents.expiry_source VARCHAR(20)` CHECK + (`single_batch|matched_exact|matched_fragment|inferred_fefo|manual`) + + `expiry_match_score REAL NULL`; `parse/route.ts` Product branch and + `v1/scan-product/route.ts` call the matcher after the §12.2 in-stock filter + and include `resolvedBatch`/`source`/`score` in `metadata.productScan` and + the response (no second GPU call — same pattern as 11.1); PUT persists + `expiry_source` (client override ⇒ `'manual'`); documents list gains an + `?expiry_source=` filter for the end-of-day review of `inferred_fefo` sales. + Blocked on 13.1. +- **13.3** [TODO] **Phase 2 — multi-frame evidence union.** Accept burst + uploads (N frames per scan) on the product-scan path; classify on the best + frame, union OCR `text_lines` across all frames before matching (glare moves + between frames — fragments accumulate). Pairs with a mounted camera at the + cashier (root §10.3). Blocked on 13.2. +- **13.4** [TODO] **Phase 3 — on-prem dot-matrix recognizer.** Synthetic + dot-matrix/inkjet date-crop generator (dot dropout, scratch, fade, curvature, + glare augmentation) + real-data flywheel (harvest scan crops weakly labeled + by the batch registry's staff-typed expiry values); fine-tune a small rec + model on the RTX 2060; deploy as an additional reader in + `classify_ocr_server.py` feeding the same matcher. Goal: shrink the + `inferred_fefo` residue. **No cloud — hard constraint.** Blocked on 13.2; + independent of 13.3. + +*Suggested order: 13.1 → 13.2 → (13.3 and/or 13.4 as needed once Phase-1 +accuracy is measured in the field).* + --- *Sections 1-4 migrated 2026-07-08 from root `plans/next-enhancements.md` sections diff --git a/backend/scripts/experiment-rerank.mjs b/backend/scripts/experiment-rerank.mjs deleted file mode 100644 index 758d086..0000000 --- a/backend/scripts/experiment-rerank.mjs +++ /dev/null @@ -1,178 +0,0 @@ -// Offline experiment: OCR-evidence re-ranking of DINOv2 top-K candidates. -// -// Reads sources/product_scan_fullcap.json (captured live responses, see -// capture-scan-responses.mjs) + sources/product_manual_labels.json (ground -// truth) and simulates candidate re-ranking without touching the GPU stack, -// reporting fixed-vs-broken counts per parameter combination. The winning -// parameters get ported into pfm-web-app/src/utils/product-scan.ts. -// -// Idea: DINOv2's near-twin confusions (same brand, different flavor/size) -// are exactly the cases where the *printed variant words* differ - and -// PaddleOCR usually reads some of them. So within a narrow similarity band -// of the top-1, prefer the candidate whose distinguishing name tokens -// actually appear in the OCR'd text. -// -// Usage: node scripts/experiment-rerank.mjs -import fs from "fs"; -import path from "path"; - -const cap = JSON.parse(fs.readFileSync(path.join("sources", "product_scan_fullcap.json"), "utf8")); -const labels = JSON.parse(fs.readFileSync(path.join("sources", "product_manual_labels.json"), "utf8")); -const gtBySku = new Map(labels.map((l) => [l.filename, l.no_sku])); - -function classSku(className) { - // Class names are foto-kemasan-v2 folder names: " " - return (className || "").trim().split(/\s+/)[0] || ""; -} - -function tokenize(name) { - return name - .toUpperCase() - .split(/[^A-Z0-9]+/) - .filter((t) => t.length >= 2); -} - -function editDistance1(a, b) { - // true if edit distance <= 1 (same length: 1 substitution; off-by-one: 1 indel) - if (a === b) return true; - const la = a.length, lb = b.length; - if (Math.abs(la - lb) > 1) return false; - if (la === lb) { - let diff = 0; - for (let i = 0; i < la; i++) if (a[i] !== b[i]) diff++; - return diff <= 1; - } - const [s, l] = la < lb ? [a, b] : [b, a]; - let i = 0, j = 0, skipped = false; - while (i < s.length && j < l.length) { - if (s[i] === l[j]) { i++; j++; } - else if (!skipped) { skipped = true; j++; } - else return false; - } - return true; -} - -function buildOcrIndex(textLines) { - const joined = textLines.join(" ").toUpperCase(); - const squashed = joined.replace(/[^A-Z0-9]/g, ""); - const tokens = new Set(tokenize(joined)); - return { squashed, tokens }; -} - -function tokenInOcr(token, ocrIdx, fuzzy) { - if (token.length >= 4 && ocrIdx.squashed.includes(token)) return true; - if (ocrIdx.tokens.has(token)) return true; - if (fuzzy && token.length >= 5) { - for (const t of ocrIdx.tokens) { - if (Math.abs(t.length - token.length) <= 1 && editDistance1(token, t)) return true; - } - } - return false; -} - -function ocrEvidenceScore(candTokens, bandTokenCounts, bandSize, ocrIdx, fuzzy) { - // Coverage-normalized, rarity-weighted evidence: fraction of this - // candidate's *distinctive* name tokens (weighted by band rarity) that - // actually appear in the OCR'd text. Normalizing by the candidate's own - // distinctive-token mass is what stops generic packaging words from - // hijacking the ranking - a candidate whose name promises FRENCH + - // INSTITUSI + 2KG but whose package shows only "French Fries" scores - // 1/3, losing to a candidate whose 2 distinctive tokens both appear. - let matched = 0; - let total = 0; - for (const tok of new Set(candTokens)) { - const nWith = bandTokenCounts.get(tok) || 1; - if (nWith >= bandSize) continue; // shared by all -> no signal - const w = 1 / nWith; - total += w; - if (tokenInOcr(tok, ocrIdx, fuzzy)) matched += w; - } - return total > 0 ? matched / total : 0; -} - -function skuFuzzyBoost(extractedSku, candidateSku) { - if (!extractedSku || extractedSku.length < 7) return 0; - if (extractedSku === candidateSku) return 10; // exact (normally pinned upstream anyway) - return editDistance1(extractedSku, candidateSku) ? 1 : 0; -} - -function runConfig({ K, BAND, MARGIN, FUZZY, SKU_BOOST_W }) { - let baselineCorrect = 0, rerankCorrect = 0, fixed = [], broken = []; - for (const item of cap) { - if (item.error) continue; - const gt = gtBySku.get(item.filename); - if (!gt) continue; - const probs = item.classification?.all_probabilities || []; - if (!probs.length) continue; - - const top1Sku = classSku(probs[0].name); - const baselineRight = top1Sku === gt; - if (baselineRight) baselineCorrect++; - - // Candidate band: within BAND of top-1 similarity, capped at K - const top1Sim = probs[0].confidence; - const band = probs.slice(0, K).filter((p) => p.confidence >= top1Sim - BAND); - - const ocrIdx = buildOcrIndex(item.ocr?.text_lines || []); - const candInfos = band.map((p) => { - const sku = classSku(p.name); - const tokens = tokenize(p.name.replace(sku, "")); - return { sku, sim: p.confidence, tokens }; - }); - const bandTokenCounts = new Map(); - for (const c of candInfos) { - for (const tok of new Set(c.tokens)) { - bandTokenCounts.set(tok, (bandTokenCounts.get(tok) || 0) + 1); - } - } - for (const c of candInfos) { - c.ocrScore = ocrEvidenceScore(c.tokens, bandTokenCounts, candInfos.length, ocrIdx, FUZZY) - + SKU_BOOST_W * skuFuzzyBoost(item.ocr?.extracted_sku || "", c.sku); - } - - // Switch away from top-1 only when a band-mate has clearly stronger OCR evidence - let chosen = candInfos[0]; - for (const c of candInfos.slice(1)) { - if (c.ocrScore >= chosen.ocrScore + MARGIN) chosen = c; - } - - const rerankRight = chosen.sku === gt; - if (rerankRight) rerankCorrect++; - if (!baselineRight && rerankRight) fixed.push(item.filename); - if (baselineRight && !rerankRight) broken.push(item.filename); - } - return { baselineCorrect, rerankCorrect, fixed, broken }; -} - -const grid = []; -for (const K of [5, 8, 12]) { - for (const BAND of [0.04, 0.06, 0.08, 0.12]) { - // Coverage scores live in [0, 1]; margin is the minimum coverage lead a - // band-mate needs over the current pick before we switch away from it. - for (const MARGIN of [0.15, 0.25, 0.35, 0.5]) { - for (const FUZZY of [true, false]) { - for (const SKU_BOOST_W of [0, 2]) { - grid.push({ K, BAND, MARGIN, FUZZY, SKU_BOOST_W }); - } - } - } - } -} - -const results = grid.map((cfg) => ({ cfg, ...runConfig(cfg) })); -results.sort((a, b) => (b.rerankCorrect - b.broken.length * 0.01) - (a.rerankCorrect - a.broken.length * 0.01)); - -console.log(`Images evaluated: ${cap.filter((i) => !i.error && gtBySku.has(i.filename)).length}`); -console.log(`Baseline (DINOv2 top-1) correct: ${results[0].baselineCorrect}\n`); -console.log("Top 12 configs by re-ranked correct count:"); -for (const r of results.slice(0, 12)) { - console.log( - ` correct=${r.rerankCorrect} (+${r.fixed.length}/-${r.broken.length}) ` + - `K=${r.cfg.K} BAND=${r.cfg.BAND} MARGIN=${r.cfg.MARGIN} FUZZY=${r.cfg.FUZZY} SKUW=${r.cfg.SKU_BOOST_W}` - ); -} - -const best = results[0]; -console.log(`\nBest config detail: ${JSON.stringify(best.cfg)}`); -console.log(` fixed (${best.fixed.length}): ${best.fixed.join(", ")}`); -console.log(` broken (${best.broken.length}): ${best.broken.join(", ")}`); diff --git a/backend/scripts/seed-validation-labels.mjs b/backend/scripts/seed-validation-labels.mjs deleted file mode 100644 index b713bcf..0000000 --- a/backend/scripts/seed-validation-labels.mjs +++ /dev/null @@ -1,130 +0,0 @@ -// One-off script: fill ground-truth labels for backend/sources/product-test-images/ -// (the product-scan accuracy harness's Validation Set, previously 0 labeled images). -// Run once from backend/: node scripts/seed-validation-labels.mjs -import fs from "fs"; -import path from "path"; - -const IMAGES_DIR = path.join("sources", "product-test-images"); -const LABELS_PATH = path.join("sources", "product_manual_labels.json"); - -// [no_sku, nama_item, expiry_date ("" = not legible in photo, needs re-shoot)] -const DATA = [ - ["11110059", "CEKER BERKUKU FROZEN PACK 1 KG(*)", "13/06/2027"], - ["11140051", "AMPELA FROZEN PACK 1 KG(*)", "26/11/2026"], - ["11620056", "SBL (FILLET PAHA) 1 KG(*)", "27/02/2027"], - ["11650053", "PAHA ATAS 1 KG(*)", "26/02/2027"], - ["11660050", "PAHA BAWAH (1 KG)(*)", "22/06/2027"], - ["11710051", "DADA UTUH (1 KG)(*)", ""], - ["11818300", "CP-BEBEK GORENG 400GR/PAC", ""], - ["1195008A", "RTC CHICKEN KALASAN 400 GR (PAC)", ""], - ["11959937", "SATE AYAM FRESHMART 360 GR (PAC)", "24/11/2026"], - ["12010111", "FIESTA CRISPY BUBBLE 400 GR/PAC", "07/05/2027"], - ["12010115", "FIESTA NUGGET ZOO 400 GR/PAC", "01/12/2026"], - ["12010117", "FIESTA NUGGET HAPPY STAR 400 GR/PAC", "27/08/2026"], - ["12010119", "FIESTA NUGGET CHEESE 123 400 GR/PAC", "09/04/2027"], - ["12010121", "FIESTA NUGGET PIZZABC 400 GR/PAC", "12/11/2026"], - ["12010127", "FIESTA SPICY NUGGET 400 GR/PAC", "09/12/2027"], - ["12010509", "CHAMP CRUNCHY NUGGET 450 GR/PAC", "15/04/2027"], - ["12010515", "CHAMP KOIN KOMBINASI 450 GR/PAC", "15/04/2027"], - ["12010519", "CHAMP NUGGET STICK 900 GR/PAC", "08/03/2027"], - ["12012202", "ASIMO NUGGET KOMBINASI 1 KG/PAC", "13/05/2027"], - ["12012501", "AKUMO CHICKEN NAGET 250 GR", "21/05/2027"], - ["12012503", "AKUMO CHICKEN NUGGET 1000 GR", "17/06/2027"], - ["12012505", "AKUMO KOIN 400 GR/PAC", "16/11/2026"], - ["12020102", "FIESTA SPICY WING 400 GR/PAC", "22/05/2027"], - ["12030102", "FIESTA STIKIE 200 GR/PAC", "26/02/2027"], - ["12030403", "GOLDEN FIESTA STIKIE W/ SWEET CHILLI SAUCE 500GR", "07/04/2027"], - ["12032502", "AKUMO CHICKEN STIK 500 GR", "09/03/2027"], - ["12040101", "FIESTA SCHNITZEL 400 GR/PAC", "09/10/2026"], - ["12040102", "FIESTA CRISPY BUBBLE KATSU 400 GR/PAC", "12/04/2027"], - ["12060103", "FIESTA KARAGE 200 GR/PAC", ""], - ["12060402", "GOLDEN FIESTA KARAGE CHILI SAUCE 500GR", "15/05/2027"], - ["12080101", "FIESTA SPICY CHICK 400 GR/PAC", "26/04/2027"], - ["12130102", "FIESTA CRISPY BURGER 360 GR (NEW)", "27/04/2027"], - ["12150201", "FIESTA DS CRISPY CRUNCH 300 GR/PAC", "03/06/2027"], - ["12150501", "CHAMP CRUNCHY HOTZZ 300 GR/PAC", ""], - ["12190103", "FIESTA DELISTRIPE 400 GR/PAC", "07/04/2027"], - ["12240103", "FIESTA YAKINIKU R/BITES 400 GR/PAC", "07/05/2027"], - ["13010111", "FIESTA SOSIS BRATWURST 300 GR", "25/03/2027"], - ["13010116", "FIESTA SSG ORIGINAL 300 GR", "23/06/2027"], - ["13010118", "FIESTA RTG SSG 65 GR/PAC", ""], - ["13010120", "FIESTA RTG C/CHEESY MELTS 65 GR/PAC", ""], - ["13010122", "FIESTA RTG SAUSAGE WITH HOT LAVA 60G", ""], - ["13010123", "FIESTA RTG SAUSAGE WITH CHEESE LAVA 60G", "25/10/2026"], - ["13010125", "FIESTA RTG SAUSAGE WITH MENTAI LAVA 60GR", "05/11/2026"], - ["13010518", "CHAMP SSG JUMBO BAKAR 500 GR/PAC", ""], - ["13010524", "CHAMP SSG JUMBO BAKAR 500 GR/PAC (NEW)", ""], - ["13012206", "ASIMO SOSIS AYAM KOMBINASI 500 GR", "15/03/2027"], - ["13030501", "CHAMP CHICK MEATBALL 200 GR", ""], - ["13070506", "CHAMP FRANKFURTER SSG 375GR", "13/03/2027"], - ["13100512", "CHAMP CHICK SSG S/SANTAP ORIG 546GR (CAN)", ""], - ["15010101", "FIESTA SHOESTRING 500 GR", "04/06/2027"], - ["15010102", "FIESTA SHOESTRING 1000 GR", "05/03/2027"], - ["15010107", "FIESTA FRENCH F SHOESTRING INSTITUSI 2KG", "17/06/2027"], - ["15020101", "FIESTA STRAIGHT CUT 500 GR", "19/06/2027"], - ["15020102", "FIESTA STRAIGHT CUT 1000 GR", "18/05/2027"], - ["15030101", "FIESTA CRINKLE CUT 500 GR", ""], - ["15030102", "FIESTA CRINKLE CUT 1000 GR", "07/04/2027"], - ["16060113", "FIESTA CHICK SIOMAY 180GR (NEW)", "24/02/2027"], - ["16060114", "FIESTA GYOZA 180 GR (NEW)", "18/05/2027"], - ["17200109", "FIESTA RTS C/TERIYAKI 300GR/PAC", ""], - ["17210106", "FIESTA RTS B/YAKINIKU 300GR/PAC", "19/05/2027"], - ["17210107", "FIESTA RTS B/RENDANG 300GR/PAC", "23/06/2027"], - ["17210108", "FIESTA RTS B/BLACKPEPPER 300GR/PAC", "09/05/2027"], - ["17210109", "FIESTA RTS B/BULGOGI 300GR/PAC", "18/05/2027"], - ["20040101", "FIESTA RAMEN BEKU 570 GR/PAC", "23/06/2026"], - ["20120102", "FIESTA T/B AYAM GORENG 80 GR", ""], - ["20120105", "FIESTA T/B SERBAGUNA Â 80 GR", "09/03/2027"], - ["20120115", "FIESTA RACIK AYAM GORENG 20 GR/PAC", ""], - ["20120116", "FIESTA RACIK NASI GORENG 20 GR/PAC", "13/10/2026"], - ["21000123", "FIESTA RICE W/GEPREK CHICKEN 320GR/PAC", "30/04/2027"], - ["21000126", "NEW FIESTA CHICK RENDANG W RICE 320GR (PAC)", "16/04/2027"], - ["21000130", "NEW FIESTA RICE W/C CHEESE BULDAK 320GR (PAC)", "09/04/2027"], - ["21000137", "FIESTA HAINAMESE CHICKEN RICE 320GR (PAC)", "12/02/2027"], - ["21010101", "FIESTA TRUFFLE GYUDON 320 GR/PAC", "18/03/2027"], - ["21200107", "NEW FIESTA SPAGHETTI CARBONARA 300GR (PAC)", "20/05/2027"], -]; - -const files = fs - .readdirSync(IMAGES_DIR) - .filter((f) => f !== "README.md" && f !== "filelist.txt") - .sort(); - -if (files.length !== DATA.length) { - throw new Error(`File count ${files.length} != DATA count ${DATA.length}`); -} - -const existing = JSON.parse(fs.readFileSync(LABELS_PATH, "utf8")); -const now = new Date().toISOString(); - -let added = 0; -let skipped = 0; -let unreadable = 0; - -for (let i = 0; i < files.length; i++) { - const filename = files[i]; - const [no_sku, nama_item, expiry_date] = DATA[i]; - - if (existing.some((l) => l.filename === filename)) { - skipped++; - continue; - } - - if (!expiry_date) unreadable++; - - existing.push({ - filename, - no_sku, - nama_item, - expiry_date, - top1_confidence: null, - notes: expiry_date - ? "" - : "expiry date not legible in photo (cropped/blurry/out of frame) - needs re-shoot", - saved_at: now, - }); - added++; -} - -fs.writeFileSync(LABELS_PATH, JSON.stringify(existing, null, 2), "utf8"); -console.log(`Added ${added} labels (${unreadable} flagged with no expiry_date), skipped ${skipped} already-labeled.`); diff --git a/docs/expiry-tracking-plan.md b/docs/expiry-tracking-plan.md new file mode 100644 index 0000000..8883e82 --- /dev/null +++ b/docs/expiry-tracking-plan.md @@ -0,0 +1,244 @@ +# Per-Sale Expiry Tracking — Batch Registry + Candidate Matching + +Written 2026-07-16 after a one-question-at-a-time grilling session with the user +(see chat history — decisions recorded below, do not re-litigate). This is the +context doc for backend [`plans/next-enhancements.md`](../backend/plans/next-enhancements.md) +§13 and root [`plans/next-enhancements.md`](../plans/next-enhancements.md) §10. +It **extends** [`stock-feature-plan.md`](stock-feature-plan.md) (backend §12 / +root §9) — read that first; this doc assumes its schema and flows exist. + +**Status: planned, not yet implemented.** Depends on the Stocks feature +(§12.1/§12.2 backend, §9.1–9.4 Flutter), which is itself not yet built. + +## Problem + +The client (retail store) must record the expiry date of every product sold to a +customer. The expiry is printed on the pack, usually dot-matrix/inkjet on frozen +plastic — frequently degraded (printer defects, scratches, ice, glare). + +Measured evidence (79-image frozen validation set, 2026-07-14/15): + +- Overall product-scan accuracy 79.7%; **expiry-date field only 64.6%** (51/79). +- The 28 expiry misses = 21 pure non-detections + 7 garbles. +- The 21 non-detections were probed against **every reader in the stack** + (PP-OCRv6 det/rec, PP-OCRv5-server, VL layout-parsing, direct VLM chat, plus + upscale/blur/CLAHE/threshold preprocessing recipes via the temporary + `/probe-ocr` endpoint): none can read these prints. A human can. This is a + **model capability ceiling, not a pipeline bug** — free-form OCR of these + prints cannot reach the target no matter how the code is tuned. Realistic + free-read ceiling ≈ 82–85%. + +## Confirmed decisions (grilling record, 2026-07-16) + +1. **Success = full automation.** The scan happens at the cashier during + checkout; added wait time is forbidden. Manual entry at the cashier is not + acceptable as a routine step. +2. **Method is open** — not restricted to OCR. Whatever reliably yields the + expiry date wins. +3. **Upstream data**: the Primafood DO paper does **not** carry batch data in a + parseable-enough way to rely on; instead, **staff enter batch code + expiry + date per line item on the DO confirmation page** (the stock-entry step of + the Stocks feature), reading the values **off the physical packs** during + goods receiving — no time pressure there. ~100% of sellable stock arrives + via scanned DOs, so the batch registry will be complete. +4. **Data purpose: per-sale guarantee** — the expiry of the physical unit sold, + per transaction. Softened by decision 5 into "per-sale best evidence, + honestly flagged when inferred". +5. **Residual case** (multiple batches in stock AND print unmatchable): + **auto-record the FEFO batch (earliest expiry) + flag the record + `inferred`** — zero cashier interaction, never block or prompt. +6. **Cashier hardware**: camera does both SKU and date (no barcode reliance); + a fixed **mounted camera** at the checkout is a likely Phase-2 addition. +7. **No cloud at all** — hard on-prem requirement. Flagged records may only be + improved by on-prem means (Phase 3 recognizer, optional human review screen). +8. **Build order**: Phase 1 (batch backbone + candidate matching, existing + hardware) → Phase 2 (mounted camera, multi-frame) → Phase 3 (fine-tuned + dot-matrix recognizer). + +## The reframe + +Stop treating checkout as a *reading* problem ("OCR this damaged print") and +treat it as a *matching* problem: + +> The true expiry of every unit in the store is already known — staff recorded +> it once per batch at intake. At the cashier, the camera only has to decide +> **which of the 1–3 known in-stock batches** this pack belongs to. + +Consequences: + +- **One batch in stock** (the common case in a small store): the lookup alone + is per-unit exact. Zero reading. Milliseconds. +- **Multiple batches**: even a garbled OCR fragment (`...2026`, `2?10`, + `112026`) is enough to pick between candidates whose dates differ. Matching + against 2–3 known strings is drastically easier than free-form reading — + most of the 21 "failed" images produced partial fragments that would + disambiguate fine. +- **Unmatchable**: FEFO + `inferred` flag (decision 5). Checkout never waits. + +## End-to-end data flow + +``` +INTAKE (no time pressure) CHECKOUT (hard latency budget) +───────────────────────── ────────────────────────────── +DO photo → OCR → editor pack photo → SKU classify + → confirm (PUT) → in-stock batch lookup (candidates) + → stock-entry screen → resolveExpiryFromEvidence() + staff types batch_code + 1 candidate → single_batch + expiry_date per line item exact date → matched_exact + (read off the packs) fragment win → matched_fragment + → stock_batches rows born else → inferred_fefo (flag) + (kode_toko, no_sku, → auto-select batch, confirm + batch_code, expiry_date, qty) → decrement batch (§12.2) + → sale row: expiry + source + score +``` + +## The matching algorithm — `resolveExpiryFromEvidence()` + +New pure TypeScript util `backend/pfm-web-app/src/utils/expiry-matcher.ts` +(pure = offline-testable against the 79 captured OCR line-sets, no server +needed). + +**Inputs** +- `candidates`: the scanned SKU's in-stock batches for this store — + `[{batchId, batchCode, expiryDate}]`, from `stock_batches` (§12.1). +- `evidence`: the OCR text lines returned by the classify server for this scan + (`text_lines` — already includes tiled full-res pass + VL-merged lines), plus + the cascade's parsed date (`date_extract.py` output) if any. + +**Stages** (first hit wins) +1. `single_batch` — exactly one candidate: return it. No evidence needed. +2. `matched_exact` — the cascade's parsed date equals one candidate's + `expiry_date`: return that batch. +3. `matched_fragment` — for each candidate, render its expected print forms + (`DDMMYYYY`, `DD/MM/YYYY`, `DD MM YY`, `DD.MM.YYYY`, `BB DDMMYYYY`, + 2-digit-year variants — reuse the format knowledge already encoded in + `date_extract.py`); score every evidence line against every form with + digit-confusion-aware fuzzy matching (Levenshtein over digit subsequences, + with cheap substitutions for known OCR confusions: 0↔8, 1↔7, 5↔6, 2↔7, + 3↔8; also credit partial anchors like a matching year + month pair). + Candidate score = max over (lines × forms). Return the top candidate iff + `topScore ≥ SCORE_MIN` **and** `topScore − runnerUpScore ≥ MARGIN_MIN` + (both thresholds tuned offline — see Testing). +4. `inferred_fefo` — otherwise: return the candidate with the earliest + `expiry_date`, flagged. + +**Output**: `{batchId, expiryDate, source, score, margin}` where +`source ∈ {single_batch, matched_exact, matched_fragment, inferred_fefo}`. + +**Also matched**: the `batch_code` string itself is a second fragment-matching +target — batch codes are often printed adjacent to the date and give an +independent disambiguation signal for free. + +**Verification step before building**: confirm the classify server's response +to the gateway actually carries `text_lines` (the offline capture scripts got +them from the server, so it almost certainly does); if not, add them to the +response payload — small change in `config/classify_ocr_server.py`. + +## Schema & API deltas (on top of stock-feature-plan.md) + +- `documents` (or the Product-branch metadata): add `expiry_source VARCHAR(20)` + with `CHECK (expiry_source IN ('single_batch','matched_exact', + 'matched_fragment','inferred_fefo','manual'))` and + `expiry_match_score REAL NULL`. `'manual'` covers legacy/edited rows. +- `api/parse/route.ts` (Product branch) and `api/v1/scan-product/route.ts`: + after `classifyAndMatchProduct()` + the §12.2 in-stock candidate filter, call + `resolveExpiryFromEvidence()` and include the resolution + (`resolvedBatch` + `source` + `score`) in the persisted + `metadata.productScan` and in the response, so the Flutter editor can + pre-select without any second call (same pattern as task 11.1). +- `v1/documents/[id]/route.ts` PUT: persist `expiry_source` alongside the + existing §12.2 `stock_batch_id` decrement. If the client overrides the + auto-selected batch, source becomes `'manual'`. +- Review surface: `GET /api/v1/documents?expiry_source=inferred_fefo` filter + (admin + own-store), powering an optional end-of-day review list. + +## Flutter deltas (root §10; builds on §9.1–9.4) + +- **Fast path at cashier**: the §9.4 Product-Scan editor auto-selects the + resolved batch. When `source` is `single_batch`/`matched_exact`/ + `matched_fragment`, the flow should be confirmable in **one tap** (or + auto-confirm — decide at pickup with a grill question) with the resolved + expiry displayed prominently. When `inferred_fefo`, same flow plus a small + amber "perkiraan" badge — never a blocking prompt (decision 5). +- **Flag visibility**: history/documents list shows the badge on inferred + sales; an end-of-day review entry point lists them (uses the new filter). + Review is optional and zero-checkout-impact by design. +- **Phase 2 capture mode**: burst capture (N frames over ~1s) in the camera + layer for mounted use; upload frames together; backend unions evidence lines + across frames before matching (glare moves between frames — fragments + accumulate). + +## Phases + +**Phase 1 — batch backbone + matcher (the PoC).** Prereqs: §12.1, §12.2, +§9.1–9.4. New work: `expiry-matcher.ts` + offline tuning harness, schema +columns, route wiring, Flutter fast-path + badge. No new hardware or models. + +**Phase 2 — capture upgrade.** Mounted camera at the cashier (a cheap phone +running the existing Flutter app on a mount is acceptable hardware), burst/ +multi-frame capture, evidence union across frames. Expected to lift fragment +quality substantially — fixed focus distance + controlled lighting beat +hand-held single shots. + +**Phase 3 — on-prem recognizer upgrade.** Fine-tune a small recognition model +specifically on dot-matrix/inkjet date prints: +- **Synthetic data**: render dates in dot-matrix/inkjet fonts over pack-like + backgrounds; augment with dot dropout, scratches, fade, curvature, glare, + ice speckle. Thousands of labeled crops for free. +- **Real data flywheel**: every intake stock-entry (staff-typed batch+expiry) + plus every product-scan photo of that batch = weakly-labeled real training + pairs accumulating automatically in normal operation. Harvest crops from + `uploads/` matched to registry values. +- Train on the RTX 2060 (PaddleOCR rec fine-tune or similar small model); + deploy as an additional reader in `classify_ocr_server.py`; its lines feed + the same matcher. Shrinks the `inferred_fefo` residue. **No cloud, ever** + (decision 7). + +## Expected accuracy (why this reaches ~90%+ where free OCR cannot) + +Let p = share of scans where the SKU has exactly one batch in stock (small +store, fast turnover → p is high, plausibly 0.6–0.8). Those are 100% correct +by lookup. Of the rest, exact + fragment matching succeeds wherever OCR yields +*any* usable fragment — on the 79-set evidence, most misses still produced +fragments; matching 2–3 candidates needs far less signal than free reading. +The residue is auto-FEFO'd — and FEFO itself is right whenever the customer +took from the older batch, so even the flagged slice is mostly correct. +Net: per-sale correctness ~90%+ in Phase 1, rising with Phases 2–3, with +**zero silent garbage** — every record carries its provenance (`source`). + +Two honest caveats to monitor: +- **SKU misclassification poisons the lookup** (wrong SKU → wrong candidates). + Mitigation: §12.2's in-stock filter shrinks the effective class space to + what the store actually stocks; near-twin SKU confusion keeps improving via + reference photos. Track `sku/name` accuracy alongside expiry. +- **Candidates with near-identical dates** (differ by one digit) can fail the + margin test → FEFO+flag. Correct behavior; expected to be rare. + +## Testing plan + +- **Offline matcher tuning (before any wiring)**: replay the 79 captured OCR + line-sets (`sources/product_scan_detail_*.json` + fullcap captures) against + synthetic candidate sets built from the ground-truth labels (1, 2, and 3 + candidates at varying date distances). Tune `SCORE_MIN`/`MARGIN_MIN` for + zero wrong-candidate picks (a wrong confident match is worse than a flagged + FEFO). This reuses the frozen benchmark as a matcher benchmark. +- **Unit tests**: `expiry-matcher` pure-function tests (form rendering, + confusion-aware scoring, margin logic, FEFO tiebreak) — TS side; Flutter + pure-logic tests for fast-path/badge state per `source` value. +- **Live E2E** (per repo convention, against the Docker stack): scan a DO → + stock-entry with 2 batches of one SKU → product-scan a pack of the older + batch → verify `matched_*` resolution, decrement of the right batch, and + `expiry_source` in Postgres; then repeat with an unreadable pack → verify + `inferred_fefo` + flag, no prompt shown. + +## Relationship to existing plans + +- **Extends** `stock-feature-plan.md`: §12.2's "closest-to-OCR-expiry batch + auto-selected" dropdown becomes the *manual-override* UI behind the new + automated resolution; the §12.2 decrement/400-on-bad-batch semantics are + unchanged. +- Backend tasks: `backend/plans/next-enhancements.md` **§13**. +- Flutter tasks: root `plans/next-enhancements.md` **§10**. +- The 79-image frozen benchmark and its capture tooling (accuracy work, + 2026-07-14/15) become the matcher's offline test bed — nothing there is + wasted by this reframe. diff --git a/docs/stock-feature-plan.md b/docs/stock-feature-plan.md index f81353c..1008c01 100644 --- a/docs/stock-feature-plan.md +++ b/docs/stock-feature-plan.md @@ -11,6 +11,14 @@ backlog entries in root [`plans/next-enhancements.md`](../plans/next-enhancement codebase yet — this doc is the design record to build from when the tasks below are picked up via `n`/`next`. +**Extended 2026-07-16 by [`expiry-tracking-plan.md`](expiry-tracking-plan.md)** +(backend §13 / root §10): the Product Scan batch *selection* described in §5/§10 +below becomes an automated candidate-matching resolution at the cashier +(fragment-match OCR evidence against the in-stock batches this plan registers; +auto-FEFO + `inferred` flag as fallback). The dropdown UX below survives as the +manual-override path. Schema, decrement semantics, and everything else in this +doc are unchanged. + ## Context The app currently tracks Delivery Order (DO) documents and a "Product Scan" diff --git a/plans/next-enhancements.md b/plans/next-enhancements.md index 5eba486..29d6dae 100644 --- a/plans/next-enhancements.md +++ b/plans/next-enhancements.md @@ -487,15 +487,53 @@ implemented** — no code for this feature exists in the codebase yet. 9.4 (needs backend §12.2's decrement hook, not just §12.1's CRUD). 9.5-9.7 are design-completeness fixes to fold into 9.1-9.4's implementation, not a separate pass.* +## 10. Cashier Fast Path — Automated Expiry Resolution +`lib/features/editor/` (product editor), `lib/features/documents/`, +`lib/features/camera/` + +Added 2026-07-16 from a user-directed grilling session (ad-hoc feature per +`AGENTS.md` §7). **Read +[docs/expiry-tracking-plan.md](../docs/expiry-tracking-plan.md) first** — full +design and confirmed decisions (checkout must be fully automated: zero typing, +zero blocking prompts; unresolvable scans auto-record the FEFO batch with an +`inferred` flag). Consumes backend §13's resolution +(`resolvedBatch`/`source`/`score` inside `productScan`). **Blocked on 9.1–9.4 +and backend §13.2.** + +- **10.1** [TODO] **One-tap (or zero-tap) confirm at the cashier.** The Product + Scan editor pre-selects backend §13's resolved batch; when `source` is + `single_batch`/`matched_exact`/`matched_fragment`, the flow collapses to a + single confirm tap with the resolved expiry shown prominently — grill at + pickup whether to go full auto-confirm (no tap) for high-confidence + resolutions. The §9.4 batch dropdown remains as the manual-override path + only (override ⇒ backend records `expiry_source = 'manual'`). +- **10.2** [TODO] **`inferred` badge + end-of-day review list.** Sales resolved + as `inferred_fefo` show a small amber "perkiraan" badge in the editor and in + the history/documents list — informational only, never a blocking prompt + (confirmed decision). New review entry point (drawer or History filter) + listing flagged sales via backend §13.2's `?expiry_source=inferred_fefo` + filter, so staff can optionally correct them after hours with the packs in + hand — zero checkout impact by design. +- **10.3** [TODO] **Phase 2 — burst capture mode for a mounted camera.** Camera + layer gains a burst mode (N frames over ~1s) intended for a fixed-mounted + phone at the checkout counter; frames upload together and backend §13.3 + unions OCR evidence across them. Includes a settings toggle (hand-held + single-shot vs mounted burst). Blocked on backend §13.3. + +*Suggested order: 10.1 → 10.2 (10.1's `source` plumbing feeds 10.2's badge); +10.3 only after Phase-1 field measurement says fragment quality is the +bottleneck.* + --- -*Sections 1-9 (Flutter) are the only sections this file tracks. Backend +*Sections 1-10 (Flutter) are the only sections this file tracks. Backend enhancements (formerly sections 5-8 here, removed 2026-07-08) now live exclusively in [backend/plans/next-enhancements.md](../backend/plans/next-enhancements.md); that file's §9 holds the backend counterparts to this file's §6-7, §10 -holds the backend counterparts to this file's §8, and §12 holds the backend -counterparts to this file's §9 (see -[docs/api-contract-map.md](../docs/api-contract-map.md) and -[docs/stock-feature-plan.md](../docs/stock-feature-plan.md) for the shared +holds the backend counterparts to this file's §8, §12 holds the backend +counterparts to this file's §9, and §13 holds the backend counterparts to +this file's §10 (see [docs/api-contract-map.md](../docs/api-contract-map.md), +[docs/stock-feature-plan.md](../docs/stock-feature-plan.md), and +[docs/expiry-tracking-plan.md](../docs/expiry-tracking-plan.md) for the shared design docs).*