reTraining/docs/requirements.md

Requirements

Status: agreed (planning session, 2026-07-31). Changes only with the user's approval.

Goal

A system for enriching a dataset and improving an existing detection model, iteratively, from an archive of recorded video. One full round:

pick a project → browse the video archive → pick a batch → trim a time range → extract frames → auto-annotate with SAM3 → review and correct every frame → approve → merge into the master dataset → fine-tune from the base model → compare against the base.

The system is generic: the sack case is only the first project. Other cases are created as new projects with their own base model, classes, and video archive — no code changes.

Non-goals (for this version)

• Login, multi-user, tenants, quotas. The architecture leaves room for them; the features are not built.
• Tracking or annotation propagation between frames.
• Collaborative annotation by several people at once.
• Public internet deployment.

---

A. Project

• REQ-001 — The user can create, list, and delete projects. A project has: name, label type, base model, video archive root, and a class list.

• REQ-002 — Each project picks a label type: bbox (YOLO detect) or polygon (YOLO segment). This determines the export format, the editor's behaviour, and which model variant is trained. It cannot be changed once a batch has been merged.

• REQ-003 — The user uploads a base model .pt. The class list is read from the model (model.names). It cannot drift on its own: nothing adds or removes a class as a side effect of another action. Deliberate deletion is REQ-007.

• REQ-004 — A project may be created without a base model. In that case the user types the class list, and the first training starts from pretrained weights (yolo11n.pt / yolo11n-seg.pt).

• REQ-005 — Each class carries its own SAM3 text prompt, which may differ from the class name (e.g. class sack with prompt "woven plastic sack"). Prompts can be edited at any time without affecting existing data.

• REQ-006 — All of a project's data (base model, master dataset, batch frames, trained weights) lives under one project folder, so it can be copied or backed up whole.

• REQ-007 — The user can delete a class at any point in a project's life, including after batches have been merged. Deleting one:

  • removes every annotation of that class, in batches under review and in the master dataset alike;
  • renumbers the classes above it, in the database and in every label file already written to disk, because a YOLO label is an integer index and leaving a gap would make old labels silently name the wrong class;
  • regenerates data.yaml;
  • is refused for a project's last remaining class.

  Before confirming, the user is told how many shapes will be destroyed. The action cannot be undone. If the project's base model was trained on the old class list, it stops being comparable — which REQ-063 already reports rather than hides.

• REQ-008 — The user can add a new class (name & prompt) to an existing project at any time. The new class receives the next sequential class_id, and data.yaml is regenerated if a master dataset exists.

B. Video archive

• REQ-010 — The video archive lives at a local path (disk or mount); videos are not uploaded through the browser.
• REQ-011 — Archive structure: <video_root>/<date>/<batch>.<ext>. The system lists the dates, and within each date the videos with their batch labels parsed from the filename.
• REQ-012 — Each video shows its duration, resolution, and whether it has already been used as a batch in this project.
• REQ-013 — Videos play in the browser with seeking (HTTP Range), without copying the file first.

C. Trim & frame extraction

• REQ-020 — The user sets the in/out range with a timeline slider on the player, and can also type precise timestamps.
• REQ-021 — The user sets the extraction frames per second (default 1 fps). The resulting frame count is shown before extraction runs.
• REQ-022 — Extraction runs as a background job with progress, producing sequentially numbered JPEG files inside the batch folder.
• REQ-023 — One video may be used more than once with different time ranges; each extraction produces its own batch.

D. Auto-annotation

• REQ-030 — Once frames are extracted, the system runs SAM3 over all of them using each class's prompt, as a background job with progress and cancellation.
• REQ-031 — Detections that overlap across prompts are deduplicated (greedy IoU NMS), so one object is not labelled as two classes at once.
• REQ-032 — The confidence threshold is configurable per job.
• REQ-033 — A frame with no detections is valid and still enters the dataset as a negative sample — it is not a failure.
• REQ-034 — Auto-annotation can be re-run on the same batch; previous automatic results are replaced, but the user's manual corrections must never be lost.
• REQ-035 — Auto-annotation can be started in resume mode, which skips frames that already carry automatic annotations. Resume is always an explicit choice and never the default, because a full re-run is also how the confidence threshold (REQ-032) is changed — the system cannot tell the two intentions apart, so it asks. A frame SAM3 legitimately found nothing on (REQ-033) writes no annotations, so a resume re-does it; that is accepted rather than tracked.

E. Review & correction

• REQ-040 — The user reviews frames one at a time, with fast navigation (left/right arrows, thumbnail filmstrip, jump to the next unreviewed frame).
• REQ-041 — Each frame has a status: pending, approved, or rejected. Rejected frames never enter the dataset.
• REQ-042 — The user can draw a new shape, move it, resize it, delete it, and change its class.
• REQ-043 — The user can ask SAM3 for help inside the editor: click or drag a box around one object and the model produces its shape.
• REQ-044 — All annotations and review statuses are persistent — they survive a server restart, unlike today's in-memory sessions.
• REQ-045 — Review progress is visible (e.g. "120/300 reviewed"), and a batch can only be approved once no frame is still pending.
• REQ-046 — The user can delete/clear all annotations of a specific class across all frames in the current batch from the Review editor.

F. Master dataset

• REQ-050 — Approving a batch merges its approved frames and their labels into the project's master dataset (accumulating across batches).
• REQ-051 — The master dataset is train-ready YOLO format: images/{train,val}, labels/{train,val}, and a data.yaml regenerated from the project's class list.
• REQ-052 — Stable val split: once a frame is placed in val, it stays in val across every later merge. New frames are split with an every-Nth pattern.
• REQ-053 — The system records which batches have entered the master dataset, when, and how many images/labels each added.
• REQ-054 — The master dataset can be downloaded as a .zip (e.g. to import into Roboflow or train on another machine).

F2. Data Prep as the merge gate

• REQ-130 — The Batches page supports multi-select. "Prepare & Merge Selected" opens Data Prep scoped to exactly those batches (#/projects/{id}/data-prep?batches=1,2,3). No dataset exists at this point.
• REQ-131 — Data Prep is the merge gate. Filters and augmentation are tuned against the selected batches' shapes; "Confirm merge" names or picks the target dataset and queues one merge job for the whole selection. There is no path from Batches or Review straight to a dataset.
• REQ-132 — The rules in force when a merge is confirmed are snapshotted onto the dataset (datasets.rules_json). The merge runs under the snapshot, and later edits to the project's rules never rewrite an existing dataset. Only an explicit Resync adopts today's rules — and it re-stamps the snapshot with them.

F3. Counting correctness

• REQ-140 — Ghost rejection (entry_travel_min) and spatial dedup (dedup_radius) are separate parameters. They pull in opposite directions, so one number cannot serve both.
• REQ-141 — A track that vanishes parks its history; a new track id born within handoff_radius of its velocity-projected position inherits it. This is what keeps an ID switch at the counting line from either losing a count (the sack's "was above" evidence dies with the old id) or duplicating one (the new id has no "already counted" verdict).
• REQ-142 — A counted direction is the track's last verdict, not a permanent one. A sack genuinely taken back out and reloaded counts again; unloading requires unload_confirm_frames sustained frames above the band, so repositioning by hand cannot cancel a real count.
• REQ-143 — Per-track state is evicted once a track has been gone for track_ttl, so a long shift does not grow state without bound.
• REQ-144 — Every finished track is written to a per-session JSONL with its trajectory and the reason it did or did not count, so a miss can be attributed to the model, the tracker, or the counter.

F4. Counting accuracy bench

• REQ-150 — A page lists every archive video as a row: date, batch, length, and the counter's counted in / counted out / net for it. Videos never counted are still rows — the table is the work list.
• REQ-151 — Each row has an editable ground truth (what a human counted). The scored figure is the signed delta counted_in - ground_truth, so over- and under-counting stay distinguishable. Accuracy is 1 - |delta| / ground_truth.
• REQ-152 — Accuracy totals are computed only over rows where a ground truth is filled in. An uncounted or unscored video never enters the denominator.
• REQ-153 — Counting runs as a queued background job over a selection of videos, or all of them, holding the GPU lock. It renders nothing — no annotated frame, no JPEG encode — which is what makes counting a 30-minute video practical. A run records the parameters and model it used.

F5. Real recording times and working days

• REQ-160 — Each recording's start time is read from the timestamp the camera burns into the top-right of every frame. Folder names and file mtimes are both unreliable: mtimes are file copy times, not recording times.
• REQ-161 — A working day runs 06:00 to 06:00. A recording that started before 06:00 belongs to the previous working day. A recording spanning the boundary is assigned by its start.
• REQ-162 — Recordings are numbered 1..N within a working day, ordered by real start time. The original file path stays the file's identity and is shown beside the number, so results already recorded against it survive.
• REQ-164 — The counting table is grouped into collapsible cycles, newest first, with the recordings inside each one in the order they were made. A cycle's header carries its video count, its AI/ground-truth totals, signed delta and accuracy, and how many of its rows have an unverified start time. Only the newest cycle is expanded by default.
• REQ-165 — The Video Archive page browses the archive by cycle, not by folder. The left-hand list holds cycles newest first; the table shows the recordings of the selected cycle in the order they were made, with their cycle batch number and the time read from the overlay. A recording pulled in from another folder is marked with the folder it sits in.
• REQ-166 — Each recording is checked for a truck with the project's newest model, sampling a handful of frames rather than the whole file. The recording trigger is truck arrival and departure, so one file is one batch — this check is what proves that assumption per file, and flags any recording where it does not hold.
• REQ-167 — The production counter's counting day turns over at the same hour as the archive's cycles, 06:00, so batch_number on the Jetson and the batch order in Video Archive mean the same thing. It stays overridable per deployment via DAILY_CUTOFF_TIME.
• REQ-168 — The recorder writes each archive file at the frame rate the stream actually delivers, and paces writes against the wall clock, so a file's duration equals the real duration of the recording however unevenly the capture loop runs.
• REQ-170 — Recording happens once, on the streaming server, and is stored as a rolling buffer. The truck detector does not encode video: when a session ends it downloads that time range as a copy, so the archive keeps the camera's own codec, resolution and frame rate. Each clip carries a sidecar with the server's start time, which the app trusts over reading the burned-in overlay. Archive folder names remain calendar dates; the app derives cycles from the real start time, so the folder name is never read as a date.
• REQ-163 — Nothing in the archive is moved, renamed or written to; it is mounted read-only. The grouping lives in an index beside it. Timestamps that could not be read, or were read with low confidence, are flagged and can be hand-entered; a hand-entered time outranks any reading and is never overwritten by a rescan.

G. Training & evaluation

• REQ-060 — The user starts training from the project page. Training fine-tunes from the project's base model on the merged master dataset (old + new).
• REQ-061 — A fallback option "train on the latest batch only" (lower LR, fewer epochs) exists for cases where the old dataset is unavailable. It is not the default, and the UI warns about catastrophic forgetting.
• REQ-062 — Default batch, imgsz, and device are derived from the hardware detected at runtime (VRAM), and all of them can be overridden — so moving to a bigger machine needs no code change.
• REQ-063 — After training, the system validates the base model and the new model on the exact same val set, then shows mAP50 and mAP50-95 for both side by side with the delta.
• REQ-064 — Each training run produces a stored model version (weights + metrics). The user can download the weights and promote that version to be the project's new base model for the next round.
• REQ-065 — SAM3 and training must never hold VRAM at the same time; the system releases the SAM3 model before training starts.

H. System

• REQ-070 — Heavy work (extraction, auto-annotation, training) runs as queued jobs, one at a time, because there is a single GPU. Jobs show progress and logs, and can be cancelled.
• REQ-071 — Jobs and their progress are persistent; after a server restart the job list is still there with its final statuses.
• REQ-072 — The application runs via docker compose up with GPU access, and every path (video archive, data folder) is configured through environment/volumes — never hardcoded.
• REQ-073 — The health endpoint reports: detected device/GPU, ffmpeg availability, whether the HuggingFace token was picked up, and database reachability.
• REQ-074 — The system never writes anything into the user's video archive folder.