reTraining/docs/design.md

Design

Serves ./requirements.md. Each section names the REQs it fulfils.

Overview

Browser (React + Vite, served by nginx)
        │  /api → proxy
        ▼
FastAPI  ──►  jobs (1 worker thread, 1 GPU)
   │            ├─ extract  : ffmpeg              (CPU)
   │            ├─ autolabel: SAM3                (GPU)
   │            ├─ merge    : copy + write labels (CPU)
   │            └─ train    : Ultralytics + eval  (GPU)
   ├──► SQLite  (metadata & status)
   └──► data/   (frames, master dataset, weights)

Storage split rule: SQLite holds metadata and status; the disk holds pixels, final labels, and weights. The master dataset must stay useful even if the database is lost (REQ-006, REQ-054).

Disk layout (REQ-006)

data/                                   # Docker volume
  app.db                                # SQLite (WAL)
  projects/<slug>/
    base/model.pt                       # the project's active base model (REQ-003)
    dataset/                            # MASTER, accumulative (REQ-050…052)
      images/{train,val}/…jpg
      labels/{train,val}/…txt
      data.yaml
    batches/<batch-id>/
      frames/000001.jpg …               # extraction output (REQ-022)
    models/<n>/
      best.pt
      metrics.json                      # base vs new metrics (REQ-063)
      runs/                             # Ultralytics run directory

Master dataset filenames: <batch-id>__<frame number>.jpg — unique across batches and self-documenting about where each image came from. The user's video archive is read-only (REQ-074).

SQLite schema

Created by an idempotent migration in backend/db.py at startup.

projects(
  id, slug UNIQUE, name, label_type CHECK(bbox|polygon),
  base_model_path, base_model_kind CHECK(uploaded|pretrained|trained),
  video_root, val_every DEFAULT 5, created_at)

project_classes(
  id, project_id → projects, class_id INT, name, prompt,
  UNIQUE(project_id, class_id))            -- class_id = the YOLO class index (REQ-003/005)

batches(
  id, project_id → projects, video_path, date_label, batch_label,
  start_sec REAL, end_sec REAL, fps REAL,
  status CHECK(extracting|extracted|labeling|reviewing|approved|merged|failed),
  frame_count INT, created_at, merged_at)

frames(
  id, batch_id → batches, idx INT, filename, width INT, height INT,
  review_status CHECK(pending|approved|rejected) DEFAULT 'pending',
  UNIQUE(batch_id, idx))

annotations(
  id, frame_id → frames, class_id INT,
  geometry TEXT,                           -- JSON; see "Geometry format"
  score REAL, source CHECK(auto|manual), created_at)

dataset_items(                             -- master dataset membership (REQ-052)
  id, project_id → projects, frame_id → frames UNIQUE,
  split CHECK(train|val), image_rel, label_rel, added_at)

model_versions(
  id, project_id → projects, version INT, weights_path,
  parent_model_path, metrics TEXT, base_metrics TEXT, created_at,
  UNIQUE(project_id, version))

jobs(                                      -- persistent (REQ-071)
  id, project_id, batch_id, type CHECK(extract|autolabel|merge|train),
  status CHECK(queued|running|done|failed|cancelled),
  progress INT, total INT, message, error, log TEXT,
  created_at, started_at, finished_at)

The stable val split (REQ-052) is enforced by dataset_items: an existing row never changes its split. On merge, only frames without a row are assigned, using a per-project round-robin counter (val_every) that continues from the previous count.

Geometry format. One JSON column covers both label types (REQ-002):

• bbox → {"type":"bbox","points":[x0,y0,x1,y1]}
• polygon → {"type":"polygon","points":[[x,y], …]}

Coordinates are stored normalized 0–1 against the frame size, so neither the editor nor the exporter needs to know the display size. SAM3 mask → polygon conversion is review.mask_to_polygons(); for bbox projects the mask is only used to take its bounding box.

Backend modules

app/ moves to backend/. Reuse existing code wherever possible:

Module	Role	Status
sam3_engine.py	SAM3 singleton, open_state/apply_prompts/segment_at	reused, plus a release() for REQ-065
labeling.py	per-frame detection + cross-prompt NMS (REQ-031)	reused; the folder-walking half went with the old flow
exporters.py	YOLO label writing	deleted — dataset.py writes labels, mask_to_polygons moved to review.py
sessions.py	exemplar/tap interaction	deleted — see below
jobs.py	single-worker queue	extended: job types + persistence
training.py	Ultralytics fine-tune	changed: starts from the base model, args from hardware.py
db.py	connection + migration	new
projects.py	project CRUD, reads classes from a .pt	new
library.py	scans <video_root>/<date>/<batch>	new
video.py	ffprobe, Range streaming, ffmpeg extraction	new
batches.py	batch lifecycle	new
review.py	annotation CRUD, frame status, click-assist	new
autolabel.py	the SAM3 job over a whole batch	new
dataset.py	merge into the master dataset, stable split	new
evaluate.py	validate base vs new model	new
hardware.py	VRAM detection → training defaults	new
api/	the FastAPI routes, one module per domain	new

Removed: uploads.py, static/index.html, and the old flow's endpoints.

sessions.py was meant to be reused for click-assist, but it existed to hold GPU-resident state for an interactive session — a whole eviction policy, an undo stack, and a per-session annotation store, all of which the database and the stateless review.assist now cover. Adapting 357 lines to do what 60 lines do was not worth it, so the module is gone. The one thing it knew that mattered — SAM3 wants exemplar boxes as normalized centre-x, centre-y, width, height — moved with it.

The 400-line file limit (see ../AGENTS.md) applies to all of the above. It is why the routes live in backend/api/{projects,batches,review,models,jobs}.py rather than in main.py, which now only builds the app and owns startup. Route modules import their domain module under an alias (from backend import projects as project_store) so the two namespaces stay distinguishable.

API contract

GET    /api/health                              REQ-073

GET    /api/projects                            REQ-001
POST   /api/projects                            REQ-001,002,004,005
GET    /api/projects/{id}                       # includes label_type_locked: bool (REQ-002)
DELETE /api/projects/{id}
PATCH  /api/projects/{id}                       # class prompts, val_every (REQ-005)
POST   /api/projects/{id}/classes               # add new class {name, prompt} (REQ-008)
DELETE /api/projects/{id}/classes/{class_id}       # delete class, delete shapes, reindex classes (REQ-007)
POST   /api/projects/{id}/base-model            # upload .pt, read classes (REQ-003)
GET    /api/projects/{id}/dataset               # master dataset summary (REQ-053)
GET    /api/projects/{id}/dataset/download      # zip (REQ-054)

GET    /api/projects/{id}/library               # list of dates (REQ-011)
GET    /api/projects/{id}/library/{date}        # videos + duration/resolution (REQ-012)
GET    /api/projects/{id}/video?rel=…           # Range streaming (REQ-013)

POST   /api/projects/{id}/batches               # {rel, start_sec, end_sec, fps} → extract job
GET    /api/batches/{id}                        # status + review progress (REQ-045)
GET    /api/batches/{id}/frames                 # frames + statuses
POST   /api/batches/{id}/autolabel              # {threshold} → job (REQ-030,032,034)
DELETE /api/batches/{id}/classes/{class_id}/annotations # clear all shapes of class in batch (REQ-046)
POST   /api/batches/{ids}/approve               # one or many, comma-separated → one merge job (REQ-131)
GET    /api/batches/{ids}/triage/summary        # one or many, comma-separated (REQ-130)
GET    /api/batches/{ids}/triage/shapes
GET    /api/batches/{ids}/triage/suggest
POST   /api/batches/{ids}/triage/simulate

GET    /api/frames/{id}/image?w=…               # frame image / thumbnail
GET    /api/frames/{id}/annotations
POST   /api/frames/{id}/annotations             # add a manual shape (REQ-042)
PATCH  /api/annotations/{id}                    # move/resize/reclass
DELETE /api/annotations/{id}
POST   /api/frames/{id}/assist                  # click/box → SAM3 shape (REQ-043)
POST   /api/frames/{id}/status                  # approved | rejected | pending (REQ-041)

POST   /api/projects/{id}/train                 # → train job (REQ-060,061,062)
GET    /api/projects/{id}/models                # versions + metrics (REQ-063,064)
GET    /api/models/{id}/weights                 # download best.pt
POST   /api/models/{id}/promote                 # make it the project's base model (REQ-064)

GET    /api/jobs?project_id=…                   REQ-070,071
GET    /api/jobs/{id}
POST   /api/jobs/{id}/cancel

Job flows

extract (REQ-020…023). ffmpeg -ss <start> -to <end> -i <video> -vf fps=<n> -q:v 2 frames/%06d.jpg. frames rows are written once the files exist; the batch's frame count is updated. Range and fps live on the batch, so one video can be used repeatedly.

autolabel (REQ-030…034). Per frame: one set_image, then loop each class's prompt (see the domain invariants in ../AGENTS.md), cross-prompt NMS, write annotations rows with source='auto'. A re-run deletes only source='auto' rows — manual corrections (source='manual') survive — and returns already-approved frames to pending, because that approval was given against labels that no longer exist. No overlay images are written: the review canvas draws the shapes from the annotation rows, so a second rendering of the same data on the server would only be a second thing to keep in sync.

Deleting a class (REQ-007). projects.delete_class removes the class's annotations, decrements every class_id above it in annotations and project_classes, then calls dataset.drop_class_from_labels to do the same edit to every .txt already written to disk, and rewrites data.yaml. The renumbering is the whole job: a YOLO label is an integer index, so a class list and a set of label files that disagree do not fail loudly — they train a model on the wrong names. Refused for a project's last class.

merge (REQ-050…053, REQ-131…132). One job covers the whole selected set of batches, and datasets.rules_json holds the triage rules frozen at the moment the merge was confirmed — the resolver is built from that snapshot, never from the project's live rules. For every approved frame not yet in dataset_items: assign a split (continuing the round-robin), copy the JPEG to dataset/images/<split>/, write the YOLO .txt from the frame's annotations, record the row. Finally rewrite data.yaml. Frames with no annotations produce an empty .txt (REQ-033).

train (REQ-060…065). Release the SAM3 engine → YOLO(base/model.pt) (or pretrained if the project has no base yet) → .train(data=dataset/data.yaml, **hardware.defaults()) → evaluate.py runs .val() for both the base model and the new one against the same data.yaml → store models/<n>/best.pt + metrics.json.

hardware.py picks defaults from the detected VRAM:

VRAM	batch	imgsz
< 8 GB	8	640
8–16 GB	16	640
> 16 GB	32	768

CPU-only: batch=4, imgsz=512, with a warning that training will be very slow. These are form defaults only (REQ-062).

Frontend

React + Vite, no heavy UI library; plain fetch, job polling once a second as today. Design system and validation checklist follow ui-ux-pro-max — see ../AGENTS.md §7.

Design constraints specific to this app:

• Dense tool, not a landing page. Neutral surface, one accent colour, tight spacing. The frame and the canvas own the screen; panels are chrome.
• Dark by default — annotation work happens on video stills, and a bright surround distorts the judgement of what's in the frame. Light mode is a toggle, not an afterthought.
• Class colours are data, not decoration. One fixed, colour-blind-safe hue per class index, identical in the filmstrip, the canvas, and the class panel.
• Keyboard first in Review. Every action there has a shortcut and a visible focus state; the mouse is for drawing shapes, not for navigating.
• Progress is always visible. Long jobs (extraction, auto-annotation, training) show progress, elapsed time, and a cancel affordance — never a spinner with no end.

Layout Architecture:

• Roboflow-Replica Layout: Dense left sidebar (Sidebar.jsx) with Workspace switcher, navigation sections:
  • WORKSPACE: Projects (/projects)
  • DATA: Video Archive (/projects/{id}), Annotate / Review (/batches/{id}), Master Dataset (/projects/{id}/dataset)
  • MODELS: Train & Select Engine (/projects/{id}/models), Model Monitoring & Analytics
  • DEPLOY: Model Deployments & Base Model Promotion (/projects/{id}/deploy)
• System Health Footer: Hardware GPU, Free VRAM, SAM3 readiness, and ffmpeg status.

Pages:

1. Projects — workspace project cards + new project form (name, label type, base model, video root, classes & prompts).
2. Library — dates column → videos/batches with duration, resolution, used marker.
3. Trim — <video> + in/out timeline, fps input, estimated frame count, extract button.
4. Review — status-coloured filmstrip, canvas editor, class panel, shortcuts (←/→ frame, A approve, X reject, Del delete shape), Approve batch button.
5. Models — model engine selection cards (Custom Training vs NAS/Pretrained), train button, job progress, base-vs-new mAP comparison table, download & promote.

The canvas editor is hand-written; the normalized-coordinate conventions already exist in sessions.py (annotations_payload, detections_payload) as a reference.

Docker (REQ-072)

• Dockerfile — python 3.12, ffmpeg, uv, CUDA torch, uv pip install -e sam3/. The known traps still apply: setuptools<81 (because sam3 imports pkg_resources) and the undeclared einops + pycocotools dependencies of the vendored SAM3.
• docker-compose.yml — a backend service (GPU passthrough, ./data volume, video archive mounted read-only, .env for HF_TOKEN) and a frontend service (nginx: static files + /api proxy).
• GPU access uses CDI (devices: nvidia.com/gpu=all), not the legacy runtime: nvidia. Docker Engine 27+ discovers nvidia.com/gpu through the container toolkit; the runtime entry in /etc/docker/daemon.json is not registered with the daemon here.
• The frontend pins Vite 7. Vite 8's default bundler (Rolldown) ships a native binding that dies with a bus error on this machine; Vite 7's Rollup path works.