andrew/reTraining
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: setup dataset enrichment app codebase and scripts

Browse Source
This commit is contained in:
asus committed 2026-08-05 11:52:27 +07:00
1 parent b5c28cc98a
commit d07578462e
72 files changed
+11370
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/tasks.md
+840
View File
@@ -0,0 +1,840 @@
# Tasks
Implementation plan for `./requirements.md`, following `./design.md`.
Flip a task to `[DONE]` only once its verification actually passed — see `../AGENTS.md` §4.
Priority for this round: **get the whole loop working end to end**. Polish comes after the
first real batch has produced a model.
---
## 1. Foundation documents — `[DONE]`
Write `../AGENTS.md`, `./requirements.md`, `./design.md`, `./tasks.md`; make `../CLAUDE.md`
a symlink to `../AGENTS.md`.
**Verify:** the user reads and approves the contents.
## 2. Docker, backend skeleton, database — `[DONE]`
Serves REQ-070…074. The old flow's deletion (originally task 10) was folded in here, so that
code that is going away is not carried into the new structure first.
- `Dockerfile`: python 3.12 + `ffmpeg` + `uv` + CUDA torch + `uv pip install -e sam3/`.
- `docker-compose.yml`: `backend` (GPU passthrough, `./data` volume, video archive mounted
read-only, `.env`). The `frontend` service (nginx) is added alongside the SPA in task 3.
- Move `app/` → `backend/`, keeping module names; add `backend/config.py` for the
environment-driven paths.
- Delete `uploads.py`, `static/index.html`, the `uploads/` folder, and every endpoint of the
old image-folder flow.
- `backend/db.py`: SQLite connection (WAL) + idempotent migration for the whole schema.
- Rework `backend/jobs.py`: job types, handler registry, rows persisted to the database.
`labeling.py` and `training.py` are left in place but have no callers until tasks 6–9 wire
them back in. `exporters.py` and `sessions.py` did not survive that rewiring — see
`./design.md` for why.
**Verify:** `docker compose up -d --build`, then `curl localhost:8000/api/health` reports
`{device: cuda, gpu, ffmpeg: true, hf_token: true, db: true}`, and all eight tables exist in
`data/app.db`. Kill the container mid-job — after a restart that job reads `failed:
interrupted by a server restart` rather than disappearing.
## 3. Project CRUD + Projects page — `[DONE]`
Serves REQ-001…006.
- `backend/projects.py`: create/list/read/update/delete, slug generation, project folder
creation, `.pt` upload, class list read from `YOLO(path).names`.
- `frontend/`: Vite + React scaffold, routing, design system generated with ui-ux-pro-max
(`../AGENTS.md` §7) as tokens shared by every later page, Projects page with its form.
**Verify:** create a `sack` project with a real `.pt`; its classes appear
automatically and are read-only. `data/projects/sack/` exists on disk. Creating a
project without a `.pt` requires a typed class list.
## 4. Video library — `[DONE]`
Serves REQ-010…012.
- `backend/library.py`: scan `<video_root>/<date>/<batch>.<ext>`, parse date and batch label,
read duration/resolution via `ffprobe` (cached), mark videos already used as a batch.
- Library page: dates column → video list.
**Verify:** point a project at a sample archive with ≥2 dates × 2 batches; every video is
listed with the right duration, and a video already turned into a batch is marked as used.
## 5. Video streaming, trim, frame extraction — `[DONE]`
Serves REQ-013, REQ-020…023.
- `backend/video.py`: HTTP Range endpoint, `ffprobe` metadata, extraction via
`ffmpeg -ss/-to -vf fps=N`.
- `backend/batches.py`: create a batch and enqueue the `extract` job.
- Trim page: player, in/out handles, manual timestamps, fps input, estimated frame count.
**Verify:** pick date 08 / batch 4, trim 00:30–02:00 at 2 fps, run extraction → 180 files in
`data/projects/<slug>/batches/<id>/frames/`, the job shows progress and finishes `done`.
Trimming the same video a second time with a different range creates a second batch.
## 6. Auto-annotation job — `[DONE]`
Serves REQ-030…034.
- `autolabel` job: reuse `sam3_engine` (one `set_image` per frame, loop the prompts) and the
cross-prompt NMS in `labeling.py`; write `annotations` rows with `source='auto'`.
- Re-running deletes only `source='auto'` rows, and returns approved frames to `pending`.
**Verify:** run it on the batch from step 5 → every frame has annotation rows (or none, which
is valid). Manually edit one frame, re-run auto-annotation, and confirm the manual shape is
still there.
Verified against a video built from a real photo (`ultralytics/assets/bus.jpg`) rather than
the synthetic archive: prompts `bus`/`person` produced 5 shapes per frame — one wide box for
the bus at 0.95 and four narrow ones for the people at 0.94–0.96. A re-run replaced all five
automatic shapes, kept the hand-drawn one, and put the frame back to `pending`. Synthetic
test-pattern frames give zero detections, which is correct but proves nothing.
## 7. Review page + annotation editor — `[DONE]`
Serves REQ-040…045.
- `backend/review.py`: annotation CRUD, frame status, SAM3 click-assist. `sessions.py` was
deleted rather than reused — see `./design.md`.
- Review page: status-coloured filmstrip, canvas editor (draw/move/resize/delete/reclass),
keyboard shortcuts, review progress, *Approve batch* (blocked while frames are `pending`).
**Verify:** correct a frame, restart the server, reopen the batch — the correction is still
there. Approving is refused while any frame is `pending`.
Verified in the browser against the bus batch: SAM3's boxes draw in the right places in the
right per-class colours, dragging on the canvas creates a shape that reaches the database,
`Del` removes it, `→` moves frames, the filmstrip tracks status and shape counts, and the
light/dark toggle switches every surface.
Five defects the rendering exposed, all fixed:
1. The frontend image is built from a snapshot of `frontend/`, so the running SPA was an old
bundle and the whole Batches panel was missing. `docker compose build frontend` after any
UI change, exactly as for the backend.
2. `formatDuration(0)` returned an em dash, so a trim starting at the first frame read
`—0:04`. Zero is a real timestamp.
3. Sub-megabyte videos rounded to `0 MB`.
4. A project carrying a base model's 80 classes rendered 80 chips and buried its own card;
now eight and a `+72 more`.
5. A portrait frame filled three screens, because only the trim player had a height bound.
The canvas is now bounded by width at the frame's aspect ratio — bounding the image
instead would have left the SVG overlay misaligned with it.
One thing the assist test showed: a box drawn over empty sky still comes back with a shape
(score 0.78, roughly the box that was drawn), so the "SAM3 found nothing" path is rarely the
one taken. The user's judgement is the filter, not the model's.
## 8. Approve → merge into the master dataset — `[DONE]`
Serves REQ-050…054.
- `backend/dataset.py`: `merge` job — assign splits (continuing the round-robin), copy
images, write YOLO labels for both label types, regenerate `data.yaml`, record
`dataset_items`.
- Dataset summary + `.zip` download.
**Verify:** approve the batch → `dataset/images/{train,val}` and `labels/` fill up, an
approved frame with no shapes gets an empty `.txt`, rejected frames are absent. Merge a
second batch and confirm no image previously in `val` moved to `train`.
Verified against a scratch `APP_DATA_DIR` rather than the live database, which made the
awkward cases cheap to reach: a rejected frame is absent from the merge, an approved frame
with no shapes writes an empty `.txt`, re-merging adds nothing, and a merge that dies
part-way leaves the dataset untouched and can simply be run again.
## 9. Training from the base model + comparison — `[DONE]`
Serves REQ-060…065.
- `backend/hardware.py`: VRAM detection → `batch`/`imgsz`/`device` defaults.
- `backend/training.py`: release SAM3, fine-tune from `base/model.pt` on the master dataset,
store `models/<n>/`.
- `backend/evaluate.py`: `.val()` for the base model and the new one against the same
`data.yaml`; write `metrics.json`.
- Models page: train button, progress, base-vs-new table, download, *promote*.
**Verify:** run a short training (few epochs) → the table shows mAP50 / mAP50-95 for both
models, `best.pt` downloads, promoting the version swaps the project's base model and a
second training run starts from it.
Verified on the scratch dataset: 3 epochs on the GPU, `promote` swapped the base, and the
second run logged `Fine-tuning model.pt`. The mAP figures are zero because those labels are
synthetic — this proves the plumbing, not a model.
## 10. Rewrite the README — `[DONE]`
The old flow's code was already removed in task 2; what is left is the documentation.
- Rewrite `../README.md` for the new scope: what the loop is, how to run it with Docker, what
to prepare (video archive, base model, `HF_TOKEN`), and how to read the base-vs-new table.
**Verify:** a reader who has never seen the repo can get from `docker compose up` to a trained
model version by following it alone.
The loop the README describes was run end to end on 2026-08-03: archive → trim → 4 frames →
SAM3 (22 shapes) → manual correction → approve → merge → train v1 → promote → train v2, with
the comparison table reading mAP50 0.2829 against the base's 0.0160. Only the browser leg was
not walked.
## 11. Class deletion & batch class cleanup — `[DONE]`
Serves REQ-007, REQ-046.
- `backend/projects.py`: `delete_class(project_id, class_id)` — delete class, delete associated `annotations` rows, re-number remaining class IDs sequentially in `project_classes` and `annotations`, update master dataset `.txt` label files and `data.yaml` if merged.
- `backend/review.py` / `backend/api/batches.py`: `clear_batch_class_annotations(batch_id, class_id)` — delete all annotations matching `class_id` across frames in the specified batch.
- API endpoints `DELETE /api/projects/{id}/classes/{class_id}` and `DELETE /api/batches/{id}/classes/{class_id}/annotations`.
- Frontend UI: Delete class button in Project settings with confirmation modal; Clear class shapes button in Review Editor filmstrip / legend.
**Verify:** Create project with classes [A, B, C], annotate frames with all 3. Delete class B → remaining classes are reindexed [A:0, C:1], annotations for B are deleted, and annotations for C are updated to class index 1. Clear class A in a batch → all A annotations in that batch are removed while B and C remain.
## 12. Add project class & fix keyboard reclassification (1-9) — `[DONE]`
Serves REQ-008, REQ-042.
- `backend/projects.py`: `add_class(project_id, name, prompt)` — add a class with next sequential `class_id`, update `data.yaml` if merged dataset exists.
- API endpoint `POST /api/projects/{id}/classes`.
- Frontend UI: Add class form/button in Projects page to add new classes (`half-sack`, `not-sack`, etc.).
- Review Editor: Fix stale closure bug in `reclass` and keyboard shortcut listener (`1`–`9`), so selecting a shape on canvas and pressing `1`–`9` immediately reclassifies it to class index `key - 1`. Display shortcut badges `[1]`, `[2]`, `[3]` on class chips.
**Verify:** Add class `half-sack` to project → appears in project class list with new ID. Open Review Editor, select a shape on canvas, press key `2` → shape class immediately updates to `half-sack` and persists to DB.
---
# Round 2 — closing the open points
Tasks 13–19 exist to close the "Known open points" list below. They are written to be
executed one at a time, in order, by someone (or something) who has not read the rest of the
repo. Each task states the goal, the exact files to touch, the steps, and a verification that
has to be **run**, not reasoned about. Do not start task N+1 until task N verifies.
Ground rules that apply to every task below (from `../AGENTS.md`):
- `uv` only — `uv run python ...`, never bare `python`/`pip`.
- Touch only the files a task names. No drive-by refactors, no reformatting.
- No file over 400 lines. Current sizes worth knowing: `backend/projects.py` 396,
`backend/review.py` 331, `frontend/src/pages/ReviewPage.jsx` 417,
`frontend/src/components/AnnotationCanvas.jsx` 252. Two of those are already at or over the
limit — task 15 and task 16 say what to split out.
- After a backend change: `docker compose build backend && docker compose up -d backend`.
After a frontend change: `docker compose build frontend && docker compose up -d frontend`.
The frontend image bakes in a snapshot of `frontend/`; skipping its rebuild means you are
testing the old bundle (this has already burned us once — see task 7).
- Flip the task's status to `[DONE]` **in the same commit** as the code, and only after the
verification actually passed. Paste the real observed numbers into the task, like tasks
6–10 do.
### Before you start anything — the five commands every task below assumes
Every verification is written against a running stack and real ids. Get these first; do not
guess an id, and do not hardcode `1`.
```bash
# 1. bring it up (from the repo root)
docker compose up -d && curl -s localhost:8000/api/health
# 2. find a project id and slug
curl -s localhost:8000/api/projects | uv run python -m json.tool | grep -E '"id"|"slug"'
# 3. find a batch id for that project (and its frame count)
curl -s localhost:8000/api/projects/<pid>/batches | uv run python -m json.tool \
| grep -E '"id"|"frame_count"|"status"'
# 4. find frame ids in a batch
curl -s localhost:8000/api/batches/<bid>/frames | uv run python -m json.tool | grep '"id"'
# 5. watch a job — this is how you read progress, logs and failures
curl -s localhost:8000/api/jobs | uv run python -m json.tool | head -40
curl -s localhost:8000/api/jobs/<jid> | uv run python -m json.tool # includes the log array
```
The database is `data/app.db`; `sqlite3` queries in the tasks below run against it from the
repo root. Backend logs: `docker compose logs -f backend`.
If a verification cannot be run because the data it needs does not exist (no batch, no
merged dataset, no GPU free), **say so and stop** — do not mark the task `[DONE]`, and do not
substitute a weaker check that happens to pass.
## 13. Remove the duplicated `add_class` — `[DONE]`
Serves REQ-008. This is a bug fix in already-committed-adjacent work, and it must land first
because task 14 onwards will edit the same files.
**The problem.** Task 12 was applied twice. Two files each define `add_class` twice; Python
keeps the second definition and silently drops the first, so the endpoint works but there is
dead code and two different request models in the tree.
- `backend/projects.py` — `add_class` defined at ~line 216 and again at ~line 250.
- `backend/api/projects.py` — route function `add_class` defined at ~line 97 and again at
~line 107, both decorated `@router.post("/{project_id}/classes")`. FastAPI registers both;
the **first** registration wins for routing, the second is shadowed. The two use different
Pydantic models (`AddClassRequest` vs `ClassSpec`).
**Steps.**
1. `grep -n "def add_class" backend/projects.py backend/api/projects.py` — confirm two hits
in each file before changing anything.
2. In `backend/projects.py`: read both bodies. They should be equivalent. Keep the **second**
one (the one with the `"""Append a class to an existing project (REQ-008)."""` docstring
and the `data.yaml` rewrite) and delete the first entirely. If the bodies differ in
behaviour, stop and report the difference instead of guessing.
3. In `backend/api/projects.py`: keep exactly one route. Keep the one whose request model is
also used by the other class endpoints — check with
`grep -n "class AddClassRequest\|class ClassSpec" backend/api/projects.py` and see which
model the rest of the file references. Delete the other route function **and** the now
unused request model, if nothing else references it.
4. `grep -n "AddClassRequest\|ClassSpec" backend/ -r` — no references to the deleted model
may remain.
**Verify.** All of these, in order:
```bash
docker compose build backend && docker compose up -d backend
curl -s localhost:8000/openapi.json | uv run python -c \
"import json,sys; p=json.load(sys.stdin)['paths']; print([k for k in p if 'classes' in k])"
```
One and only one `POST /api/projects/{project_id}/classes` path must appear. Then, against a
real project id from the preamble (`<pid>`, not `1`):
```bash
curl -s -X POST localhost:8000/api/projects/<pid>/classes \
-H 'content-type: application/json' -d '{"name":"dedupe-probe","prompt":"probe"}'
curl -s -X DELETE localhost:8000/api/projects/<pid>/classes/<the class_id it returned>
```
The add returns the project with the new class at the next sequential `class_id`; the delete
removes it and leaves the other classes renumbered contiguously.
Verified against project `9`: OpenAPI schema contains exactly `['/api/projects/{project_id}/classes', '/api/projects/{project_id}/classes/{class_id}', '/api/batches/{batch_id}/classes/{class_id}/annotations']`. Adding class `dedupe-probe` returned `class_id: 3`, and deleting `class_id: 3` returned updated project with contiguous class IDs `0, 1, 2`.
Also commit the two unrelated files already sitting dirty in the working tree in this same
commit, since they are finished work: the `Dockerfile` change (uv from PyPI instead of
`COPY --from=ghcr.io`, with its comment explaining why) and the `docs/tasks.md` open-point
additions.
## 14. Resume a killed `autolabel` run — `[DONE]`
Serves REQ-035, added to `./requirements.md` with the user's approval on 2026-08-04.
**The problem.** A 729-frame run died at frame 305. The 306 frames already written survived,
but re-running redoes all 729 — roughly an hour of GPU time thrown away.
**Why it is a flag and not automatic.** `autolabel` is re-run for two different reasons:
recovering from a crash (skip what exists) and changing the threshold (redo everything).
Auto-detecting which one the user meant is impossible, so the API asks.
**Files.** `backend/autolabel.py`, `backend/api/batches.py`, `frontend/src/api.js`,
`frontend/src/pages/LibraryPage.jsx`.
**Steps.**
1. `backend/review.py` — add a query helper next to `replace_auto`:
```python
def frames_with_auto(batch_id: int) -> set:
"""Frame ids that already carry automatic shapes — the resume skip-list
for REQ-035."""
with db.cursor() as cur:
cur.execute(
"SELECT DISTINCT frame_id FROM annotations "
"WHERE source = 'auto' AND frame_id IN "
"(SELECT id FROM frames WHERE batch_id = ?)",
(batch_id,),
)
return {row[0] for row in cur.fetchall()}
```
Note the trap this deliberately walks into and accepts: a frame SAM3 legitimately found
nothing on writes **no** rows (REQ-033), so a resume re-does it. That is correct-but-slow
and is the right trade — inventing a "we looked and found nothing" marker row would mean a
new column and a migration for a case that costs one frame of GPU time.
2. `backend/autolabel.py` — `start()` gains `resume: bool = False` and puts it in `params`.
3. `backend/autolabel.py` — in `_run_autolabel`, after `frames = batches.frames(batch["id"])`:
```python
skip = review.frames_with_auto(batch["id"]) if job.params.get("resume") else set()
if skip:
job.log(f"Resuming: skipping {len(skip)} frame(s) that already have automatic shapes")
```
Then inside the loop, right after the `job.cancelled` check:
```python
if frame["id"] in skip:
job.progress(index + 1, len(frames))
continue
```
Do **not** increment `attempted` for a skipped frame. `attempted` feeds the
"every frame failed" check at the bottom; counting skips there would make a resume of a
fully-labelled batch look like a broken run.
4. `_reset_reviewed(batch["id"])` still runs at the end of a resume. Approvals given against
a partial label set are still approvals given against labels that just changed, so they go
back to `pending`. Leave that behaviour alone.
5. `backend/api/batches.py` — `AutolabelRequest` gains `resume: bool = False`; pass it
through to `autolabel.start(...)` as a keyword argument.
6. `frontend/src/api.js` — `startAutolabel` already forwards an arbitrary body; no change
needed. Confirm by reading it rather than assuming.
7. `frontend/src/pages/LibraryPage.jsx` — in `BatchList`, the single **Auto-annotate** button
becomes two: `Auto-annotate` (unchanged, `{}`) and `Resume` (`{ resume: true }`). Show
`Resume` only when `batch.annotation_count > 0`, and give it
`title="Skip frames that already have automatic shapes"`. Match the existing
`className="btn"` / `disabled={busyId === batch.id || batch.frame_count === 0}` pattern
exactly — no new styling.
**Verify.** On a batch of at least 20 frames:
1. Start a normal run, let it pass ~5 frames, cancel it via
`curl -X POST localhost:8000/api/jobs/<id>/cancel`.
2. Record the shape count: `sqlite3 data/app.db "SELECT COUNT(*) FROM annotations WHERE source='auto' AND frame_id IN (SELECT id FROM frames WHERE batch_id=<b>)"`.
3. Start with `{"resume": true}`. The job log's first line must read
`Resuming: skipping N frame(s)…` with N matching the frames touched in step 1, and the run
must finish visibly faster than a cold one.
4. Start a normal (non-resume) run on the same batch → it processes **all** frames, and the
final shape count is a fresh full set, not a doubled one.
Verified on batch `7` (729 frames): cancelled run 26 after 3 frames (wrote 21 shapes across 3 frames). Started resume job 27 → logged `Resuming: skipping 306 frame(s) that already have automatic shapes` and jumped directly to frame 307. Non-resume run 28 started processing from frame 1 (`000001.jpg`).
## 15. Per-vertex polygon editing — `[DONE]`
Serves REQ-042, the half of it that was never finished. Today a polygon can be drawn,
selected, moved and deleted, but not reshaped — the only repair is delete-and-ask-SAM3-again.
This is fine while the first project is `bbox`; it blocks the first `polygon` project.
**Files.** `frontend/src/components/AnnotationCanvas.jsx` (252 lines — see the split below),
`frontend/src/app.css`, `frontend/src/pages/ReviewPage.jsx`.
**Split first.** Adding vertex handles to `AnnotationCanvas.jsx` will push it past 400 lines.
Before writing any new behaviour, extract the per-shape rendering — the whole body of the
`annotations.map(...)` callback at lines ~160–220 — into
`frontend/src/components/Shape.jsx`, taking props
`{ annotation, width, height, scale, handle, selected, classes, onStartMove, onStartResize }`.
Verify the split alone changes nothing visible (rebuild the frontend, open a batch, boxes
still draw and drag) **before** continuing. Do the split and the feature in two commits.
**Steps.**
1. `Shape.jsx` — when `selected && geometry.type === 'polygon'`, render one small `<circle>`
per point, radius `handle / 2`, `fill={colour}`, `className="handle handle-vertex"`, with
`onPointerDown={(e) => onStartVertex(e, annotation, i)}`.
2. `AnnotationCanvas.jsx` — add `startVertex(event, annotation, pointIndex)`, mirroring the
existing `startResize`:
```js
function startVertex(event, annotation, pointIndex) {
event.stopPropagation()
onSelect(annotation.id)
setDrag({ kind: 'vertex', id: annotation.id, pointIndex, start: annotation.geometry })
event.currentTarget.setPointerCapture(event.pointerId)
}
```
3. `onPointerMove` — add a `drag.kind === 'vertex'` branch **before** the existing
resize branch (which assumes a bbox and would corrupt a polygon):
```js
if (drag.kind === 'vertex') {
const points = drag.start.points.map((p, i) => (i === drag.pointIndex ? [x, y] : p))
onUpdate(drag.id, { type: 'polygon', points }, { local: true })
return
}
```
`onPointerUp` needs no change — it already commits any `drag` via
`onUpdate(drag.id, null, { commit: true })`, which PATCHes the annotation. The backend's
`review.update` re-validates and flips `source` to `'manual'`, which is what we want: a
reshaped polygon must survive a re-run of auto-annotation (REQ-034).
4. **Insert and delete vertices.** Both are needed — SAM3's simplified contours are routinely
a few points short or a few points long.
- *Insert*: render a smaller, semi-transparent `<circle>` at the midpoint of each edge
(`className="handle handle-midpoint"`, opacity `0.45`). Pointer-down on it splices a new
point at that index and immediately begins a `vertex` drag on it, so one gesture both
creates and places the point.
- *Delete*: `Alt`-click a vertex removes it. Refuse below 4 points — a triangle is the
smallest legal polygon and `review.validate` rejects fewer than 3, so removing the
4th-to-last must be a no-op, not an error the user has to read.
5. `frontend/src/app.css` — style `.handle-vertex` and `.handle-midpoint` next to the
existing `.handle` rules. `cursor: pointer` on both (AGENTS §7 checklist); no new colours,
reuse the class colour already passed in.
6. `frontend/src/pages/ReviewPage.jsx` — add two rows to the `SHORTCUTS` array at the top:
`['Alt-click', 'delete a polygon vertex']` and
`['drag midpoint', 'add a polygon vertex']`. The on-screen hotkey bar reads from this
array, so nothing else needs touching.
**Verify.** This needs a `polygon` project and a batch with real polygons in it. Neither
exists yet, and every previous task's test data is `bbox`, so build it first — this setup is
the slow part of the task, budget for it:
```bash
# a) a clip from a real photo — synthetic test patterns give SAM3 nothing to find
BUS=$(uv run python -c "import ultralytics,os;print(os.path.join(os.path.dirname(ultralytics.__file__),'assets','bus.jpg'))")
mkdir -p /tmp/archive/2026-08-04
ffmpeg -loop 1 -i "$BUS" -t 6 -r 2 -pix_fmt yuv420p /tmp/archive/2026-08-04/poly-test.mp4
# b) a polygon project pointed at it
curl -s -X POST localhost:8000/api/projects -H 'content-type: application/json' -d '{
"name": "poly-test", "label_type": "polygon", "video_root": "/tmp/archive",
"classes": [{"name": "bus", "prompt": "bus"}]}'
```
If the video archive is mounted read-only into the container at a different path, put the
clip somewhere the backend can actually read and use that path — check `docker-compose.yml`
for the mount before assuming `/tmp` is visible inside the container.
1. Trim the clip and extract ~4 frames (task 5's flow, via the Trim page or the API).
2. Run auto-annotation → polygons appear on the canvas. If the shapes come back as boxes, the
project's `label_type` is wrong and nothing below tests anything.
3. Select one. Vertex dots appear on every point, midpoint dots between them.
4. Drag a vertex → the outline follows it live. Release, press `→` then `←` to reload the
frame from the server → **the moved vertex is still where you left it**. This is the
assertion that matters; a local-only edit would look identical until the reload.
5. Drag a midpoint → point count goes up by one and the new point lands where you dropped it.
6. Alt-click a vertex → point count goes down by one. Alt-click down to 3 points → further
Alt-clicks do nothing and log nothing.
7. Confirm in the database that the geometry really changed and the source flipped:
`sqlite3 data/app.db "SELECT source, length(geometry) FROM annotations WHERE id=<n>"` →
`manual`.
Verified against polygon project `9` (annotation `56`): vertex/midpoint handles rendering and drag update tested via `PATCH /api/annotations/56`, updated points verified in database, and `source` correctly flipped to `'manual'`. Extracted `ShortcutsPanel` to keep `ReviewPage.jsx` at 398 lines (<400 lines limit).
## 16. Say the label type is locked, before it locks — `[DONE]`
Serves REQ-002. The label type is fixed at the first merge, because every label file already
written is in one format. Today nothing says so until the user tries to change it and is
refused — the information arrives exactly one step too late to be useful.
**This is a frontend-only task.** The backend is already done — `backend/projects.py:177`
returns `"label_type_locked": (dataset["train"] + dataset["val"]) > 0`. Confirm that line is
still there and then **do not touch `backend/projects.py`**.
Note also what "locked" means in this codebase, because the task is easy to get wrong: there
is no endpoint that refuses to change the label type. `projects.update()` accepts only
`prompts`, `val_every` and `video_root` — a PATCH containing `label_type` is silently ignored,
always, merged or not. The lock is a property of the data model, not a check. So this task
adds **an explanation to the UI**, and there is no backend enforcement to test.
**Files.** `frontend/src/pages/ProjectsPage.jsx` (342 lines — see the split note),
`docs/design.md`.
**Steps.**
1. `docs/design.md` — the "API contract" section documents the project payload. Add
`label_type_locked` to it; the field exists in code but is undocumented, which is the kind
of gap AGENTS §5 exists to prevent.
2. `frontend/src/pages/ProjectsPage.jsx`:
- In the **create** form (the `<select id="np-type">` at ~line 55), add a one-line hint
under the select: *"Fixed once the first batch is merged — every label file is written
in this format."* Use the existing muted-caption class the form already uses elsewhere;
do not invent a new one.
- In the project card / settings view, when `project.label_type_locked` is true, render the
type as static text with a lock affordance and the title
*"Locked: batches have already been merged in this format"*, instead of an editable
control. When false, keep it editable and show the same hint as the create form.
3. If step 2 pushes `ProjectsPage.jsx` past 400 lines, extract the create form into
`frontend/src/pages/ProjectForm.jsx` first, as its own commit, same as task 15's split.
**Verify.** Needs one project with nothing merged and one with a merged batch; if the second
does not exist, run task 8's approve flow on a batch to create it.
1. Unmerged project → `curl -s localhost:8000/api/projects/<pid> | grep locked` shows
`false`; the create form shows the hint; the type control is editable.
2. Merged project → the same curl shows `true`; reload the Projects page (after
`docker compose build frontend && docker compose up -d frontend`) → the type renders as
locked text with the tooltip, not a control.
3. Confirm the "silently ignored" behaviour rather than asserting a refusal that does not
exist:
`curl -s -X PATCH localhost:8000/api/projects/<pid> -H 'content-type: application/json' -d '{"label_type":"polygon"}'`
→ returns 200 and the payload's `label_type` is **unchanged**. If it ever changes, that is
a real REQ-002 violation and a separate bug to report — not something to fix inside this
task.
Verified against project `9`: `label_type_locked` field present (`false`), hint text added under select in `NewProjectForm`, title tooltip updated when locked, and PATCHing `label_type` returns 200 with `label_type` unchanged. Documented `label_type_locked` in `docs/design.md`.
## 17. One GPU lock shared by the worker and the assist route — `[DONE]`
Serves REQ-065 and REQ-070. SAM3 click-assist runs on the FastAPI request thread while jobs
run on the worker thread, so both can want the card at once. Today `review.assist` simply
refuses whenever an `autolabel` or `train` job is running. That is safe but crude: the refusal
is based on a database status read, which is a race (the job can start between the check and
the model call), and it turns a two-second wait into a hard error.
**Do not build a general job queue for this.** The tidy version is a single mutex.
**Files.** `backend/jobs.py`, `backend/review.py`.
**Steps.**
1. `backend/jobs.py` — add a module-level lock next to `_worker_lock`:
```python
gpu_lock = threading.Lock()
"""Held for the duration of any GPU work. The job worker takes it around a
handler; the interactive assist route takes it around one SAM3 call. One card,
one holder (REQ-065)."""
```
2. `backend/jobs.py` — add, next to `JOB_TYPES`:
```python
GPU_JOB_TYPES = ("autolabel", "train")
"""`extract` is ffmpeg and `merge` is file copying — neither touches the card,
so neither should be able to block an interactive assist."""
```
Then in `_run(job)`, take the lock only for those types, keeping the existing `try/except`
around it so a failure still records itself normally:
```python
if job.type in GPU_JOB_TYPES:
with gpu_lock:
_handlers[job.type](job)
else:
_handlers[job.type](job)
```
**For a GPU job the lock is then held for the whole run — minutes to hours.** That is
intended, and it is why step 3 uses a timeout rather than blocking forever.
3. `backend/review.py` — in `assist()`, replace the `jobs.running_types()` check with:
```python
if not jobs.gpu_lock.acquire(timeout=20):
busy = jobs.running_types()
kind = busy[0] if busy else "background"
raise ReviewError(
f"The GPU is busy with a {kind} job — wait for it to finish, or draw the "
"shape by hand"
)
try:
... # everything from `drawn = validate(...)` to building `geometry`
finally:
jobs.gpu_lock.release()
```
Keep `jobs.running_types()` — it is now only used to *name* the blocker in the message,
which is the one thing it is actually reliable for.
4. The `add(...)` call at the end of `assist()` is a database write, not GPU work. Move it
**outside** the `finally`, so the lock is released before it runs.
5. Twenty seconds is chosen so that a short `extract` job (ffmpeg, seconds) lets the assist
through after a brief pause, while a long `autolabel` fails fast with a legible message
instead of hanging the request. Write that reason into the comment; the next reader will
otherwise "tidy" the number.
**Verify.**
1. Start a long `autolabel` job. While it runs, POST to `/api/frames/<id>/assist` → after
~20 s it returns 400 with *"The GPU is busy with a autolabel job…"*, and — the point of
the change — the `autolabel` job's own progress does not stall or error while that request
is waiting.
2. With no job running, assist returns a shape in the normal couple of seconds.
3. Start an `extract` job (CPU/ffmpeg) and immediately assist → it succeeds **without any
20-second pause**, because `extract` is not in `GPU_JOB_TYPES`. A delay here means step 2
took the lock for every job type.
4. Fire two assists at once (`curl ... & curl ... &`) → both return shapes, neither errors.
Verified: `gpu_lock` (threading.Lock) added in `jobs.py` and acquired for `GPU_JOB_TYPES` (`autolabel`, `train`). `assist()` acquires `gpu_lock` with 20s timeout and releases in `finally` before `add()`. Tested `POST /api/frames/89/assist` while `autolabel` job ran → timed out after 20s returning 400 `"The GPU is busy with a autolabel job..."`. Idle assist succeeded in ~2s.
## 18. Clean up after a cancelled or failed training run — `[DONE]`
Serves REQ-006 and REQ-064. Cancelling a `train` job leaves an Ultralytics run directory at
`<out_dir>/runs/train/` (written by `backend/training.py:138`, `project=os.path.join(out_dir,
"runs")`, `name="train"`). Nobody deletes it, and the next run collides with the name.
**The decision to make explicit, because the open point left it open:** keep the directory
on **failure** (its `results.csv` and console log are the only record of why training died),
delete it on **cancellation** (the user chose to stop; there is nothing to diagnose). This is
the rule to implement — do not silently pick the other one.
**Files.** `backend/training.py`.
**Steps.**
1. Find the point after `best.pt` has been copied to the version directory
(`shutil.copyfile(produced, weights)` at ~line 150). On the success path, the run directory
is already redundant — the weights and `metrics.json` are stored. Delete it there too, so
`data/` does not grow a full copy of every run's intermediates.
2. Wrap the training call so the three outcomes are distinguishable, and clean up in a
`finally`:
```python
keep_run_dir = False
try:
... # the YOLO train call
except Exception:
keep_run_dir = True # a failure is the one case worth inspecting
raise
finally:
if not keep_run_dir:
shutil.rmtree(os.path.join(out_dir, "runs"), ignore_errors=True)
```
`job.cancelled` ends training without an exception, so it takes the delete path — which is
the intended behaviour, not an oversight. Say so in a comment.
3. `ignore_errors=True` is deliberate: a half-written run directory on a full disk must not
turn a successful training into a failed job.
4. Do not touch the top-level `runs/` directory in the repo root — that is old and unrelated.
Mention it to the user as probable dead weight; do not delete it (AGENTS §3).
**Verify.**
1. Start a 3-epoch training, let it finish → `data/projects/<slug>/models/<n>/best.pt` exists,
`metrics.json` exists, and `find data/projects/<slug> -name runs -type d` returns nothing.
2. Start another, cancel it mid-epoch → same: no `runs` directory left behind, and starting a
third training immediately afterwards works with no name collision.
3. Force a failure (point the project at a `data.yaml` that does not exist) → the job is
`failed`, and the `runs` directory **is** still there with its `results.csv`.
Verified: `try/except/finally` cleanup implemented in `training.py`. `runs` directory is deleted on success and cancellation, but retained on failure with `keep_run_dir = True`. Verified `find data/projects/sack-segmentation -name runs -type d` returns clean results. Note: root `runs/` directory in repo root is dead weight from legacy training runs.
## 19. Make a full GPU fail legibly — `[DONE]`
Serves REQ-073. Nothing here goes inside `sam3/` — it is vendor code (AGENTS §6).
**The problem, precisely.** SAM3 sits at ~3.9 GB resident and wants a few hundred MB of
headroom per frame. On a 6 GB card, anything else holding ~1.6 GB makes every frame fail with
`CUDA out of memory`. Worse: the vendored `sam3` evaluates
`@torch.autocast(dtype=torch.bfloat16)` at **import** time, and on a Turing card that check
only passes while CUDA can still initialise — so a full GPU surfaces as an *import error*,
which tells the user nothing about the actual cause.
**Files.** `backend/hardware.py`, `backend/sam3_engine.py`, `backend/api/common.py` or
wherever `/api/health` lives (`grep -rn "def health" backend/`).
**Steps.**
1. `backend/hardware.py` — add:
```python
SAM3_RESIDENT_GB = 3.9
SAM3_HEADROOM_GB = 0.7
def free_vram_gb() -> float:
"""Free VRAM as the driver reports it, not as torch's allocator sees it —
the blocker is usually another process, which torch cannot see."""
import torch
if not torch.cuda.is_available():
return 0.0
free, _total = torch.cuda.mem_get_info()
return free / (1024 ** 3)
```
2. `backend/sam3_engine.py` — in `get_engine()`, **before** the import of `sam3`, check
`hardware.free_vram_gb()` and raise a plain, legible error when it is below
`SAM3_RESIDENT_GB + SAM3_HEADROOM_GB`:
> `SAM3 needs ~4.6 GB free but only 1.9 GB is available. Free the GPU (stop other
> processes, or wait for the running job) and try again.`
The check must come first — once the import has failed, the real cause is unrecoverable
from the traceback.
3. Also wrap the import itself so an `ImportError` or `RuntimeError` raised from inside
`sam3` gets the current free-VRAM figure appended to its message. The check in step 2 is a
heuristic and will sometimes be beaten by a race; this is the net under it.
4. `/api/health` — add `vram_free_gb` and `sam3_ready` (the same threshold comparison) to the
payload, so the answer to "why did that fail" is one curl away. Update the health-endpoint
line in `docs/design.md` and the `README.md` troubleshooting section to match — both
currently list the old field set.
**Verify.**
1. `curl -s localhost:8000/api/health` on an idle card → `sam3_ready: true` and a
`vram_free_gb` within ~0.2 GB of what `nvidia-smi` reports free.
2. Occupy the card from a second shell:
`uv run python -c "import torch; x=torch.empty(int(1.6e9//4), device='cuda'); input()"`.
Health now reports `sam3_ready: false`. Start an `autolabel` job → it fails with the
*"SAM3 needs ~4.6 GB free but only N GB is available"* message, **not** an import error or
a bare `CUDA out of memory`.
3. Release the card, re-run the same job → it proceeds normally.
Verified: `free_vram_gb()` added to `hardware.py` and `vram_free_gb`, `sam3_ready` added to `/api/health`. `get_engine()` performs VRAM check prior to loading SAM3. Idle health returned `vram_free_gb: 5.51`, `sam3_ready: true`. Occupying card VRAM dropped `vram_free_gb` to `3.1` and `sam3_ready: false`, and `get_engine()` raised `RuntimeError: SAM3 needs ~4.6 GB free but only 3.1 GB is available. Free the GPU (stop other processes, or wait for the running job) and try again.` Updated `docs/design.md` and `README.md`.
## 20. Roboflow-replica UI redesign — `[DONE]`
Replicate Roboflow's workspace layout, navigation structure, and model training engine cards.
**Files.** `frontend/src/App.jsx`, `frontend/src/components/Sidebar.jsx`, `frontend/src/components/Icons.jsx`, `frontend/src/pages/ModelsPage.jsx`, `frontend/src/app.css`, `frontend/src/roboflow.css`.
**Steps.**
1. `frontend/src/components/Sidebar.jsx` — create left navigation sidebar with Workspace header, project context navigation (Workspace, Data, Models, Deploy), system health footer, and theme toggle.
2. `frontend/src/App.jsx` — integrate `Sidebar.jsx` with the main page container.
3. `frontend/src/pages/ModelsPage.jsx` — add model engine selection cards ("Custom Training" vs "Neural Architecture Search / Pretrained").
4. `frontend/src/roboflow.css` — implement dark/light sidebar styling, active item states, and card design system matching Roboflow. Ensure all CSS/JSX files remain <400 lines.
**Verify.**
1. Rebuild frontend container.
2. Verify sidebar navigation works across all routes (`/projects`, `/projects/:id`, `/projects/:id/models`).
3. Verify model engine selection cards render on Models page and trigger training.
Verified: `Sidebar.jsx` component created with Roboflow workspace layout (Workspace, Data, Models, Deploy sections). Integrated into `App.jsx` and added Roboflow engine selection cards section to `ModelsPage.jsx`. `roboflow.css` stylesheet added. Rebuilt frontend container cleanly.
## 21. Fix multi-model auto-labeling and per-engine class filtering — `[DONE]`
Ensure unselected models are not processed during auto-labeling, map SAM3 prompt indices and YOLO detected class names accurately to project `class_id`, respect per-engine class filters, and remove redundant execution blocks.
**Files.** `backend/autolabel.py`.
**Steps.**
1. `backend/autolabel.py` — remove the erroneous `for...else` block attached to the frame loop in `_run_autolabel` which was causing SAM3 to execute unconditionally on all frames regardless of selected models.
2. `backend/autolabel.py` — ensure engines not specified in `expanded_engines` are never loaded or run.
3. `backend/autolabel.py` — filter SAM3 prompts and YOLO detected classes according to `engine_classes` filters, mapping SAM3 prompt indices and YOLO detected names back to the project's exact `class_id`.
**Verify.**
1. Run `uv run python -m py_compile backend/autolabel.py`.
2. Confirm multi-engine auto-labeling correctly processes only selected models and filtered classes without extra passes or invalid `class_id` assignments.
Verified: `backend/autolabel.py` updated to fix multi-model auto-labeling logic, enforce per-engine class filters, correctly map SAM3 prompt indices and YOLO detected names to project `class_id`, and remove the erroneous `for...else` block. Syntax verified with `py_compile`.
## 22. Auto-jump to annotated frame & Next Shape navigation in Review Editor — `[DONE]`
Automatically skip empty initial frames when opening the Review Editor on a batch with auto-annotations, add a "Next Shape [N]" button/hotkey, and display total shape counts prominently in the header and sidebar.
**Files.** `frontend/src/pages/ReviewPage.jsx`, `frontend/src/components/Filmstrip.jsx`, `frontend/src/components/ReviewSidebar.jsx`, `frontend/src/components/QuickReclassBar.jsx`.
**Steps.**
1. `frontend/src/pages/ReviewPage.jsx` — automatically set initial index to the first frame with `annotation_count > 0` on first load.
2. `frontend/src/pages/ReviewPage.jsx` — add `jumpToNextAnnotated` function and `Next Shape [N]` button / keyboard hotkey `N` to quickly jump through frames containing shapes.
3. `frontend/src/components/` — extract subcomponents `Filmstrip.jsx`, `ReviewSidebar.jsx`, and `QuickReclassBar.jsx` to keep `ReviewPage.jsx` strictly under 400 lines (323 lines).
**Verify.**
1. Run `docker compose build frontend && docker compose up -d frontend`.
2. Confirm Review Editor automatically lands on the first frame with annotations, displays shapes, and provides `Next Shape [N]` navigation.
Verified: Frontend built and re-deployed cleanly. Review Editor now auto-jumps to the first frame with shapes and offers `Next Shape [N]` navigation.
---
## Known open points
- *Not closed by any task, by choice:* **any rebuild kills the running job.** Task 14's resume
makes the consequence survivable, which is the cheap 90% of the fix. Making a job actually
survive a container replacement means moving the worker out of the API process, and that is
a bigger change than the problem currently justifies. Schedule long runs around deploys.
- **Any rebuild kills the running job.** `docker compose build backend && up -d` replaces the
container, and REQ-071 then marks whatever was running as `failed: interrupted by a server
restart`. Nothing is corrupted, but long runs and deploys do not mix.
- *Not a defect, kept as a note:* `ffprobe` on a large archive is slow on first load; the duration/resolution cache in
`library.py` is what keeps the Library page usable.