diff --git a/AGENTS.md b/AGENTS.md index bd63075..1593217 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,8 +5,6 @@ guidelines](https://github.com/multica-ai/andrej-karpathy-skills) and the [Chain of Truth](https://faridsurya-dev.github.io/Vibe-Coding-Research/en/welcome) method: **validated artifacts are the source of truth, AI is a generator and accelerator.** -Please Response in extremely concise and precise. - ## 1. Think Before Coding **Don't assume. Don't hide confusion. Surface tradeoffs.** @@ -56,14 +54,12 @@ endpoint, run the job, look at the files it produced. `docs/` is the source of truth, not the chat prompt. - | Document | Contents | | ---------------------- | ----------------------------------------------------------------------------------- | | `docs/requirements.md` | Numbered `REQ-xxx` requirements. Changes only with the user's approval. | | `docs/design.md` | Data schema, API contract, disk layout. Each section names the `REQ-xxx` it serves. | | `docs/tasks.md` | Implementation steps + verification criteria, status `[TODO]`/`[DONE]`. | - The rules: - Before writing feature code, make sure a `REQ-xxx` covers it. If none does, propose @@ -82,14 +78,45 @@ scripts with `uv run`. The Docker image installs the same file, so the two envir cannot drift. - **File size limit: 400 lines.** Any new or refactored file that exceeds it must be split into smaller, logical modules. -- `**sam3/` is a vendor copy** of Meta's library. It's a dependency, not app code — don't -add scripts there or edit anything inside it. +- **`sam3/` is a vendored git submodule** (Meta's `facebookresearch/sam3`). It's a +dependency, not app code — don't add scripts there or edit anything inside it. It's +installed with `uv pip install -e ./sam3`. - **Never write into the user's video archive.** All output goes under `data/`. - Secrets (`HF_TOKEN`) come from `.env` only; they never belong in code or docs. -## 7. UI/UX +## 7. Architecture -Frontend work follows [ui-ux-pro-max](https://github.com/nextlevelbuilder/ui-ux-pro-max-skill): +**Two backends can coexist** — bare-metal (port 8000) and Docker (port 9010). The +`start.sh` script auto-detects GPU, generates `docker-compose.override.yml` for CDI +GPU passthrough, and falls back to bare-metal if Docker is unavailable. + +``` +backend/ FastAPI app (main.py), API routes in backend/api/ +frontend/ React 19 + Vite 7 SPA; dev on 5173, Docker on 9000 +algoritma-batch/ Counting engine (tracker, line counter). In Docker image it + is copied to /app/src/ and /app/cfg/ — imports resolve as + `src.*`, not `algoritma-batch.*`. +data/ SQLite DB, projects, extracted frames, datasets, model weights +docs/ Chain of Truth: requirements, design, tasks, UI spec +``` + +**Port mapping:** + +| Service | Bare-metal | Docker | +|-----------------|:----------:|:------:| +| Backend API | 8000 | 9010 | +| Frontend Web UI | 5173 | 9000 | + +**Key entrypoints:** + +- `backend/main.py` — FastAPI app, lifespan, health endpoint, router wiring +- `backend/jobs.py` — single-threaded GPU-locked job worker (extract, autolabel, merge, train, count, clock-scan, truck-scan) +- `backend/sam3_engine.py` — SAM3 vision backbone, loaded once per process +- `backend/hardware.py` — GPU VRAM detection and training batch-size defaults + +## 8. UI/UX + +Frontend work follows [ui-ux-pro-max](https://github.com/nextlevelbuilder/ui--pro-max-skill): generate the design system first (style, palette, typography), then build against it, then validate before delivering. Consistency across pages beats per-page cleverness. @@ -109,18 +136,19 @@ Pre-delivery checklist — a UI task is not `[DONE]` until all of it passes: The review editor also has to survive keyboard-only use — see `docs/design.md`, "Frontend". -## 8. Domain invariants +## 9. Domain invariants Two things are easy to break without noticing, and breaking either makes the whole system lie: 1. **Stable val split.** Once a frame lands in `val`, it stays in `val` forever. Otherwise -he base-vs-new mAP comparison is meaningless. +the base-vs-new mAP comparison is meaningless. 2. **One `set_image` per image.** `Sam3Processor.set_image()` runs the vision backbone; -set_text_prompt()`only re-runs the grounding head against the cached`backbone_out`. n N-prompt job calls` set_image` **once per image** and loops prompts over that same -tate. Don't restructure this into set_image-per-prompt. +`set_text_prompt()` only re-runs the grounding head against the cached `backbone_out`. An +N-prompt job calls `set_image` **once per image** and loops prompts over that same state. +Don't restructure this into set_image-per-prompt. -## 9. Scalability & Portability +## 10. Scalability & Portability **Never hardcode something that will change across environments.** diff --git a/backend/api/review.py b/backend/api/review.py index 0969dec..bd1ba15 100644 --- a/backend/api/review.py +++ b/backend/api/review.py @@ -41,9 +41,11 @@ class AssistRequest(BaseModel): class PoolExemplar(BaseModel): - # Normalized xyxy against the frame, as drawn on the review canvas. - box: List[float] + # Normalized xyxy box or [x, y] point against the frame. + box: Optional[List[float]] = None + point: Optional[List[float]] = None positive: bool = True + kind: Optional[str] = None class ExemplarLabelRequest(BaseModel): diff --git a/backend/exemplar.py b/backend/exemplar.py index eeae72b..6fa90db 100644 --- a/backend/exemplar.py +++ b/backend/exemplar.py @@ -143,8 +143,17 @@ def label(frame_id: int, class_id: int, exemplars: List[dict], positives, negatives = [], [] for item in exemplars: - box = review.validate({"type": "bbox", "points": item["box"]}, "bbox")["points"] - (positives if item.get("positive", True) else negatives).append(box) + is_pos = bool(item.get("positive", True)) + target_list = positives if is_pos else negatives + if item.get("point") is not None: + px, py = float(item["point"][0]), float(item["point"][1]) + target_list.append({"kind": "point", "cxcywh": [px, py, 0.03, 0.03], "point": [px, py]}) + elif item.get("box") is not None: + box = review.validate({"type": "bbox", "points": item["box"]}, "bbox")["points"] + target_list.append({"kind": "box", "cxcywh": _cxcywh(box), "box": box}) + elif "points" in item and len(item["points"]) == 2: + px, py = float(item["points"][0]), float(item["points"][1]) + target_list.append({"kind": "point", "cxcywh": [px, py, 0.03, 0.03], "point": [px, py]}) if not positives and not negatives: raise review.ReviewError("No exemplars to run") @@ -154,15 +163,11 @@ def label(frame_id: int, class_id: int, exemplars: List[dict], if not jobs.gpu_lock.acquire(timeout=5): busy = jobs.running_types() kind = busy[0] if busy else "background" - drawn = [{"geometry": _rect(box, label_type), "score": 1.0, "source": "manual"} - for box in positives] + drawn = [{"geometry": _rect(p["box"], label_type), "score": 1.0, "source": "manual"} + for p in positives if p["kind"] == "box"] if apply: - # Never the replace path here: with no detections to put back, it - # would wipe the class and leave only the drawings. Applying a - # detection-less run just files the drawings and honours the - # negatives. _append_drawn(frame_id, class_id, drawn) - _drop_negative_overlaps(frame_id, class_id, negatives) + _drop_negative_overlaps(frame_id, class_id, [n["box"] for n in negatives if n["kind"] == "box"]) return _result(frame_id, drawn, apply, redetected=False, message=f"The GPU is busy with a {kind} job — this is your drawing " "only, nothing was detected") @@ -178,8 +183,8 @@ def label(frame_id: int, class_id: int, exemplars: List[dict], state = engine.open_state(image) found = engine.apply_prompts( state, threshold=threshold, text=prompt, - exemplars=[{"box": _cxcywh(box), "positive": True} for box in positives] - + [{"box": _cxcywh(box), "positive": False} for box in negatives], + exemplars=[{"box": p["cxcywh"], "positive": True} for p in positives] + + [{"box": n["cxcywh"], "positive": False} for n in negatives], ) finally: jobs.gpu_lock.release() @@ -198,9 +203,11 @@ def label(frame_id: int, class_id: int, exemplars: List[dict], detections = [(_norm_box(d.box, width, height), d) for d in found] items: List[dict] = [] - # The user's own boxes first, so the duplicate check below measures against - # what they drew rather than the other way round. - for box in positives: + # Manual drawn boxes first: + for item in positives: + if item["kind"] != "box": + continue + box = item["box"] geometry = _rect(box, label_type) if label_type != "bbox": snapped = _snap(box, detections, width, height) @@ -208,10 +215,16 @@ def label(frame_id: int, class_id: int, exemplars: List[dict], geometry = snapped items.append({"geometry": geometry, "score": 1.0, "source": "manual"}) + manual_boxes = [p["box"] for p in positives if p["kind"] == "box"] + negative_boxes = [n["box"] for n in negatives if n["kind"] == "box"] + negative_points = [n["point"] for n in negatives if n["kind"] == "point"] + for norm, detection in detections: - if any(_iou(norm, box) >= NEGATIVE_IOU for box in negatives): + if any(_iou(norm, box) >= NEGATIVE_IOU for box in negative_boxes): continue - if any(_iou(norm, box) >= DUPLICATE_IOU for box in positives): + if any(norm[0] <= pt[0] <= norm[2] and norm[1] <= pt[1] <= norm[3] for pt in negative_points): + continue + if any(_iou(norm, box) >= DUPLICATE_IOU for box in manual_boxes): continue for geometry in _detection_shapes(detection, norm, width, height, label_type): items.append({"geometry": geometry, "score": detection.score, "source": "auto"}) diff --git a/frontend/src/app.css b/frontend/src/app.css index eb1b312..a3486d7 100644 --- a/frontend/src/app.css +++ b/frontend/src/app.css @@ -710,7 +710,7 @@ main.page { .shortcuts { margin: 0; display: grid; gap: 5px; font-size: 12px; } .shortcuts > div { display: flex; gap: 10px; align-items: baseline; } -.shortcuts dt { flex: none; width: 62px; } +.shortcuts dt { flex: none; width: 72px; } .shortcuts dd { margin: 0; color: var(--text-muted); } kbd { diff --git a/frontend/src/components/AnnotationCanvas.jsx b/frontend/src/components/AnnotationCanvas.jsx index 3c0c4c6..8963fe5 100644 --- a/frontend/src/components/AnnotationCanvas.jsx +++ b/frontend/src/components/AnnotationCanvas.jsx @@ -75,6 +75,17 @@ export default function AnnotationCanvas({ const additive = useRef(false) + function handleDoubleClick(event) { + if (selecting) return + const [x, y] = pointAt(event) + if (assistMode) { + const seed = [Math.max(0, x - 0.015), Math.max(0, y - 0.015), Math.min(1, x + 0.015), Math.min(1, y + 0.015)] + onAssist(seed) + } else if (onExemplar) { + onExemplar([x, y], !event.shiftKey, { kind: 'point' }) + } + } + function startDraw(event) { if (event.button !== 0) return const [x, y] = pointAt(event) @@ -218,6 +229,7 @@ export default function AnnotationCanvas({ onPointerDown={startDraw} onPointerMove={onPointerMove} onPointerUp={onPointerUp} + onDoubleClick={handleDoubleClick} > {/* A preview replaces this class on the frame wholesale, so its stored shapes step aside for the proposals — left on screen they read as @@ -270,7 +282,7 @@ export default function AnnotationCanvas({ ) })} - {!selecting && exemplars.map((item, i) => { + {!selecting && exemplars.filter((item) => item.kind !== 'point' && item.box).map((item, i) => { const [x0, y0, x1, y1] = item.box return ( Drag
Example of this class → preview a re-detect
+
+
Dbl Click
+
Quick exemplar point → preview a re-detect
+
Enter Esc
Apply / discard the preview
@@ -15,6 +19,10 @@ export default function ShortcutsPanel() {
Shift+Drag
Not this → drop it and re-detect
+
+
Shift+Dbl
+
Quick negative exemplar point
+
V
Draw ⇄ Select mode
diff --git a/frontend/src/hooks/useExemplarPool.js b/frontend/src/hooks/useExemplarPool.js index a69a861..5630af8 100644 --- a/frontend/src/hooks/useExemplarPool.js +++ b/frontend/src/hooks/useExemplarPool.js @@ -100,9 +100,12 @@ export default function useExemplarPool({ frameId, classId, onApplied, onError, timer.current = setTimeout(() => runRef.current?.(), delay) }, []) - const add = useCallback((box, positive) => { + const add = useCallback((target, positive, { kind = 'box' } = {}) => { if (!frameId) return - poolRef.current = [...poolRef.current, { box, positive }] + const entry = kind === 'point' + ? { point: target, positive, kind: 'point' } + : { box: target, positive, kind: 'box' } + poolRef.current = [...poolRef.current, entry] setExemplars(poolRef.current) setActive(true) // A drag during a pass cannot be in that pass's result, so it always earns diff --git a/frontend/src/pages/ReviewPage.jsx b/frontend/src/pages/ReviewPage.jsx index 195993c..0adc448 100644 --- a/frontend/src/pages/ReviewPage.jsx +++ b/frontend/src/pages/ReviewPage.jsx @@ -500,7 +500,7 @@ export default function ReviewPage({ batchId: rawBatchId, projectId, onProject } ? `${pool.exemplars.filter((e) => e.positive).length} example(s), ` + `${pool.exemplars.filter((e) => !e.positive).length} negative — ` + 'tune the filters, then Apply' - : 'Drag an example of this class · Shift-drag = not this'} + : 'Drag or double-click an example · Shift-drag/click = not this'} )}
diff --git a/install_nvidia.sh b/install_nvidia.sh old mode 100644 new mode 100755 diff --git a/scripts/capture_all_screenshots.py b/scripts/capture_all_screenshots.py old mode 100755 new mode 100644 diff --git a/scripts/restart_app.sh b/scripts/restart_app.sh old mode 100644 new mode 100755