diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..f5bf6cd --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,74 @@ +# Changelog + +All notable changes to this project are documented here. Format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) (dated entries; no +version tags are cut in this repo — POC stage, `main` is the release line). + +## [Unreleased] + +Nothing pending. + +## 2026-09-17 — Unified `config.yaml` with extensible model presets + +### Added +- `config.yaml` — single canonical config (stream, models, counting knobs, + batch, output paths, camera). Secrets/deployment-only values stay in `.env`. +- `src/config_loader.py` — stdlib-dataclasses + pyyaml loader, zero new + dependencies. Missing file falls back to `.env` + legacy defaults with a + `UserWarning`; `MODEL_MODE` env still honoured once with a `DeprecationWarning`. +- `tests/test_config_loader.py` — 8 smoke tests (load/validate, precedence, + atomic mode-switch round-trip, YAML-only mode-E extension, legacy warnings). +- Per-class `iou` and `min_bbox_area` in `models.detection_params` + (shared across modes; `iou` default `0.7` = Ultralytics default, no behaviour change). +- `--config` CLI flag on `predict.py` (default: `config.yaml` next to `predict.py`). + +### Changed +- **Model modes are data**: `config.yaml → models.modes` holds engines + (path key + contributed classes) and class filters only. `predict.py` + derives tracker roles structurally — adding mode E/F/... needs no code change, + and the dashboard `/api/model-modes` lists new modes automatically. +- Dashboard `POST /api/batch/mode {"model_mode": "X"}` validates against + `config.yaml` and persists atomically (tmp+replace, comments preserved) to + `models.active_mode`. Manual `karung-counter` restart still required. +- `batch_mode.json` keeps only the manual/auto batch `mode`; its legacy + `model_mode` key is ignored (warned when it disagrees). +- `zones.json` keeps geometry only (polygons + left/right limits); legacy knob + keys there are ignored with a warning — `config.yaml counting.*` is canonical. +- `deploy_to_jetson.py` also syncs `config.yaml`. +- `src/tracking.py` / `src/detection.py` accept an `iou` parameter + (default `0.7`, v3 callers unaffected). + +## 2026-09-15 — Default model mode B → C + +- Production default is now **Mode C** (combined v4 sack+truck + yolo11n + box-only on a dedicated tracker) instead of Mode B (shared sack+box tracker). + Changed in `predict.py`, `counter_dashboard.py`, `.env.example`, and docs. + Sack path uses the proven v4 model; sack/box track-ID spaces no longer collide. + +## 2026-09-14 — `models/modelREADME.md` rename + +- `models/README.md` → `models/modelREADME.md` to avoid confusion with the + root README. No content change. + +## 2026-09-11 — Model weights tracked, per-mode matrix + +- `models/*.pt` + `*.onnx` are now git-tracked (clone-ready); `*.engine` + stays gitignored (rebuildable via `export_model.py` on the Jetson). +- All weights moved under `models/` with a per-mode detector/filter matrix + (`models/modelREADME.md`); `predict.py` paths, `deploy_to_jetson.py`, and + docs updated. +- Multi-model modes (A/B/C/D, default B at the time) + box counting + + manual-mode banner (`6e4af50`). + +## 2026-09-10 — Single `predict.py` entrypoint, experiments archived + +- Production + dev CLI unified in `predict.py` (runs as `karung-counter.service` + with zero args). Retired experiments moved to `archive/` via `git mv` + (`predict_new.py`, `rpo_iki/`, `simple_predict.py`, check/merge/test scripts). +- Docs guides added (`docs/architecture.md`, `models.md`, `configuration.md`, + `scripts.md`) plus DB/batch helper scripts; pytest smoke tests + CI. + +## 2026-08-24 — Manual & auto batch modes, dual-port dashboards + +- Manual/auto batch counting, operator (port 5000) + monitoring (port 5721) + dashboards, historical batch data views and corrections. diff --git a/README.md b/README.md index 64ad228..d6f5425 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ persists results to SQLite/CSV, and serves a live Flask dashboard. > (`karung` = sack/bag, `truk` = truck, `muat` = load). Class names inside the YOLO > models are English (`sack`, `box`, `truck`, `person`). -## Pipeline (v3, `src/`) +## Pipeline (production, `predict.py`) ``` Frame → TruckDetect → ROI → SackTrack → Stabilize → Count → Dashboard / DB @@ -39,16 +39,17 @@ cp .env.example .env # then edit RTSP_URL / paths # Dev / offline (any flags omitted = production defaults): python predict.py --source path/to/video.mp4 --env .env python predict.py --source vid.mp4 --no-dashboard --no-db --output-dir /tmp/out --max-frames 500 -python predict.py --help # all flags: --model/--output-json/--sack-conf/--truck-conf/--box-conf/--batch-timeout +python predict.py --help # all flags: --config/--model/--model-mode/--output-json/--sack-conf/--truck-conf/--box-conf/--box-model/--batch-timeout # Production (Jetson systemd, zero flags): sudo systemctl restart karung-counter karung-counter-dashboard ``` Production on the Jetson runs `predict.py` + `counter_dashboard.py` as systemd services -(see [`docs/deployment.md`](docs/deployment.md)) — the `src/` package is the clean -re-implementation; `predict.py` / `predict_new.py` are the deployed legacy pipelines that -add SQLite persistence, live-frame publishing to `/dev/shm`, and zone polygons. +(see [`docs/deployment.md`](docs/deployment.md)) — `predict.py` is the production +pipeline (SQLite persistence, live-frame publishing to `/dev/shm`, zone polygons); +the `src/` package is the shared detection/tracking/counting library it builds on +(the old v3 `src/main.py` loop is deprecated). Retired experiments live in `archive/`. ## Models @@ -72,10 +73,10 @@ support matrix. | File | Purpose | |---|---| -| `.env` (see `.env.example`) | DB paths, camera name, object label, cutoff time, ports, RTSP URL | -| `zones.json` | Calibrated pallet / truck / counting polygons + tuning knobs | +| `config.yaml` | Canonical config: stream, model modes/presets, counting knobs, batch, output paths, camera | +| `.env` (see `.env.example`) | Secrets + deployment only: RTSP URL, dashboard host/ports/secret/site | +| `zones.json` | Calibrated pallet / truck / counting polygons + left/right limits (geometry only) | | `cfg/tracker.yaml` | FastTrack/ByteTrack occlusion tuning (buffer 60, re-ID windows) | -| `rpo_iki/configs/` | Alternate counting engine configs (areas, params, model registry, cameras) | See [`docs/configuration.md`](docs/configuration.md) for every variable. @@ -84,23 +85,27 @@ See [`docs/configuration.md`](docs/configuration.md) for every variable. Pages in `templates/`: `monitoring.html`, `operator.html` (manual batch start/stop), `history.html`, `analytics.html`. JSON APIs under `/api/*` (live video MJPEG, current/ previous batch, summary, daily data, CSV/Excel export). Data source: -`jetson_counter.db` + `current_batch.json` (+ `batch_mode.json`). +`jetson_counter.db` + `current_batch.json` (+ `batch_mode.json` for the +manual/auto batch switch; the model mode lives in `config.yaml`). ## Repository layout ``` src/ Shared library (streaming, detection, tracking, stabilizer, - truck_roi, counting, batch, dashboard, logger, config, main) + truck_roi, counting, batch, dashboard, logger, config_loader, main) +config.yaml Canonical config (stream/models/counting/batch/output/camera) cfg/tracker.yaml Tracker tuning predict.py Single entrypoint: production service + dev CLI (see --help) counter_dashboard.py Flask dashboard + APIs | templates/*.html pages +tests/ Pytest smoke tests (counting, batch, config, config_loader) archive/ Retired experiments (predict_new.py, rpo_iki/, simple_predict.py, check/merge/test scripts) — history preserved, not imported export_model.py Export .pt → TensorRT .engine (FP16) | export_v4.py variant deploy_to_jetson.py Paramiko sync + service restart *.service systemd units (counter, dashboard, mediamtx) -zones.json Calibrated zones +zones.json Calibrated zone geometry batch_history_folder/ Per-batch JSON reports +CHANGELOG.md Release history (dated entries, Keep-a-Changelog style) ``` Script-by-script reference: [`docs/scripts.md`](docs/scripts.md). @@ -112,3 +117,4 @@ Script-by-script reference: [`docs/scripts.md`](docs/scripts.md). - [`docs/configuration.md`](docs/configuration.md) — all config files/variables - [`docs/deployment.md`](docs/deployment.md) — Jetson services, TensorRT, deploy flow - [`docs/scripts.md`](docs/scripts.md) — entry points & utility scripts +- [`CHANGELOG.md`](CHANGELOG.md) — release history diff --git a/docs/architecture.md b/docs/architecture.md index 8268702..dcc651d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -104,10 +104,12 @@ stats panel (Loading / Unloading / Net / last-3-batch history), and a bottom sta (IDLE / stabilizing + progress / counting + duration-idle / waiting). ### Config & logging -- `src/config.py` — frozen `Config` dataclass loaded from `.env` (`load_config`). - Note: its variable names (`LOCAL_RTSP`, `MODEL_SACK_PATH`, …) differ from the - production `.env` keys (`RTSP_URL`, …) used by `predict.py` — see - [`configuration.md`](configuration.md). +- `src/config_loader.py` — canonical loader: `config.yaml` (stream, models, + counting, batch, output, camera) + `.env` (secrets/deployment only) + + `zones.json` polygons + `cfg/tracker.yaml` reference. Model modes are data + (`models.modes`); missing file falls back to `.env` + legacy defaults. +- `src/config.py` — **deprecated** frozen v3 `Config` (different key names); + do not add keys here. - `src/logger.py` — `CSVLogger` appends `batch_summary.csv` (`batch_id,start,end,duration,loading,unloading,net`) and `sack_events.csv` (`timestamp,batch_id,sack_crossed,T-,direction`). @@ -117,8 +119,9 @@ stats panel (Loading / Unloading / Net / last-3-batch history), and a bottom sta Same stage order as v3 but implemented with shapely polygons (`zones.json`: pallet/truck/counting), visual-similarity + Re-ID registries, static-frame debounce, SQLite persistence, and live-frame publishing to `/dev/shm` for the dashboard. Uses the -repo `src/` modules (imports at `predict.py`). Zero CLI flags = systemd behaviour; -dev flags (`--source`, `--no-dashboard`, …) are additive overrides — see +repo `src/` modules (imports at `predict.py`). Zero CLI flags = systemd behaviour +(`config.yaml` + `.env`); dev flags (`--source`, `--config`, `--model-mode`, +`--no-dashboard`, …) are additive overrides — see `predict.py --help` and `docs/scripts.md`. (`init_db()` and the `http.server` imports are dead code — persistence goes through `finalize_batch`/`save_active_batch_state`, dashboard integration is file-based.) diff --git a/docs/configuration.md b/docs/configuration.md index 9916a6b..daef3c6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -27,18 +27,20 @@ to `models.active_mode`. **Manual `karung-counter` restart still required** batch `mode`; its legacy `model_mode` key is ignored (warned). `MODEL_MODE` env var still overrides for one run but is deprecated (warned). -Three layers: environment file → zone polygons → tracker/counter tuning. Template: `.env.example`. Production values live in `.env` (git-ignored). +Missing `config.yaml` falls back to `.env` + built-in defaults with a warning +(see `src/config_loader.py`); explicit legacy path env vars below still override +when set. ## 1. `.env` (secrets & deployment — `predict.py` / `counter_dashboard.py`) | Key | Default | Meaning | |---|---|---| -| `OUTPUT_DIR` | `/opt/jetson-counter` | Base dir for DB + state files | -| `DB_PATH` | `$OUTPUT_DIR/jetson_counter.db` | SQLite batches/daily summaries | -| `STATE_FILE` | `$OUTPUT_DIR/current_batch.json` | Live batch state (recovered on restart) | -| `BATCH_MODE_FILE` | `$OUTPUT_DIR/batch_mode.json` | Manual vs auto batch mode | -| `LIVE_STREAM_FRAME_PATH` | `/dev/shm/jetson-counter/live_frame.jpg` | Latest annotated frame (RAM disk, dashboard MJPEG reads this) | +| `OUTPUT_DIR` | `/opt/jetson-counter` | Legacy override of `output.dir` when set | +| `DB_PATH` | `$OUTPUT_DIR/jetson_counter.db` | Legacy override of the SQLite path when set | +| `STATE_FILE` | `$OUTPUT_DIR/current_batch.json` | Legacy override of the live batch state path when set | +| `BATCH_MODE_FILE` | `$OUTPUT_DIR/batch_mode.json` | Legacy override (file keeps only manual/auto batch mode now) | +| `LIVE_STREAM_FRAME_PATH` | `/dev/shm/jetson-counter/live_frame.jpg` | Legacy override of the annotated frame path when set | | `CAMERA_NAME` | `CC1` | Camera tag stored per batch | | `OBJECT_LABEL` | `karung-pakan` | Object tag stored per batch | | `DAILY_CUTOFF_TIME` | `06:00` | Counting-day boundary (`get_counting_date`) | @@ -53,7 +55,11 @@ Template: `.env.example`. Production values live in `.env` (git-ignored). On Windows dev machines these resolve to `d:/Belajar/menghitung karung/...`. -## 2. `src/config.py` keys (v3 `src/main.py --env`) +## 2. `src/config.py` keys (deprecated v3 `src/main.py --env`) + +⚠️ **Deprecated** — the v3 loop is retired; production uses `config.yaml` via +`src/config_loader.py`. Documented here only because the old keys still exist +in code. Do not add new keys here. ⚠️ **Different names** from the table above — the v3 loader uses its own keys: @@ -69,18 +75,20 @@ On Windows dev machines these resolve to `d:/Belajar/menghitung karung/...`. CLI: `python -m src.main --source video.mp4 --env .env`. -## 3. `zones.json` (calibrated polygons, 1920×1080 reference) +## 3. `zones.json` (calibrated geometry, 1920×1080 reference) -- `palet` / `truck` / `counting` — zone polygons (override the hardcoded defaults in - `predict_new.py:429-437`); scaled to actual resolution at startup. +- `palet` / `truck` / `counting` — zone polygons; scaled to actual resolution + at startup. - `left_limit` / `right_limit` (`0.27578` / `0.72578`) — counting X band. -- `duplicate_circle_radius` / `jarak_toleransi_duplikat` — spatial anti-double-count. -- `min_valid_area` (`15000`) — perspective-area floor for valid sacks. -- `max_reid_transit_distance` (`400`), `circle_stay_timeout_sec` (`10.0`). -- `inference_stride` (`2`), `confirm_delay_sec` (`0.5`), `exit_confirm_delay_sec` (`6.0`). - `external_stream_url` — MediaMTX restream endpoint. +- Legacy knob keys (`duplicate_circle_radius`, `min_valid_area`, + `jarak_toleransi_duplikat`, `max_reid_transit_distance`, + `circle_stay_timeout_sec`, `inference_stride`, `confirm_delay_sec`, + `exit_confirm_delay_sec`) are **ignored with a warning** — they moved to + `config.yaml counting.*` / `stream.inference_stride`. Keep only geometry here. -Recalibrate with `get_coordinates.py` / `get_calib_frame.py` (`calib_frame.jpg`). +Recalibrate with `archive/get_coordinates.py` / `archive/get_calib_frame.py` +(`calib_frame.jpg`). ## 4. `cfg/tracker.yaml` (FastTrack/ByteTrack tuning) @@ -90,7 +98,7 @@ Recalibrate with `get_coordinates.py` / `get_calib_frame.py` (`calib_frame.jpg`) `occ_cover_thresh=0.6`, Kalman offsets + `init_iou_suppress=0.65`. Falls back to stock `bytetrack.yaml` if the file is missing (`src/tracking.py:39`). -## 5. `rpo_iki/configs/` (alternate engine) +## 5. `archive/rpo_iki/configs/` (retired alternate engine, not imported) `area_truk*.json` (per-camera truck areas), `cameras.json` (cam1/cam2 RTSP), `counting_params.json` (+ `counting_params_last_truck.json`, hot-reloaded by diff --git a/docs/deployment.md b/docs/deployment.md index 9524c5e..0a3a982 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -21,11 +21,12 @@ systemctl status karung-counter karung-counter-dashboard --no-pager ## Deploy flow (`deploy_to_jetson.py`) Paramiko sync of `templates/{operator,monitoring,base}.html`, `counter_dashboard.py`, -`predict.py`, `.env` **plus `models/*.engine`** (v4-best, yolo11n-sack+box, best, -truck-detector, model_karung_truk) → `192.168.192.96:/home/jetson/karung/` -(creates remote `models/` if missing, skips missing local files), then restarts both -services and checks status + ports (5000/5721). Run from the dev machine. -`.pt`/`.onnx` stay local-only (dev/export). +`predict.py`, `config.yaml`, `.env` **plus `models/*.engine`** (v4-best, +yolo11n-sack+box, best, truck-detector, model_karung_truk) → +`192.168.192.96:/home/jetson/karung/` (creates remote `models/` if missing, +skips missing local files), then restarts both services and checks status + +ports (5000/5721). Run from the dev machine. `.pt`/`.onnx` stay local-only +(dev/export). ## TensorRT export On the Jetson (needs CUDA): `python3 export_model.py models/.pt` exports to @@ -35,17 +36,20 @@ loads `.engine` only — see `models/modelREADME.md` for which weights each mode ## Runtime data files - SQLite `jetson_counter.db`: `batches(counting_date, batch_number, camera_name, object_label, count, start/end_time)`, `daily_summaries(...)`. -- `current_batch.json` (crash recovery), `batch_mode.json` (manual/auto), +- `current_batch.json` (crash recovery), `batch_mode.json` (manual/auto batch + mode only — the model mode lives in `config.yaml`), `batch_history_folder/batch_.json` + `hasil_perhitungan.json` (per-batch reports). - Live frame: `/dev/shm/jetson-counter/live_frame.jpg` (written every 2nd frame, consumed by `/api/live-video` MJPEG). -- Helpers: `backup.py`, `dump_db.py`, `check_jetson_db.py`, `migrate_jetson_db.py`, - `merge_batches_*.py`, `update_batches.py`, `diagnose_truck_jetson.py`. +- Helpers: `check_jetson_db.py` (root); retired ops scripts in `archive/` + (`backup.py`, `dump_db.py`, `migrate_jetson_db.py`, `merge_batches_*.py`, + `update_batches.py`, `diagnose_truck_jetson.py`). ## Dashboard (`counter_dashboard.py`) Pages: `/` + `/monitoring`, `/operator` (manual start/stop, mode switch), `/history`, `/analytics`. Key APIs: `/api/live-video`, `/api/current-batch`, -`/api/previous-batch`, `/api/batch/{start,stop,mode}`, `/api/summary`, -`/api/daily-data`, `/api/day-detail/`, `/api/recent-batches`, -`/api/available-dates`, `/api/export-daily-csv`, `/api/export-day-csv/` -(CSV + Excel via openpyxl). +`/api/previous-batch`, `/api/batch/{start,stop,mode}`, `/api/model-modes` +(mode list is derived from `config.yaml`, so future modes appear automatically), +`/api/summary`, `/api/daily-data`, `/api/day-detail/`, +`/api/recent-batches`, `/api/available-dates`, `/api/export-daily-csv`, +`/api/export-day-csv/` (CSV + Excel via openpyxl). diff --git a/docs/models.md b/docs/models.md index 9651ad5..4743211 100644 --- a/docs/models.md +++ b/docs/models.md @@ -64,9 +64,11 @@ Two dedicated models run in parallel on the same frames: - `src/` pipeline: `MODEL_SACK_PATH` / `MODEL_TRUCK_PATH` env vars (defaults: `./models/best.engine` / `./models/truck-detector.engine`; see [`configuration.md`](configuration.md)) or `--source` for files. -- `predict.py`: `MODEL_PATH` env var, else auto-picks `models/v4-best.engine` > - `models/v4-best.pt` > `models/v4-best (1).pt` > `models/model_karung_truk.engine` > - `models/model_karung_truk.pt`; `--model-mode A|B|C|D` (or `MODEL_MODE` env) picks - the pipeline; `--box-model` overrides the yolo11n weights. +- `predict.py`: `config.yaml` `models.paths` is canonical (per-mode presets in + `models.modes`, per-class conf/iou/min_bbox in `models.detection_params`). + Legacy overrides still work: `MODEL_PATH` env or `--model` (single v4 file), + `--box-model` (yolo11n weights), `--model-mode` / deprecated `MODEL_MODE` env + for the preset, `--sack-conf` / `--truck-conf` / `--box-conf` for thresholds. - TensorRT: `python export_model.py models/.pt` (FP16 `.engine`) on the Jetson; - production loads `.engine` only. `deploy_to_jetson.py` syncs the `.engine` files. + production loads `.engine` only (missing `.engine` falls back to the `.pt` + sibling with a warning). `deploy_to_jetson.py` syncs `config.yaml` + the `.engine` files. diff --git a/docs/scripts.md b/docs/scripts.md index a54df66..7b9242d 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -4,9 +4,10 @@ - `predict.py` — **single entrypoint: production service (zero flags) + dev CLI.** Combined sack+truck model + repo `src/` modules, shapely zones, visual-similarity/Re-ID debounce, SQLite, live-frame publish. Run by - `karung-counter.service`. Flags: `--source/--env/--model/--output-dir/ - --output-json/--sack-conf/--truck-conf/--box-conf(placeholder)/ + `karung-counter.service`. Flags: `--source/--env/--config/--model/--model-mode/ + --output-dir/--output-json/--sack-conf/--truck-conf/--box-conf/--box-model/ --batch-timeout/--max-frames/--no-dashboard/--no-db` (see `predict.py --help`). + Config: `config.yaml` (canonical) + `.env` (secrets/deployment). - `src/main.py` — deprecated shim: prints a pointer to `predict.py`, still runs the old v3 loop for backward compat. - `archive/` — retired: `predict_new.py` (duo-model state machine), @@ -14,19 +15,23 @@ (minimal CPU demo), `ini.py`, check/merge/test scripts. Not imported anywhere. ## Calibration & data tools -- `get_coordinates.py` / `get_calib_frame.py` — zone calibration (`calib_frame.jpg`, - `zones.json`); `capture_frame_native.py` — grab native frames (`live_frame_native.png`). -- `extract_random_frames.py` — sample frames for labeling; `check_*.py` +- `archive/get_coordinates.py` / `archive/get_calib_frame.py` — zone calibration + (`calib_frame.jpg`, `zones.json`); `archive/capture_frame_native.py` — grab + native frames (`live_frame_native.png`). +- `extract_random_frames.py` — sample frames for labeling; `archive/check_*.py` (`check_20_23`, `check_20_aug`, `check_all_after`) — batch verification passes. -- `test_bytetrack.py`, `test_track_crash.py` — tracker experiments/crash repro. +- `archive/test_bytetrack.py`, `archive/test_track_crash.py` — tracker experiments/crash repro. ## DB / ops helpers -`counter_dashboard.py` (dashboard), `backup.py`, `dump_db.py`, `check_jetson_db.py`, -`migrate_jetson_db.py`, `merge_batches_20_23.py`, `merge_batches_28_35.py`, -`update_batches.py`, `diagnose_truck_jetson.py`, `deploy_to_jetson.py`, -`export_model.py` / `export_v4.py` (TensorRT export). +`counter_dashboard.py` (dashboard), `check_jetson_db.py`, +`archive/backup.py`, `archive/dump_db.py`, +`archive/migrate_jetson_db.py`, `archive/merge_batches_20_23.py`, +`archive/merge_batches_28_35.py`, `archive/update_batches.py`, +`archive/diagnose_truck_jetson.py`, `deploy_to_jetson.py`, +`export_model.py` / `archive/export_v4.py` (TensorRT export). ## Ignored at runtime (`.gitignore`) `.env`, `*.db`, state JSONs, `batch_history_folder/`, media (`*.mp4/*.jpg/*.png`), -model binaries (`*.pt/*.onnx/*.engine`), venvs, IDE files — so the large weights, -videos and local DBs in this working tree are intentionally untracked. +`*.engine` (rebuildable via `export_model.py`), venvs, IDE files — so the large +weights, videos and local DBs in this working tree are intentionally untracked. +(`models/*.pt`/`*.onnx` ARE tracked so clones are inference-ready.)