first commit

2026-07-29 14:55:55 +07:00
commit c666715e29
@@ -0,0 +1,283 @@
+# Weight Estimator
+
+Offline experiment pipeline for estimating daily average broiler weight per cage
+from cam2/cam3 top-down video and physical scale readings.
+
+## Install
+
+```bash
+cd "/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken"
+python -m pip install -e .
+
+# Chicken counter (YOLO + BoT-SORT) — required for extract / QA / predict:
+python -m pip install -e "/media/asus/One Touch/LABS/chicken-counting/chicken-sukawarna-existing"
+# Optional Jetson extras:
+python -m pip install -e ".[jetson]"
+```
+
+**Venv used on this machine:**
+
+```bash
+source /media/asus/DATA/wchicken/bin/activate
+cd "/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken"
+```
+
+All commands below assume that venv is active and you are in `try-weight-chicken`.
+Config default: `configs/weight_cc2_cc3.yaml`.
+
+## Command guide
+
+### CLI map
+
+| Command | Purpose | GPU? |
+|---------|---------|------|
+| `audit` | CSV ↔ video path check | no |
+| `calibration-audit` | Save calibration frames | no |
+| `validate-calibration` | Homography error check | no |
+| `extract` | Unique-track feature extract → `detections.parquet` | **yes** |
+| `aggregate` | Detections → `cage_day_features.parquet` | no |
+| `train` | Fit vision-primary model → `model_bundle.joblib` | no |
+| `evaluate` | Late-holdout GO/NO_GO + plots | no |
+| `filter-overlay` | JPG overlays or single QA MP4 | **yes** |
+| `qa-review-videos` | Fixed 5-date × CC2/CC3 review MP4s | **yes** |
+| `predict` | One-day average weight from video(s) | **yes** |
+| `compare-filters` | Strict vs legacy filter report | no |
+| `run-all` | Full pipeline (optional `--synthetic`) | maybe |
+
+```bash
+python -m weight_estimator.cli --help
+python -m weight_estimator.cli <command> --help
+# If entrypoint installed:
+weight-estimator <command> --help
+```
+
+### 1. Audit / calibration
+
+```bash
+python -m weight_estimator.cli audit --config configs/weight_cc2_cc3.yaml
+python -m weight_estimator.cli calibration-audit --config configs/weight_cc2_cc3.yaml
+python -m weight_estimator.cli validate-calibration --config configs/weight_cc2_cc3.yaml
+```
+
+### 2. Full extract (unique chicken / track along video)
+
+Extract runs **BoT-SORT every frame**, keeps **one best-confidence observation per `track_id`**,
+target **400 unique tracks** per camera/day, single forward pass, stops on target **or** optical-flow reverse.
+
+```bash
+mkdir -p logs
+# Backup previous detections
+cp -a output/detections.parquet "output/detections_pre_extract_$(date +%Y%m%d_%H%M).parquet"
+
+tmux kill-session -t weight-extract 2>/dev/null || true
+STAMP=$(date +%Y%m%d_%H%M)
+tmux new-session -d -s weight-extract \
+  "cd '/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken' && \
+   source /media/asus/DATA/wchicken/bin/activate && \
+   PYTHONUNBUFFERED=1 python -m weight_estimator.cli extract \
+     --config configs/weight_cc2_cc3.yaml \
+     2>&1 | tee logs/extract_${STAMP}.log; \
+   echo EXIT_CODE=\$? | tee -a logs/extract_${STAMP}.log; exec bash"
+
+tail -f logs/extract_${STAMP}.log
+# Done when you see: Extracted N detections -> .../detections.parquet  and EXIT_CODE=0
+```
+
+Useful log lines:
+
+- `unique_track_mode track_every_n=1 … keep_best=True multipass=False target=400`
+- `reverse stop at frame=N`
+- `done valid=… unique=… replaced=… inferred=…`
+
+Key YAML knobs (`sampling:` in `configs/weight_cc2_cc3.yaml`):
+
+| Knob | Typical | Meaning |
+|------|---------|---------|
+| `track_every_n_frames` | `1` | YOLO/track every N frames |
+| `per_track_cap` | `1` | Max stored obs per track_id |
+| `keep_best_per_track` | `true` | Keep highest-confidence obs |
+| `target_detections` / late | `400` | Unique-track target |
+| `allow_stride_multipass` | `false` | No video re-open (keeps IDs) |
+| `stop_on_backward` | `true` | Optical-flow reverse end |
+
+### 3. Aggregate → train → evaluate (after extract)
+
+```bash
+python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml
+python -m weight_estimator.cli train --config configs/weight_cc2_cc3.yaml --model ridge
+python -m weight_estimator.cli evaluate --config configs/weight_cc2_cc3.yaml
+```
+
+- **GO** if late-holdout MAPE ≤ 15% vs consensus (`berat_aktual_g`).
+- Report: `output/validation_report.json`
+- Bundle: `output/model_bundle.joblib`
+
+Re-apply size MAD only (no re-extract):
+
+```bash
+python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml --refilter-size
+```
+
+### 4. QA review videos (visual check of valid / unique OK)
+
+Writes overlay MP4s for fixed dates: cycle days **4, 15, 21, 30, 47** × **CC2 + CC3** (10 files).
+
+Behavior matches extract tracking:
+
+- Track **every frame**
+- HUD: `unique_OK=N/400` (cumulative unique accepted `track_id`s) + `frame_OK` / `rej`
+- Boxes: `OK#track_id`
+- Stops at **target 400** (same as extract) or reverse-stop if target not reached
+- Encodes every `--frame-stride` frames (default 10)
+
+```bash
+mkdir -p logs output/qa_review_videos
+STAMP=$(date +%Y%m%d_%H%M)
+tmux kill-session -t weight-qa-videos 2>/dev/null || true
+tmux new-session -d -s weight-qa-videos \
+  "cd '/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken' && \
+   source /media/asus/DATA/wchicken/bin/activate && \
+   PYTHONUNBUFFERED=1 python -m weight_estimator.cli qa-review-videos \
+     --config configs/weight_cc2_cc3.yaml \
+     --output-dir output/qa_review_videos \
+     --frame-stride 10 \
+     2>&1 | tee logs/qa_review_${STAMP}.log; \
+   echo EXIT_CODE=\$? | tee -a logs/qa_review_${STAMP}.log; exec bash"
+
+tail -f logs/qa_review_${STAMP}.log
+```
+
+Outputs:
+
+- `output/qa_review_videos/{band}_day{N}_{date}_{CCx}.mp4`
+- `output/qa_review_videos/qa_review_summary.json`
+
+Single-video QA MP4 (same unique-OK / target logic):
+
+```bash
+python -m weight_estimator.cli filter-overlay \
+  --config configs/weight_cc2_cc3.yaml \
+  --video "/media/asus/One Touch/VIDEOS/cycle7/kandang-atas/2026-05-26/kandang_1_camera_2_2026-05-26_120057.mp4" \
+  --camera-id CC2 \
+  --cycle-day 4 \
+  --frame-stride 10 \
+  --write-video output/qa_review_videos/manual_day4_CC2.mp4
+```
+
+JPG frame overlays only (legacy tuning):
+
+```bash
+python -m weight_estimator.cli filter-overlay \
+  --config configs/weight_cc2_cc3.yaml \
+  --video /path/to/video.mp4 \
+  --camera-id CC2 \
+  --cycle-day 15 \
+  --frame-stride 30 \
+  --max-frames 50 \
+  --output-dir output/filter_overlays
+```
+
+### 5. Predict (daily use)
+
+Formula: **`DOC + max(0, vision_gain(features))`** (vision-primary). DOC is required.
+
+```bash
+python -m weight_estimator.cli predict \
+  --config configs/weight_cc2_cc3.yaml \
+  --doc-weight 32 \
+  --doc-date 2026-05-22 \
+  --date 2026-06-15 \
+  --model output/model_bundle.joblib \
+  --cc2 /path/to/kandang_1_camera_2_XXXX.mp4 \
+  --cc3 /path/to/kandang_1_camera_3_XXXX.mp4 \
+  --output-dir output/predict_2026-06-15
+```
+
+Writes `prediction.json` under `--output-dir`.
+
+### 6. tmux helpers
+
+```bash
+tmux ls
+tmux attach -t weight-extract      # or weight-qa-videos
+# detach: Ctrl+b then d
+tmux kill-session -t weight-extract
+tmux kill-session -t weight-qa-videos
+nvidia-smi
+pgrep -af 'weight_estimator.cli'
+```
+
+### 7. Comparison Excel (Manual / IoT / Vision)
+
+After evaluate, refresh comparison under `output/comparison_consensus/`
+(e.g. `Perbandingan_Manual_IoT_Vision.xlsx`: columns IoT → Manual → Vision,
+akurasi = 100% − |error|, green/yellow/red **cell background**).
+
+Typical path after a new extract:
+
+```bash
+python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml
+python -m weight_estimator.cli train --config configs/weight_cc2_cc3.yaml --model ridge
+python -m weight_estimator.cli evaluate --config configs/weight_cc2_cc3.yaml
+# then regenerate Perbandingan Excel from prediction_vs_manual_iot.csv (agent/script)
+```
+
+## Quick start (synthetic demo)
+
+When videos are not available locally:
+
+```bash
+python -m weight_estimator.cli run-all --config configs/weight_cc2_cc3.yaml --synthetic
+```
+
+Outputs land in `output/`:
+
+- `audit_report.json`
+- `calibration_audit.json`
+- `detections.parquet`
+- `cage_day_features.parquet`
+- `model_bundle.joblib`
+- `validation_report.json`
+- `bland_altman_*.png`
+
+## Configuration
+
+- [`configs/weight_cc2_cc3.yaml`](configs/weight_cc2_cc3.yaml) — data paths, sampling, quality filters
+- [`configs/calibration/CC2_homography.yaml`](configs/calibration/CC2_homography.yaml) — CC2 pixel→cm points
+- [`configs/calibration/CC3_homography.yaml`](configs/calibration/CC3_homography.yaml) — CC3 pixel→cm points
+- Chicken batch / YOLO: [`cycle7_batch.yaml`](../../chicken-counting/chicken-sukawarna-existing/configs/cycle7_batch.yaml)
+
+Labels (consensus Manual+IoT): `data/historical_weights_manual_iot_review.csv`
+
+## Quality filters
+
+Detections must pass **all** frame tiers before entering the weight sample:
+
+1. **Class gate** — class 0 only; reject not-chicken / half-chicken
+2. **Containment** — ROI overlap + sampling corridors
+3. **Isolation** — max IoU and min centroid distance
+4. **Shape** — confidence, aspect ratio, area vs frame median
+5. **Frame gate** — clean_ratio and ≥ 1 accepted bird
+6. **Size distribution (second filter)** — per camera-day MAD z-score on `area_cm2`
+
+Rejection breakdown: `output/filter_report.json`.
+
+**YOLO vs post-filter:** YOLO `conf` / `iou` live in `cycle7_batch.yaml`. Weight `quality.conf_min` must stay aligned with batch `early_cycle`. Re-extract after changing either.
+
+```bash
+python -m weight_estimator.cli compare-filters --config configs/weight_cc2_cc3.yaml
+```
+
+## Design notes
+
+- **DOC-essential:** labels require Cycle ID, DOC Date, DOC Weight (g). Age = `(Tanggal − DOC Date).days`.
+- **Vision-primary:** `predicted_avg_g = DOC + max(0, vision_gain(features))`. Age-only / Gompertz is **not** used for GO/NO_GO or comparison reports.
+- **GO/NO_GO:** late-cycle holdout MAPE ≤ 15% vs consensus (`berat_aktual_g`).
+- **Unit of analysis:** one row per `(cage, date)`; detections are aggregated, not individually labeled.
+- **Unique-track sampling:** track every frame; `per_track_cap: 1` + `keep_best_per_track`; target **400** unique tracks; `stop_on_backward`; no multipass re-open. Diversity should be ~1.0. During extract, `extract_overlay_every_n_frames: 1000` saves frame-tier overlays to `output/extract_filter_overlays/{date}_{camera}/`.
+- **Size distribution:** MAD z-score on `area_cm2`. Band `z_max`: early **3.0** (≤17), mid **2.5** (18–24), late **2.0** (≥25). Tune with `aggregate --refilter-size` when only size gates change.
+- **Early / mid cycle:** `early_cycle.day_max` **17**, `mid_cycle` **18–24**, then late/strict. Batch early YOLO: `conf` 0.12.
+- **Camera fusion:** gate cams below `fuse_min_detections`, weight by `detection_count`; else best cam. Re-aggregate after fusion changes (no re-extract).
+- **Segmentation:** YOLO-seg via `detection.model_path`; mask geometry preferred with bbox fallback. Production features: `features.preset: mask_max`.
+- **Train ages:** **3–35**. Evaluate late holdout **29–35** (fit on **3–28**).
+- **Excluded days:** `exclude_cycle_days` (cycle7: 0–2). DOC weight examples: **32 g**.