From c666715e29ef663c9a88ee290767ece91460cf2b Mon Sep 17 00:00:00 2001 From: zakariasaputra Date: Wed, 29 Jul 2026 14:55:55 +0700 Subject: [PATCH] first commit --- README.md | 283 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 283 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..3dcf328 --- /dev/null +++ b/README.md @@ -0,0 +1,283 @@ +# Weight Estimator + +Offline experiment pipeline for estimating daily average broiler weight per cage +from cam2/cam3 top-down video and physical scale readings. + +## Install + +```bash +cd "/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken" +python -m pip install -e . + +# Chicken counter (YOLO + BoT-SORT) — required for extract / QA / predict: +python -m pip install -e "/media/asus/One Touch/LABS/chicken-counting/chicken-sukawarna-existing" +# Optional Jetson extras: +python -m pip install -e ".[jetson]" +``` + +**Venv used on this machine:** + +```bash +source /media/asus/DATA/wchicken/bin/activate +cd "/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken" +``` + +All commands below assume that venv is active and you are in `try-weight-chicken`. +Config default: `configs/weight_cc2_cc3.yaml`. + +## Command guide + +### CLI map + +| Command | Purpose | GPU? | +|---------|---------|------| +| `audit` | CSV ↔ video path check | no | +| `calibration-audit` | Save calibration frames | no | +| `validate-calibration` | Homography error check | no | +| `extract` | Unique-track feature extract → `detections.parquet` | **yes** | +| `aggregate` | Detections → `cage_day_features.parquet` | no | +| `train` | Fit vision-primary model → `model_bundle.joblib` | no | +| `evaluate` | Late-holdout GO/NO_GO + plots | no | +| `filter-overlay` | JPG overlays or single QA MP4 | **yes** | +| `qa-review-videos` | Fixed 5-date × CC2/CC3 review MP4s | **yes** | +| `predict` | One-day average weight from video(s) | **yes** | +| `compare-filters` | Strict vs legacy filter report | no | +| `run-all` | Full pipeline (optional `--synthetic`) | maybe | + +```bash +python -m weight_estimator.cli --help +python -m weight_estimator.cli --help +# If entrypoint installed: +weight-estimator --help +``` + +### 1. Audit / calibration + +```bash +python -m weight_estimator.cli audit --config configs/weight_cc2_cc3.yaml +python -m weight_estimator.cli calibration-audit --config configs/weight_cc2_cc3.yaml +python -m weight_estimator.cli validate-calibration --config configs/weight_cc2_cc3.yaml +``` + +### 2. Full extract (unique chicken / track along video) + +Extract runs **BoT-SORT every frame**, keeps **one best-confidence observation per `track_id`**, +target **400 unique tracks** per camera/day, single forward pass, stops on target **or** optical-flow reverse. + +```bash +mkdir -p logs +# Backup previous detections +cp -a output/detections.parquet "output/detections_pre_extract_$(date +%Y%m%d_%H%M).parquet" + +tmux kill-session -t weight-extract 2>/dev/null || true +STAMP=$(date +%Y%m%d_%H%M) +tmux new-session -d -s weight-extract \ + "cd '/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken' && \ + source /media/asus/DATA/wchicken/bin/activate && \ + PYTHONUNBUFFERED=1 python -m weight_estimator.cli extract \ + --config configs/weight_cc2_cc3.yaml \ + 2>&1 | tee logs/extract_${STAMP}.log; \ + echo EXIT_CODE=\$? | tee -a logs/extract_${STAMP}.log; exec bash" + +tail -f logs/extract_${STAMP}.log +# Done when you see: Extracted N detections -> .../detections.parquet and EXIT_CODE=0 +``` + +Useful log lines: + +- `unique_track_mode track_every_n=1 … keep_best=True multipass=False target=400` +- `reverse stop at frame=N` +- `done valid=… unique=… replaced=… inferred=…` + +Key YAML knobs (`sampling:` in `configs/weight_cc2_cc3.yaml`): + +| Knob | Typical | Meaning | +|------|---------|---------| +| `track_every_n_frames` | `1` | YOLO/track every N frames | +| `per_track_cap` | `1` | Max stored obs per track_id | +| `keep_best_per_track` | `true` | Keep highest-confidence obs | +| `target_detections` / late | `400` | Unique-track target | +| `allow_stride_multipass` | `false` | No video re-open (keeps IDs) | +| `stop_on_backward` | `true` | Optical-flow reverse end | + +### 3. Aggregate → train → evaluate (after extract) + +```bash +python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml +python -m weight_estimator.cli train --config configs/weight_cc2_cc3.yaml --model ridge +python -m weight_estimator.cli evaluate --config configs/weight_cc2_cc3.yaml +``` + +- **GO** if late-holdout MAPE ≤ 15% vs consensus (`berat_aktual_g`). +- Report: `output/validation_report.json` +- Bundle: `output/model_bundle.joblib` + +Re-apply size MAD only (no re-extract): + +```bash +python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml --refilter-size +``` + +### 4. QA review videos (visual check of valid / unique OK) + +Writes overlay MP4s for fixed dates: cycle days **4, 15, 21, 30, 47** × **CC2 + CC3** (10 files). + +Behavior matches extract tracking: + +- Track **every frame** +- HUD: `unique_OK=N/400` (cumulative unique accepted `track_id`s) + `frame_OK` / `rej` +- Boxes: `OK#track_id` +- Stops at **target 400** (same as extract) or reverse-stop if target not reached +- Encodes every `--frame-stride` frames (default 10) + +```bash +mkdir -p logs output/qa_review_videos +STAMP=$(date +%Y%m%d_%H%M) +tmux kill-session -t weight-qa-videos 2>/dev/null || true +tmux new-session -d -s weight-qa-videos \ + "cd '/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken' && \ + source /media/asus/DATA/wchicken/bin/activate && \ + PYTHONUNBUFFERED=1 python -m weight_estimator.cli qa-review-videos \ + --config configs/weight_cc2_cc3.yaml \ + --output-dir output/qa_review_videos \ + --frame-stride 10 \ + 2>&1 | tee logs/qa_review_${STAMP}.log; \ + echo EXIT_CODE=\$? | tee -a logs/qa_review_${STAMP}.log; exec bash" + +tail -f logs/qa_review_${STAMP}.log +``` + +Outputs: + +- `output/qa_review_videos/{band}_day{N}_{date}_{CCx}.mp4` +- `output/qa_review_videos/qa_review_summary.json` + +Single-video QA MP4 (same unique-OK / target logic): + +```bash +python -m weight_estimator.cli filter-overlay \ + --config configs/weight_cc2_cc3.yaml \ + --video "/media/asus/One Touch/VIDEOS/cycle7/kandang-atas/2026-05-26/kandang_1_camera_2_2026-05-26_120057.mp4" \ + --camera-id CC2 \ + --cycle-day 4 \ + --frame-stride 10 \ + --write-video output/qa_review_videos/manual_day4_CC2.mp4 +``` + +JPG frame overlays only (legacy tuning): + +```bash +python -m weight_estimator.cli filter-overlay \ + --config configs/weight_cc2_cc3.yaml \ + --video /path/to/video.mp4 \ + --camera-id CC2 \ + --cycle-day 15 \ + --frame-stride 30 \ + --max-frames 50 \ + --output-dir output/filter_overlays +``` + +### 5. Predict (daily use) + +Formula: **`DOC + max(0, vision_gain(features))`** (vision-primary). DOC is required. + +```bash +python -m weight_estimator.cli predict \ + --config configs/weight_cc2_cc3.yaml \ + --doc-weight 32 \ + --doc-date 2026-05-22 \ + --date 2026-06-15 \ + --model output/model_bundle.joblib \ + --cc2 /path/to/kandang_1_camera_2_XXXX.mp4 \ + --cc3 /path/to/kandang_1_camera_3_XXXX.mp4 \ + --output-dir output/predict_2026-06-15 +``` + +Writes `prediction.json` under `--output-dir`. + +### 6. tmux helpers + +```bash +tmux ls +tmux attach -t weight-extract # or weight-qa-videos +# detach: Ctrl+b then d +tmux kill-session -t weight-extract +tmux kill-session -t weight-qa-videos +nvidia-smi +pgrep -af 'weight_estimator.cli' +``` + +### 7. Comparison Excel (Manual / IoT / Vision) + +After evaluate, refresh comparison under `output/comparison_consensus/` +(e.g. `Perbandingan_Manual_IoT_Vision.xlsx`: columns IoT → Manual → Vision, +akurasi = 100% − |error|, green/yellow/red **cell background**). + +Typical path after a new extract: + +```bash +python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml +python -m weight_estimator.cli train --config configs/weight_cc2_cc3.yaml --model ridge +python -m weight_estimator.cli evaluate --config configs/weight_cc2_cc3.yaml +# then regenerate Perbandingan Excel from prediction_vs_manual_iot.csv (agent/script) +``` + +## Quick start (synthetic demo) + +When videos are not available locally: + +```bash +python -m weight_estimator.cli run-all --config configs/weight_cc2_cc3.yaml --synthetic +``` + +Outputs land in `output/`: + +- `audit_report.json` +- `calibration_audit.json` +- `detections.parquet` +- `cage_day_features.parquet` +- `model_bundle.joblib` +- `validation_report.json` +- `bland_altman_*.png` + +## Configuration + +- [`configs/weight_cc2_cc3.yaml`](configs/weight_cc2_cc3.yaml) — data paths, sampling, quality filters +- [`configs/calibration/CC2_homography.yaml`](configs/calibration/CC2_homography.yaml) — CC2 pixel→cm points +- [`configs/calibration/CC3_homography.yaml`](configs/calibration/CC3_homography.yaml) — CC3 pixel→cm points +- Chicken batch / YOLO: [`cycle7_batch.yaml`](../../chicken-counting/chicken-sukawarna-existing/configs/cycle7_batch.yaml) + +Labels (consensus Manual+IoT): `data/historical_weights_manual_iot_review.csv` + +## Quality filters + +Detections must pass **all** frame tiers before entering the weight sample: + +1. **Class gate** — class 0 only; reject not-chicken / half-chicken +2. **Containment** — ROI overlap + sampling corridors +3. **Isolation** — max IoU and min centroid distance +4. **Shape** — confidence, aspect ratio, area vs frame median +5. **Frame gate** — clean_ratio and ≥ 1 accepted bird +6. **Size distribution (second filter)** — per camera-day MAD z-score on `area_cm2` + +Rejection breakdown: `output/filter_report.json`. + +**YOLO vs post-filter:** YOLO `conf` / `iou` live in `cycle7_batch.yaml`. Weight `quality.conf_min` must stay aligned with batch `early_cycle`. Re-extract after changing either. + +```bash +python -m weight_estimator.cli compare-filters --config configs/weight_cc2_cc3.yaml +``` + +## Design notes + +- **DOC-essential:** labels require Cycle ID, DOC Date, DOC Weight (g). Age = `(Tanggal − DOC Date).days`. +- **Vision-primary:** `predicted_avg_g = DOC + max(0, vision_gain(features))`. Age-only / Gompertz is **not** used for GO/NO_GO or comparison reports. +- **GO/NO_GO:** late-cycle holdout MAPE ≤ 15% vs consensus (`berat_aktual_g`). +- **Unit of analysis:** one row per `(cage, date)`; detections are aggregated, not individually labeled. +- **Unique-track sampling:** track every frame; `per_track_cap: 1` + `keep_best_per_track`; target **400** unique tracks; `stop_on_backward`; no multipass re-open. Diversity should be ~1.0. During extract, `extract_overlay_every_n_frames: 1000` saves frame-tier overlays to `output/extract_filter_overlays/{date}_{camera}/`. +- **Size distribution:** MAD z-score on `area_cm2`. Band `z_max`: early **3.0** (≤17), mid **2.5** (18–24), late **2.0** (≥25). Tune with `aggregate --refilter-size` when only size gates change. +- **Early / mid cycle:** `early_cycle.day_max` **17**, `mid_cycle` **18–24**, then late/strict. Batch early YOLO: `conf` 0.12. +- **Camera fusion:** gate cams below `fuse_min_detections`, weight by `detection_count`; else best cam. Re-aggregate after fusion changes (no re-extract). +- **Segmentation:** YOLO-seg via `detection.model_path`; mask geometry preferred with bbox fallback. Production features: `features.preset: mask_max`. +- **Train ages:** **3–35**. Evaluate late holdout **29–35** (fit on **3–28**). +- **Excluded days:** `exclude_cycle_days` (cycle7: 0–2). DOC weight examples: **32 g**.