chicken-weight-estimator/README.md

# Weight Estimator

Offline experiment pipeline for estimating daily average broiler weight per cage
from cam2/cam3 top-down video and physical scale readings.

## Install

```bash
cd "/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken"
python -m pip install -e .

# Chicken counter (YOLO + BoT-SORT) — required for extract / QA / predict:
python -m pip install -e "/media/asus/One Touch/LABS/chicken-counting/chicken-sukawarna-existing"
# Optional Jetson extras:
python -m pip install -e ".[jetson]"
```

**Venv used on this machine:**

```bash
source /media/asus/DATA/wchicken/bin/activate
cd "/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken"
```

All commands below assume that venv is active and you are in `try-weight-chicken`.
Config default: `configs/weight_cc2_cc3.yaml`.

## Command guide

### CLI map

| Command | Purpose | GPU? |
|---------|---------|------|
| `audit` | CSV ↔ video path check | no |
| `calibration-audit` | Save calibration frames | no |
| `validate-calibration` | Homography error check | no |
| `extract` | Unique-track feature extract → `detections.parquet` | **yes** |
| `aggregate` | Detections → `cage_day_features.parquet` | no |
| `train` | Fit vision-primary model → `model_bundle.joblib` | no |
| `evaluate` | Late-holdout GO/NO_GO + plots | no |
| `filter-overlay` | JPG overlays or single QA MP4 | **yes** |
| `qa-review-videos` | Fixed 5-date × CC2/CC3 review MP4s | **yes** |
| `predict` | One-day average weight from video(s) | **yes** |
| `compare-filters` | Strict vs legacy filter report | no |
| `run-all` | Full pipeline (optional `--synthetic`) | maybe |

```bash
python -m weight_estimator.cli --help
python -m weight_estimator.cli <command> --help
# If entrypoint installed:
weight-estimator <command> --help
```

### 1. Audit / calibration

```bash
python -m weight_estimator.cli audit --config configs/weight_cc2_cc3.yaml
python -m weight_estimator.cli calibration-audit --config configs/weight_cc2_cc3.yaml
python -m weight_estimator.cli validate-calibration --config configs/weight_cc2_cc3.yaml
```

### 2. Full extract (unique chicken / track along video)

Extract runs **BoT-SORT every frame**, keeps **one best-confidence observation per `track_id`**,
target **400 unique tracks** per camera/day, single forward pass, stops on target **or** optical-flow reverse.

```bash
mkdir -p logs
# Backup previous detections
cp -a output/detections.parquet "output/detections_pre_extract_$(date +%Y%m%d_%H%M).parquet"

tmux kill-session -t weight-extract 2>/dev/null || true
STAMP=$(date +%Y%m%d_%H%M)
tmux new-session -d -s weight-extract \
  "cd '/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken' && \
   source /media/asus/DATA/wchicken/bin/activate && \
   PYTHONUNBUFFERED=1 python -m weight_estimator.cli extract \
     --config configs/weight_cc2_cc3.yaml \
     2>&1 | tee logs/extract_${STAMP}.log; \
   echo EXIT_CODE=\$? | tee -a logs/extract_${STAMP}.log; exec bash"

tail -f logs/extract_${STAMP}.log
# Done when you see: Extracted N detections -> .../detections.parquet  and EXIT_CODE=0
```

Useful log lines:

- `unique_track_mode track_every_n=1 … keep_best=True multipass=False target=400`
- `reverse stop at frame=N`
- `done valid=… unique=… replaced=… inferred=…`

Key YAML knobs (`sampling:` in `configs/weight_cc2_cc3.yaml`):

| Knob | Typical | Meaning |
|------|---------|---------|
| `track_every_n_frames` | `1` | YOLO/track every N frames |
| `per_track_cap` | `1` | Max stored obs per track_id |
| `keep_best_per_track` | `true` | Keep highest-confidence obs |
| `target_detections` / late | `400` | Unique-track target |
| `allow_stride_multipass` | `false` | No video re-open (keeps IDs) |
| `stop_on_backward` | `true` | Optical-flow reverse end |

### 3. Aggregate → train → evaluate (after extract)

```bash
python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml
python -m weight_estimator.cli train --config configs/weight_cc2_cc3.yaml --model ridge
python -m weight_estimator.cli evaluate --config configs/weight_cc2_cc3.yaml
```

- **GO** if late-holdout MAPE ≤ 15% vs consensus (`berat_aktual_g`).
- Report: `output/validation_report.json`
- Bundle: `output/model_bundle.joblib`

Re-apply size MAD only (no re-extract):

```bash
python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml --refilter-size
```

### 4. QA review videos (visual check of valid / unique OK)

Writes overlay MP4s for fixed dates: cycle days **4, 15, 21, 30, 47** × **CC2 + CC3** (10 files).

Behavior matches extract tracking:

- Track **every frame**
- HUD: `unique_OK=N/400` (cumulative unique accepted `track_id`s) + `frame_OK` / `rej`
- Boxes: `OK#track_id`
- Stops at **target 400** (same as extract) or reverse-stop if target not reached
- Encodes every `--frame-stride` frames (default 10)

```bash
mkdir -p logs output/qa_review_videos
STAMP=$(date +%Y%m%d_%H%M)
tmux kill-session -t weight-qa-videos 2>/dev/null || true
tmux new-session -d -s weight-qa-videos \
  "cd '/media/asus/One Touch/LABS/weight-estimator/try-weight-chicken' && \
   source /media/asus/DATA/wchicken/bin/activate && \
   PYTHONUNBUFFERED=1 python -m weight_estimator.cli qa-review-videos \
     --config configs/weight_cc2_cc3.yaml \
     --output-dir output/qa_review_videos \
     --frame-stride 10 \
     2>&1 | tee logs/qa_review_${STAMP}.log; \
   echo EXIT_CODE=\$? | tee -a logs/qa_review_${STAMP}.log; exec bash"

tail -f logs/qa_review_${STAMP}.log
```

Outputs:

- `output/qa_review_videos/{band}_day{N}_{date}_{CCx}.mp4`
- `output/qa_review_videos/qa_review_summary.json`

Single-video QA MP4 (same unique-OK / target logic):

```bash
python -m weight_estimator.cli filter-overlay \
  --config configs/weight_cc2_cc3.yaml \
  --video "/media/asus/One Touch/VIDEOS/cycle7/kandang-atas/2026-05-26/kandang_1_camera_2_2026-05-26_120057.mp4" \
  --camera-id CC2 \
  --cycle-day 4 \
  --frame-stride 10 \
  --write-video output/qa_review_videos/manual_day4_CC2.mp4
```

JPG frame overlays only (legacy tuning):

```bash
python -m weight_estimator.cli filter-overlay \
  --config configs/weight_cc2_cc3.yaml \
  --video /path/to/video.mp4 \
  --camera-id CC2 \
  --cycle-day 15 \
  --frame-stride 30 \
  --max-frames 50 \
  --output-dir output/filter_overlays
```

### 5. Predict (daily use)

Formula: **`DOC + max(0, vision_gain(features))`** (vision-primary). DOC is required.

```bash
python -m weight_estimator.cli predict \
  --config configs/weight_cc2_cc3.yaml \
  --doc-weight 32 \
  --doc-date 2026-05-22 \
  --date 2026-06-15 \
  --model output/model_bundle.joblib \
  --cc2 /path/to/kandang_1_camera_2_XXXX.mp4 \
  --cc3 /path/to/kandang_1_camera_3_XXXX.mp4 \
  --output-dir output/predict_2026-06-15
```

Writes `prediction.json` under `--output-dir`.

### 6. tmux helpers

```bash
tmux ls
tmux attach -t weight-extract      # or weight-qa-videos
# detach: Ctrl+b then d
tmux kill-session -t weight-extract
tmux kill-session -t weight-qa-videos
nvidia-smi
pgrep -af 'weight_estimator.cli'
```

### 7. Comparison Excel (Manual / IoT / Vision)

After evaluate, refresh comparison under `output/comparison_consensus/`
(e.g. `Perbandingan_Manual_IoT_Vision.xlsx`: columns IoT → Manual → Vision,
akurasi = 100% − |error|, green/yellow/red **cell background**).

Typical path after a new extract:

```bash
python -m weight_estimator.cli aggregate --config configs/weight_cc2_cc3.yaml
python -m weight_estimator.cli train --config configs/weight_cc2_cc3.yaml --model ridge
python -m weight_estimator.cli evaluate --config configs/weight_cc2_cc3.yaml
# then regenerate Perbandingan Excel from prediction_vs_manual_iot.csv (agent/script)
```

## Quick start (synthetic demo)

When videos are not available locally:

```bash
python -m weight_estimator.cli run-all --config configs/weight_cc2_cc3.yaml --synthetic
```

Outputs land in `output/`:

- `audit_report.json`
- `calibration_audit.json`
- `detections.parquet`
- `cage_day_features.parquet`
- `model_bundle.joblib`
- `validation_report.json`
- `bland_altman_*.png`

## Configuration

- [`configs/weight_cc2_cc3.yaml`](configs/weight_cc2_cc3.yaml) — data paths, sampling, quality filters
- [`configs/calibration/CC2_homography.yaml`](configs/calibration/CC2_homography.yaml) — CC2 pixel→cm points
- [`configs/calibration/CC3_homography.yaml`](configs/calibration/CC3_homography.yaml) — CC3 pixel→cm points
- Chicken batch / YOLO: [`cycle7_batch.yaml`](../../chicken-counting/chicken-sukawarna-existing/configs/cycle7_batch.yaml)

Labels (consensus Manual+IoT): `data/historical_weights_manual_iot_review.csv`

## Quality filters

Detections must pass **all** frame tiers before entering the weight sample:

1. **Class gate** — class 0 only; reject not-chicken / half-chicken
2. **Containment** — ROI overlap + sampling corridors
3. **Isolation** — max IoU and min centroid distance
4. **Shape** — confidence, aspect ratio, area vs frame median
5. **Frame gate** — clean_ratio and ≥ 1 accepted bird
6. **Size distribution (second filter)** — per camera-day MAD z-score on `area_cm2`

Rejection breakdown: `output/filter_report.json`.

**YOLO vs post-filter:** YOLO `conf` / `iou` live in `cycle7_batch.yaml`. Weight `quality.conf_min` must stay aligned with batch `early_cycle`. Re-extract after changing either.

```bash
python -m weight_estimator.cli compare-filters --config configs/weight_cc2_cc3.yaml
```

## Design notes

- **DOC-essential:** labels require Cycle ID, DOC Date, DOC Weight (g). Age = `(Tanggal − DOC Date).days`.
- **Vision-primary:** `predicted_avg_g = DOC + max(0, vision_gain(features))`. Age-only / Gompertz is **not** used for GO/NO_GO or comparison reports.
- **GO/NO_GO:** late-cycle holdout MAPE ≤ 15% vs consensus (`berat_aktual_g`).
- **Unit of analysis:** one row per `(cage, date)`; detections are aggregated, not individually labeled.
- **Unique-track sampling:** track every frame; `per_track_cap: 1` + `keep_best_per_track`; target **400** unique tracks; `stop_on_backward`; no multipass re-open. Diversity should be ~1.0. During extract, `extract_overlay_every_n_frames: 1000` saves frame-tier overlays to `output/extract_filter_overlays/{date}_{camera}/`.
- **Size distribution:** MAD z-score on `area_cm2`. Band `z_max`: early **3.0** (≤17), mid **2.5** (18–24), late **2.0** (≥25). Tune with `aggregate --refilter-size` when only size gates change.
- **Early / mid cycle:** `early_cycle.day_max` **17**, `mid_cycle` **18–24**, then late/strict. Batch early YOLO: `conf` 0.12.
- **Camera fusion:** gate cams below `fuse_min_detections`, weight by `detection_count`; else best cam. Re-aggregate after fusion changes (no re-extract).
- **Segmentation:** YOLO-seg via `detection.model_path`; mask geometry preferred with bbox fallback. Production features: `features.preset: mask_max`.
- **Train ages:** **3–35**. Evaluate late holdout **29–35** (fit on **3–28**).
- **Excluded days:** `exclude_cycle_days` (cycle7: 0–2). DOC weight examples: **32 g**.