dashboard-cpsp/docs/superpowers/specs/2026-09-04-karung-history-backfill-design.md

# Karung Sync History Backfill + Missing-as-Zero Display

**Date:** 2026-09-04  
**Status:** Approved (awaiting written-spec confirmation)  
**Surface:** Hitung Karung sync + summary cards

## Problem

1. **Sync only writes “today.”** `request_karung` / Sync hari ini / cron use karung-web-admin `/api/v1/cycles/active/sections` with `section=today`. Older cycle days (e.g. day 0 = cycle start) never get IoT rows even when `/api/combined` already has history for those dates.
2. **Symptom:** Rincian harian shows IOT `0` and Total `-` for dates with manual data but no `FeedSacks` row (e.g. 2 Sep), while upstream has real in/use/out.
3. **Frontend:** Summary cards showed `N/A` when no same-date manual; product preference is to show `0` for missing values.

## Goals

- On each Sync (UI button and cron), also upsert IoT feed-sack days for the active cycle from `/api/combined` history.
- If upstream values change after a previous sync, the next Sync updates local `FeedSacks` for those dates.
- Zero is a valid IoT value; do not treat `0/0/0` as “needs special retry.”
- One backfill pass per Sync invocation (no retry loops).
- On Hitung Karung summary cards, missing Manual/IOT values display as `0` (not `N/A`).

## Non-goals

- Changing the 17:00 (or configured) dashboard publish cutoff.
- Re-introducing cross-day manual fallback on summary cards (already removed).
- Syncing closed cycles unless already in scope of the existing cron filter.
- Building a new karung-web-admin API; use existing `/api/combined` + sections.

## Upstream data (confirmed)

`GET /api/combined` returns:

- `masuk.history[]`: `{ date, source, counter_value }` for `K{n} - In` and `K{n} - Out`
- `tuang.history[]`: `{ date, source, counter_value }` for `K{n} - Use`
- Live `masuk.today` / `tuang.today` (not required for backfill of past days)

`GET /api/v1/cycles/active/sections` still used for **today** (and optionally yesterday_history remains available but is insufficient alone for older cycle days).

## Design

### Sync flow (single invocation)

For each active cycle (cron) or the requested cycle (UI):

1. **Today (existing):** Call `request_karung(cycle, section="today")` as today — update/create the snapshot date returned by the sections block.
2. **History backfill (new):** Fetch `/api/combined` once. For each calendar date `D` where `cycle.start_date <= D <= min(today, cycle.end_date)` **and** combined history has at least one row for this kandang’s sources on `D`:
   - Resolve `in_today`, `out_today`, `feed_use_today` from history (`K{index} - In|Out|Use`). Missing source for that date → `0`.
   - `update_or_create` `FeedSacks(cycle, date=D)` with those daily values.
3. **Totals recompute:** After today + backfill writes, recompute running `in_total`, `out_total`, `feed_use_total` for **all** `FeedSacks` of that cycle ordered by `date`, `pk` (same accumulation rule as current single-day prior-row logic).
4. **Return:** API response should include the today row (compat) plus a short summary of dates upserted from history (for debugging / UI notice optional).

### When to write / skip a date

| Situation                                                            | Behavior                                                               |
| -------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| Date in cycle range and present in combined history for this kandang | Upsert daily values from history (refresh on every Sync)               |
| Date in cycle range but **no** history rows for this kandang         | Do **not** create a synthetic all-zero row; leave missing              |
| Existing local row, upstream later changes                           | Next Sync overwrites daily fields from history, then recomputes totals |
| Upstream reports `0`                                                 | Store `0` (valid)                                                      |

Rationale: missing upstream ≠ invent zeros in DB (keeps Total `-` until data exists). Frontend still shows `0` for missing display cells.

### Frontend (Hitung Karung)

- Summary cards (`FeedStats`): when Manual or IOT daily/total value is `null`/missing for the display date, show **`0`**, not `N/A`.
- Keep same-date manual pairing (no fallback to an older manual day).
- Daily table already uses `0` for missing Manual/IOT; optional consistency: Total `-` may remain `-` when no IoT row (indicates no synced row) — **prefer keep `-` for Total when no row** so operators can see “not synced yet” vs synced zero. Daily Manual/IOT columns stay `0`.

### API / callers

- `POST /api/v1/karungs/request/` continues to accept `section` for today path; after today sync, always run history backfill for that cycle (section choice does not disable backfill).
- Cron `sync_karung_from_web_admin`: same combined flow per active cycle.
- Prefer a single service entrypoint e.g. `sync_karung_for_cycle(cycle, *, section="today")` used by view + command.

## Testing

- Unit: parse combined history → per-date in/use/out for a kandang; skip dates with no history; upsert updates existing; totals recomputed across days.
- Unit/API: request sync creates missing cycle-start day from history while refreshing today.
- Frontend: FeedStats / page test expects `0` instead of `N/A` when latest IoT day has no matching manual.

## Risks

- Combined payload size / timeout: one fetch per cycle sync (acceptable at current history sizes); reuse one client call per cycle.
- Double-count totals if recompute is wrong: always full recompute from daily fields in date order, do not add “prior + today” on top of stale totals.
- Timezone: use Asia/Jakarta “today” consistently with existing dashboard settings.

## Out of scope follow-ups

- Backfilling closed cycles.
- Exposing a separate “Sync history only” button.