dashboard-cpsp/docs/superpowers/specs/2026-09-04-karung-history-backfill-design.md

Karung Sync History Backfill + Missing-as-Zero Display

Date: 2026-09-04
Status: Approved (awaiting written-spec confirmation)
Surface: Hitung Karung sync + summary cards

Problem

1. Sync only writes “today.” request_karung / Sync hari ini / cron use karung-web-admin /api/v1/cycles/active/sections with section=today. Older cycle days (e.g. day 0 = cycle start) never get IoT rows even when /api/combined already has history for those dates.
2. Symptom: Rincian harian shows IOT 0 and Total - for dates with manual data but no FeedSacks row (e.g. 2 Sep), while upstream has real in/use/out.
3. Frontend: Summary cards showed N/A when no same-date manual; product preference is to show 0 for missing values.

Goals

• On each Sync (UI button and cron), also upsert IoT feed-sack days for the active cycle from /api/combined history.
• If upstream values change after a previous sync, the next Sync updates local FeedSacks for those dates.
• Zero is a valid IoT value; do not treat 0/0/0 as “needs special retry.”
• One backfill pass per Sync invocation (no retry loops).
• On Hitung Karung summary cards, missing Manual/IOT values display as 0 (not N/A).

Non-goals

• Changing the 17:00 (or configured) dashboard publish cutoff.
• Re-introducing cross-day manual fallback on summary cards (already removed).
• Syncing closed cycles unless already in scope of the existing cron filter.
• Building a new karung-web-admin API; use existing /api/combined + sections.

Upstream data (confirmed)

GET /api/combined returns:

• masuk.history[]: { date, source, counter_value } for K{n} - In and K{n} - Out
• tuang.history[]: { date, source, counter_value } for K{n} - Use
• Live masuk.today / tuang.today (not required for backfill of past days)

GET /api/v1/cycles/active/sections still used for today (and optionally yesterday_history remains available but is insufficient alone for older cycle days).

Design

Sync flow (single invocation)

For each active cycle (cron) or the requested cycle (UI):

1. Today (existing): Call request_karung(cycle, section="today") as today — update/create the snapshot date returned by the sections block.
2. History backfill (new): Fetch /api/combined once. For each calendar date D where cycle.start_date <= D <= min(today, cycle.end_date) and combined history has at least one row for this kandang’s sources on D:
   • Resolve in_today, out_today, feed_use_today from history (K{index} - In|Out|Use). Missing source for that date → 0.
   • update_or_create FeedSacks(cycle, date=D) with those daily values.
3. Totals recompute: After today + backfill writes, recompute running in_total, out_total, feed_use_total for all FeedSacks of that cycle ordered by date, pk (same accumulation rule as current single-day prior-row logic).
4. Return: API response should include the today row (compat) plus a short summary of dates upserted from history (for debugging / UI notice optional).

When to write / skip a date

Situation	Behavior
Date in cycle range and present in combined history for this kandang	Upsert daily values from history (refresh on every Sync)
Date in cycle range but no history rows for this kandang	Do not create a synthetic all-zero row; leave missing
Existing local row, upstream later changes	Next Sync overwrites daily fields from history, then recomputes totals
Upstream reports 0	Store 0 (valid)

Rationale: missing upstream ≠ invent zeros in DB (keeps Total - until data exists). Frontend still shows 0 for missing display cells.

Frontend (Hitung Karung)

• Summary cards (FeedStats): when Manual or IOT daily/total value is null/missing for the display date, show 0, not N/A.
• Keep same-date manual pairing (no fallback to an older manual day).
• Daily table already uses 0 for missing Manual/IOT; optional consistency: Total - may remain - when no IoT row (indicates no synced row) — prefer keep - for Total when no row so operators can see “not synced yet” vs synced zero. Daily Manual/IOT columns stay 0.

API / callers

• POST /api/v1/karungs/request/ continues to accept section for today path; after today sync, always run history backfill for that cycle (section choice does not disable backfill).
• Cron sync_karung_from_web_admin: same combined flow per active cycle.
• Prefer a single service entrypoint e.g. sync_karung_for_cycle(cycle, *, section="today") used by view + command.

Testing

• Unit: parse combined history → per-date in/use/out for a kandang; skip dates with no history; upsert updates existing; totals recomputed across days.
• Unit/API: request sync creates missing cycle-start day from history while refreshing today.
• Frontend: FeedStats / page test expects 0 instead of N/A when latest IoT day has no matching manual.

Risks

• Combined payload size / timeout: one fetch per cycle sync (acceptable at current history sizes); reuse one client call per cycle.
• Double-count totals if recompute is wrong: always full recompute from daily fields in date order, do not add “prior + today” on top of stale totals.
• Timezone: use Asia/Jakarta “today” consistently with existing dashboard settings.

Out of scope follow-ups

• Backfilling closed cycles.
• Exposing a separate “Sync history only” button.