feedmill-recounter/.opencode/plans/websocket-live-preview.md

Plan: 15+ FPS Live Preview via WebSocket

Goal

Achieve 15+ FPS at 640p live preview during video processing using WebSocket push instead of HTTP polling.

---

Current Architecture Analysis

Component	Current	Bottleneck
Frame capture	Every 10th frame (2.5 Hz at 25 FPS)	Too slow for 15 FPS target
Encoding	cv2.imencode('.jpg', viz) synchronous, full-res	Blocks pipeline
Transport	HTTP GET /api/jobs/<id>/frame every 2s	2s latency, wasted polls
Frontend	<img src> replacement	Flicker, no frame timing control

Key finding: JPEG encoding at 640p = 285-371 FPS (CPU). Not the bottleneck. The bottleneck is synchronous encoding + polling architecture.

---

Recommended Architecture: WebSocket with Threaded Encoder

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Pipeline       │     │  Encoder Thread │     │  Flask-SocketIO │
│  (detection)    │────▶│  (non-blocking) │────▶│  WebSocket      │
│  - YOLO detect  │     │  - Downscale    │     │  Server         │
│  - Track        │     │  - JPEG encode  │     │  - Broadcast    │
│  - Count        │     │  - Queue push   │     │    to clients   │
└─────────────────┘     └─────────────────┘     └─────────────────┘
         │                        │                        │
         ▼                        ▼                        ▼
   Full speed            15+ FPS @ 640p            Real-time push
   processing            to queue                   to browsers

Why This Design?

• Pipeline stays at full speed — detection/tracking never blocked by encoding
• Encoder thread — produces 15+ FPS at 640p, drops frames if queue full
• Flask-SocketIO — handles WebSocket connections, broadcasting, reconnection
• Eventlet/gevent — async I/O for many concurrent viewers

---

Implementation Plan

Phase 1: Dependencies & Server Setup

Add to pyproject.toml:

[project.optional-dependencies]
web = [
    "flask-socketio>=5.3.0",
    "eventlet>=0.33.0",  # or gevent
]

Run: pip install -e ".[web]"

Phase 2: Pipeline — Non-blocking Frame Producer

Modify src/pipeline.py:

• Add preview_queue: Queue[bytes] parameter to run_pipeline()
• In main loop: every N frames (configurable), downscale → encode → queue.put_nowait(jpeg_bytes)
• If queue full: drop frame (don't block)
• Remove frame_callback — replaced by queue

def run_pipeline(
    ...,
    preview_queue: "Queue[bytes] | None" = None,
    preview_every_n: int = 2,      # 25 FPS / 2 = 12.5 FPS → use 1 for 25 FPS
    preview_max_dim: int = 640,
    preview_jpeg_quality: int = 70,
) -> PipelineResult:
    if preview_queue is not None:
        preview_interval = max(1, preview_every_n)
    ...
    if preview_queue and frame_idx % preview_interval == 0:
        # Downscale
        h, w = viz.shape[:2]
        if max(h, w) > preview_max_dim:
            scale = preview_max_dim / max(h, w)
            preview_viz = cv2.resize(viz, (int(w*scale), int(h*scale)))
        else:
            preview_viz = viz
        ok, jpeg = cv2.imencode('.jpg', preview_viz, [cv2.IMWRITE_JPEG_QUALITY, preview_jpeg_quality])
        if ok:
            try:
                preview_queue.put_nowait(jpeg.tobytes())
            except queue.Full:
                pass  # Drop frame, don't block

Phase 3: Job Queue — WebSocket Broadcaster

Modify src/job.py:

• Add preview_queue per job
• Start encoder thread that reads queue → broadcasts via SocketIO
• Thread lifecycle tied to job

import queue
import threading

class Job:
    ...
    preview_queue: "queue.Queue[bytes]" = field(default_factory=lambda: queue.Queue(maxsize=30))
    _encoder_thread: threading.Thread | None = None
    _encoder_stop: threading.Event = field(default_factory=threading.Event)

def _start_preview_broadcaster(self, job_id: str, socketio):
    """Thread: read preview queue, emit via SocketIO."""
    while not self._encoder_stop.is_set():
        try:
            jpeg_bytes = self.preview_queue.get(timeout=0.1)
            # Emit to job's room
            socketio.emit('preview_frame', {'frame': base64.b64encode(jpeg_bytes).decode()}, room=job_id)
        except queue.Empty:
            continue

Phase 4: Flask-SocketIO Server

Modify app.py:

from flask_socketio import SocketIO, emit, join_room, leave_room

socketio = SocketIO(app, cors_allowed_origins="*", async_mode='eventlet')

@socketio.on('connect')
def on_connect():
    pass

@socketio.on('join_job')
def on_join_job(data):
    job_id = data.get('job_id')
    if job_id:
        join_room(job_id)

@socketio.on('leave_job')
def on_leave_job(data):
    job_id = data.get('job_id')
    if job_id:
        leave_room(job_id)

# In _run_job():
# socketio.start_background_task(_start_preview_broadcaster, job_id, socketio)

Run with eventlet:

# main.py or app.py
if __name__ == '__main__':
    socketio.run(app, host='0.0.0.0', port=9000, debug=False)

Phase 5: Frontend — WebSocket Consumer

Replace polling in static/app.js:

// In initStatusPage():
var socket = io();
socket.on('connect', function() {
    socket.emit('join_job', {job_id: jobId});
});

socket.on('preview_frame', function(data) {
    if (livePreviewImg && data.frame) {
        livePreviewImg.src = 'data:image/jpeg;base64,' + data.frame;
        if (previewPlaceholder) previewPlaceholder.style.display = 'none';
        livePreviewImg.style.display = 'block';
    }
});

// Cleanup on disconnect/page unload
socket.on('disconnect', function() {
    socket.emit('leave_job', {job_id: jobId});
});
window.addEventListener('beforeunload', function() {
    socket.emit('leave_job', {job_id: jobId});
});

Phase 6: Configuration & Tuning

Parameter	Default	Tuning Notes
preview_every_n	1	1 = every frame (25 FPS max), 2 = 12.5 FPS
preview_max_dim	640	480 for faster, 720 for quality
preview_jpeg_quality	70	50-80 tradeoff
queue.maxsize	30	~2 seconds buffer at 15 FPS

---

Expected Performance

Metric	Current	Target
Preview FPS	~0.5 (polling)	15-25
Latency	2+ seconds	<100ms
Pipeline impact	Blocks every 10th frame	Zero
Resolution	Full video res	640p max
Bandwidth/frame	~200-500 KB	~30-50 KB

---

Files to Modify

File	Changes
pyproject.toml	Add flask-socketio, eventlet to optional deps
src/pipeline.py	Add preview_queue param, producer logic
src/job.py	Add preview queue, encoder thread, SocketIO broadcaster
app.py	Initialize SocketIO, add connect/join/leave handlers, run with socketio.run()
static/app.js	Replace polling with SocketIO client, handle preview_frame event
templates/base.html	Add SocketIO client script (/socket.io/socket.io.js)

---

Testing Strategy

1. Unit test: run_pipeline with mock queue — verify frames enqueued at correct interval
2. Integration: Start job, connect browser, verify 15+ FPS in devtools Network tab
3. Load test: Multiple browser tabs → verify broadcast works
4. Stress test: Long video (10+ min) → verify no memory leaks, queue stays bounded

---

Risks & Mitigations

Risk	Mitigation
Eventlet monkey-patches stdlib — may break torch/CUDA	Test early; fallback to gevent or native websockets + asyncio
Queue memory growth	Bounded queue (maxsize=30), drop frames when full
Multiple clients	SocketIO rooms — single broadcast to room
Reconnection	SocketIO handles auto-reconnect; client re-joins room on reconnect

---

Dependencies Check

• flask-socketio>=5.3.0 — compatible with Flask 3.x
• eventlet>=0.33.0 — works on ARM64 (Jetson)
• python-engineio>=4.7.0 — transitive

---

Rollout

1. Add deps, install
2. Implement pipeline + job queue changes (backend only)
3. Test with python -c "from app import socketio; print('OK')"
4. Implement frontend WebSocket client
5. Test end-to-end
6. Deploy

---

Alternative: H.264 Streaming (Future)

If JPEG-over-WebSocket isn't smooth enough:

• Use cv2.VideoWriter with H.264 in encoder thread
• Stream via HTTP chunked transfer or WebRTC
• Browser <video> tag with MediaSource API
• More complex but hardware-decoded, smoother at high FPS