feat: switch live preview from WebSocket to MJPEG streaming
- Pipeline writes JPEG to /tmp/feedmill_preview_{job_id}.jpg (atomic)
- Flask serves MJPEG stream at /api/preview/{job_id}
- Frontend uses native <img src> MJPEG — zero JS needed
- Removed flask-socketio, eventlet, socket.io CDN dependencies
- 33% less bandwidth per frame, native browser decode
- Same proven approach as original karung-counting project
This commit is contained in:
jetson committed
1 parent
1e28eb49e2
commit
a5566f3b99
8 files changed
+239
-100
No files matched your search
@@ -0,0 +1,187 @@
|
||||
# Plan: Switch Live Preview from WebSocket+base64 to MJPEG Streaming
|
||||
|
||||
## Goal
|
||||
Replace the WebSocket+base64 live preview with MJPEG streaming (same approach as the original karung-counting project) for faster, smoother preview with less overhead.
|
||||
|
||||
## Why MJPEG is Faster
|
||||
|
||||
| Factor | WebSocket+base64 | MJPEG |
|
||||
|--------|-----------------|-------|
|
||||
| **Frame size** | JPEG + 33% base64 bloat + JSON wrapper | Raw JPEG bytes |
|
||||
| **Transport** | JSON over WebSocket | Binary HTTP stream |
|
||||
| **Decode** | JS sets img.src per frame (triggers decode) | Browser native `<img>` MJPEG decode |
|
||||
| **Code** | Queue + thread + SocketIO emit + JS handler | File write + HTTP generator |
|
||||
| **Dependencies** | flask-socketio, JS socket.io client | None (stdlib only) |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Pipeline (pipeline.py)
|
||||
→ write JPEG to /tmp/feedmill_preview_{job_id}.jpg (atomic tmp+replace)
|
||||
|
||||
Flask endpoint (app.py)
|
||||
GET /api/preview/<job_id>
|
||||
→ reads file in tight loop, yields multipart/x-mixed-replace MJPEG stream
|
||||
|
||||
Frontend (status.html)
|
||||
<img src="/api/preview/{job_id}" />
|
||||
→ browser handles MJPEG natively, no JS needed
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Pipeline — Write to file instead of Queue
|
||||
|
||||
**Modify `src/pipeline.py`:**
|
||||
- Add `preview_path: str | None = None` parameter
|
||||
- Replace queue put with atomic file write:
|
||||
```python
|
||||
if preview_path is not None and frame_idx % max(1, preview_every_n) == 0:
|
||||
h, w = viz.shape[:2]
|
||||
if max(h, w) > preview_max_dim:
|
||||
scale = preview_max_dim / max(h, w)
|
||||
preview_viz = cv2.resize(viz, (int(w * scale), int(h * scale)))
|
||||
else:
|
||||
preview_viz = viz
|
||||
tmp_path = preview_path + ".tmp.jpg"
|
||||
cv2.imwrite(tmp_path, preview_viz, [cv2.IMWRITE_JPEG_QUALITY, preview_jpeg_quality])
|
||||
os.replace(tmp_path, preview_path)
|
||||
```
|
||||
- Remove `preview_queue` parameter (deprecated)
|
||||
|
||||
### Phase 2: Job Queue — Compute preview path, remove broadcaster
|
||||
|
||||
**Modify `src/job.py`:**
|
||||
- Remove `preview_queue`, `_preview_thread`, `_preview_stop` fields
|
||||
- Remove `start_preview_broadcaster()` and `stop_preview_broadcaster()` methods
|
||||
- Add `preview_path: str` field (computed in `__init__`):
|
||||
```python
|
||||
preview_path: str = "" # set in add_job to /tmp/feedmill_preview_{job_id}.jpg
|
||||
```
|
||||
- In `add_job()`, set `job.preview_path = f"/tmp/feedmill_preview_{job.job_id}.jpg"`
|
||||
- In `_run_job()`, pass `preview_path=job.preview_path` to `run_pipeline()`
|
||||
- In cleanup, delete the preview file:
|
||||
```python
|
||||
try:
|
||||
os.remove(job.preview_path)
|
||||
except OSError:
|
||||
pass
|
||||
```
|
||||
|
||||
### Phase 3: Flask — Add MJPEG endpoint, remove SocketIO preview
|
||||
|
||||
**Modify `app.py`:**
|
||||
- Add MJPEG endpoint:
|
||||
```python
|
||||
import time as _time
|
||||
|
||||
@app.route("/api/preview/<job_id>")
|
||||
def api_preview(job_id):
|
||||
job = job_queue.get_job(job_id)
|
||||
if job is None:
|
||||
return jsonify({"error": "not found"}), 404
|
||||
|
||||
preview_path = job.preview_path
|
||||
if not preview_path:
|
||||
return jsonify({"error": "no preview path"}), 404
|
||||
|
||||
def generate():
|
||||
consecutive_fails = 0
|
||||
MAX_FAILS = 30
|
||||
while True:
|
||||
try:
|
||||
with open(preview_path, "rb") as f:
|
||||
jpeg = f.read()
|
||||
consecutive_fails = 0
|
||||
yield (b"--frame\r\n"
|
||||
b"Content-Type: image/jpeg\r\n\r\n" + jpeg + b"\r\n")
|
||||
except FileNotFoundError:
|
||||
consecutive_fails += 1
|
||||
if consecutive_fails >= MAX_FAILS:
|
||||
return
|
||||
_time.sleep(1.0)
|
||||
continue
|
||||
except Exception:
|
||||
consecutive_fails += 1
|
||||
if consecutive_fails >= MAX_FAILS:
|
||||
return
|
||||
_time.sleep(0.5)
|
||||
continue
|
||||
_time.sleep(0.05)
|
||||
return Response(generate(), mimetype="multipart/x-mixed-replace; boundary=frame")
|
||||
```
|
||||
- Remove SocketIO handlers for preview (`on_connect`, `on_disconnect`, `on_join_job`, `on_leave_job`)
|
||||
- Remove `job.start_preview_broadcaster()` calls from upload routes
|
||||
- Remove `flask_socketio` import and `socketio` init (if no longer needed)
|
||||
- Keep `socketio.run()` in `main()` OR switch to `app.run()` if SocketIO is fully removed
|
||||
|
||||
### Phase 4: Frontend — Replace SocketIO with MJPEG img tag
|
||||
|
||||
**Modify `templates/status.html`:**
|
||||
- Change `<img>` src to MJPEG endpoint:
|
||||
```html
|
||||
<img id="live-preview-img" class="live-preview-img"
|
||||
src="/api/preview/{{ job.job_id }}"
|
||||
alt="Annotated frame from video processing" />
|
||||
```
|
||||
- Remove `preview-placeholder` div (MJPEG auto-shows when frames arrive)
|
||||
|
||||
**Modify `static/app.js`:**
|
||||
- Remove SocketIO code:
|
||||
```javascript
|
||||
// DELETE: var socket = io();
|
||||
// DELETE: socket.on('connect', ...)
|
||||
// DELETE: socket.on('preview_frame', ...)
|
||||
// DELETE: window.addEventListener('beforeunload', ...)
|
||||
```
|
||||
- Remove `updateLivePreview()` function (no longer called)
|
||||
|
||||
### Phase 5: Cleanup — Remove SocketIO if fully unused
|
||||
|
||||
**Modify `app.py`:**
|
||||
- Remove `from flask_socketio import SocketIO, join_room, leave_room`
|
||||
- Remove `socketio = SocketIO(...)` line
|
||||
- Change `socketio.run(app, ...)` to `app.run(host, port, debug)`
|
||||
|
||||
**Modify `pyproject.toml`:**
|
||||
- Remove `flask-socketio` and `eventlet` from optional deps
|
||||
|
||||
**Modify `templates/base.html`:**
|
||||
- Remove SocketIO CDN script tag
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `src/pipeline.py` | Replace `preview_queue` with `preview_path` file write |
|
||||
| `src/job.py` | Compute `preview_path`, remove broadcaster thread/queue |
|
||||
| `app.py` | Add MJPEG endpoint, remove SocketIO handlers, use `app.run()` |
|
||||
| `templates/status.html` | Add `src="/api/preview/{job_id}"` to img tag |
|
||||
| `templates/base.html` | Remove SocketIO CDN script |
|
||||
| `static/app.js` | Remove SocketIO code and `updateLivePreview()` |
|
||||
| `pyproject.toml` | Remove `flask-socketio`, `eventlet` deps |
|
||||
|
||||
## Expected Performance
|
||||
|
||||
| Metric | Before (WebSocket) | After (MJPEG) |
|
||||
|--------|-------------------|---------------|
|
||||
| Frame size | ~50-80 KB (base64+JSON) | ~30-50 KB (raw JPEG) |
|
||||
| Transport overhead | 33% base64 + JSON | 0% |
|
||||
| Decode latency | JS img.src assignment | Native browser MJPEG |
|
||||
| Dependencies | flask-socketio, eventlet, socket.io CDN | None |
|
||||
| JS complexity | ~20 lines SocketIO code | 0 lines |
|
||||
|
||||
## Testing
|
||||
|
||||
1. Run `python -m pytest tests/ -v --tb=short` — 62/62 pass
|
||||
2. Start web UI, upload video, start job
|
||||
3. Verify MJPEG stream loads at `http://localhost:9000/api/preview/{job_id}`
|
||||
4. Verify live preview updates smoothly in browser
|
||||
5. Verify cleanup: preview file deleted after job completes
|
||||
|
||||
## Rollout
|
||||
|
||||
1. Implement all changes
|
||||
2. Test locally
|
||||
3. Commit and push
|
||||
4. Restart web UI
|
||||
Reference in new issue
Block a user