T17: backend penyimpanan bersama db.py (sqlite | mysql via PyMySQL)

- db.py: abstraksi connect/query/queryone/replace_snapshot; %s → ? utk sqlite;
  replace snapshot mysql = CREATE IF NOT EXISTS + DELETE + INSERT (MVCC)
- config/.env.example: VEX_DB_BACKEND (default sqlite) + VEX_DB_HOST/PORT/USER/PASS/NAME
- get_voters.persist → db.replace_snapshot; dashboard baca via db.query
- LIKE pakai ESCAPE '!' (backslash memutus literal string MySQL, B2);
  db.query selalu kembalikan list (PyMySQL tuple vs sqlite list, B3)
- docker-compose.yml: kontainer uji mariadb:11 (databisnisid-mariadb, 127.0.0.1:3306)
- test_mariadb.py: suite integrasi opt-in (skip bila bukan mysql)
- SPEC.md §V17/T17/B2/B3; AGENTS.md runbook
This commit is contained in:
proitlab committed 2026-08-05 12:41:17 +07:00
1 parent 063fb5e5eb
commit fdb5d0de15
12 files changed
+307 -79

No files matched your search

+11
View File
@@ -10,6 +10,17 @@ VEX_API_NODE=https://v2.vexascan.com:2096
# Lokasi file SQLite (disarankan absolut bila dijadwalkan via cron) # Lokasi file SQLite (disarankan absolut bila dijadwalkan via cron)
VEX_DB_PATH=voters.db VEX_DB_PATH=voters.db
# Backend basis data: sqlite (default) atau mysql (MariaDB/MySQL via PyMySQL)
VEX_DB_BACKEND=sqlite
# Koneksi MariaDB/MySQL — dipakai saat VEX_DB_BACKEND=mysql
# Cocokkan dengan docker-compose.yml (MARIADB_DATABASE/USER/PASSWORD)
VEX_DB_HOST=127.0.0.1
VEX_DB_PORT=3306
VEX_DB_USER=databisnis
VEX_DB_PASS=databisnis
VEX_DB_NAME=databisnisid
# Syarat minimum stake (VEX) agar pemilih disimpan # Syarat minimum stake (VEX) agar pemilih disimpan
VEX_MIN_STAKED_VEX=1000 VEX_MIN_STAKED_VEX=1000
+13 -7
View File
@@ -5,10 +5,13 @@ Two tools scan the Vexanium blockchain voters table for accounts whose only vote
## Run ## Run
- Node reference script: `node get_voters.js` (no deps, Node 18+). - Node reference script: `node get_voters.js` (no deps, Node 18+).
- Python tool (production): `./venv/bin/python get_voters.py`. venv is Python 3.12, deps `requests` + `flask` + `python-dotenv` (`requirements.txt`). Install with `./venv/bin/pip install -r requirements.txt`. - Python tool (production): `./venv/bin/python get_voters.py`. venv is Python 3.12, deps `requests` + `flask` + `python-dotenv` + `gunicorn` + `pymysql` (mysql backend only) (`requirements.txt`). Install with `./venv/bin/pip install -r requirements.txt`.
- Web dashboard (read-only, reads `voters.db`): production `./venv/bin/gunicorn -c gunicorn.conf.py dashboard:app` → http://127.0.0.1:5000/ (run from repo dir). Easier: `./run.sh` (same command, works from any cwd, `$@` passed through). `gunicorn.conf.py` imports `config` → `.env` honored; `DASH_WORKERS` (default 2) controls workers, `DASH_HOST`/`DASH_PORT` the bind. Dev server (single-process) still works via `./venv/bin/python dashboard.py`. Paged 50/page, sorted `staked DESC`, live owner search (`/api/search`). Data freshness comes from the daily scan run — the dashboard never scans. - Storage backend: `db.py` abstracts it. Default `sqlite` (`VEX_DB_PATH`, stdlib `sqlite3`, WAL). Optional `mysql` (`VEX_DB_BACKEND=mysql` + `VEX_DB_HOST/PORT/USER/PASS/NAME`, PyMySQL). Oracle tests stay on sqlite; `test_mariadb.py` is opt-in (skips unless `VEX_DB_BACKEND=mysql`). Query SQL is written once with `%s` placeholders (translated to `?` for sqlite); `db.query` always returns a list.
- Config: all tunables load from env / `.env` via `config.py` (python-dotenv): `VEX_TARGET_BP`, `VEX_API_NODE`, `VEX_DB_PATH`, `VEX_MIN_STAKED_VEX`, `DASH_PAGE_SIZE`, `DASH_HOST`, `DASH_PORT`, `DASH_WORKERS`, `VEX_STALE_DAYS`, `DATABISNIS_API`, `DASH_LIQUID_TTL`. Copy `.env.example` → `.env` to override; `.env` is gitignored. Chain constants (`vexcore`/scope/table) stay hardcoded. - Test MariaDB/MySQL via docker: `docker compose up -d` (mariadb:11 container `databisnisid-mariadb`, localhost-only `127.0.0.1:3306`, db/user/pass `databisnisid`/`databisnis`/`databisnis`, named volume, healthcheck). Stop/remove with `docker compose down`; wipe data with `docker compose down -v`. Verify with `docker compose exec mariadb mariadb -u databisnis -pdatabisnis databisnisid -e 'SELECT 1'`. Smoke against the container:
- Syntax check: `node --check get_voters.js`, `./venv/bin/python -m py_compile config.py get_voters.py dashboard.py gunicorn.conf.py test_get_voters.py test_dashboard.py`. `VEX_DB_BACKEND=mysql VEX_DB_HOST=127.0.0.1 VEX_DB_PORT=3306 VEX_DB_USER=databisnis VEX_DB_PASS=databisnis VEX_DB_NAME=databisnisid ./venv/bin/python test_mariadb.py`
- Web dashboard (read-only, reads the store via `db.py`): production `./venv/bin/gunicorn -c gunicorn.conf.py dashboard:app` → http://127.0.0.1:5000/ (run from repo dir). Easier: `./run.sh` (same command, works from any cwd, `$@` passed through). `gunicorn.conf.py` imports `config` → `.env` honored; `DASH_WORKERS` (default 2) controls workers, `DASH_HOST`/`DASH_PORT` the bind. Dev server (single-process) still works via `./venv/bin/python dashboard.py`. Paged 50/page, sorted `staked DESC`, live owner search (`/api/search`). Data freshness comes from the daily scan run — the dashboard never scans.
- Config: all tunables load from env / `.env` via `config.py` (python-dotenv): `VEX_TARGET_BP`, `VEX_API_NODE`, `VEX_DB_PATH`, `VEX_DB_BACKEND`, `VEX_DB_HOST`, `VEX_DB_PORT`, `VEX_DB_USER`, `VEX_DB_PASS`, `VEX_DB_NAME`, `VEX_MIN_STAKED_VEX`, `DASH_PAGE_SIZE`, `DASH_HOST`, `DASH_PORT`, `DASH_WORKERS`, `VEX_STALE_DAYS`, `DATABISNIS_API`, `DASH_LIQUID_TTL`. Copy `.env.example` → `.env` to override; `.env` is gitignored. Chain constants (`vexcore`/scope/table) stay hardcoded.
- Syntax check: `node --check get_voters.js`, `./venv/bin/python -m py_compile config.py get_voters.py dashboard.py db.py gunicorn.conf.py test_get_voters.py test_dashboard.py test_mariadb.py`.
- Tests (the verification oracles): `./venv/bin/python test_get_voters.py` and `./venv/bin/python test_dashboard.py` must both exit 0. They mock the network / use a temp DB — the live node is too flaky/slow for a full-scan test. Run after touching the relevant file. - Tests (the verification oracles): `./venv/bin/python test_get_voters.py` and `./venv/bin/python test_dashboard.py` must both exit 0. They mock the network / use a temp DB — the live node is too flaky/slow for a full-scan test. Run after touching the relevant file.
## Key facts ## Key facts
@@ -27,11 +30,13 @@ Two tools scan the Vexanium blockchain voters table for accounts whose only vote
## Layout ## Layout
- `SPEC.md` — spec (goal/constraints/interfaces/invariants/tasks/bug log), in caveman encoding. Build/backprop flow through it. - `SPEC.md` — spec (goal/constraints/interfaces/invariants/tasks/bug log), in caveman encoding. Build/backprop flow through it.
- `get_voters.py` — production fetcher: scan → filter (stake + freshness, basi dibuang) → SQLite `voters.db` (each run replaces the table = daily snapshot, with `scanned_at` + derived `last_vote`; never appends history). - `get_voters.py` — production fetcher: scan → filter (stake + freshness, basi dibuang) → `db.replace_snapshot` (each run replaces the table = daily snapshot, with `scanned_at` + derived `last_vote`; never appends history).
- `db.py` — storage abstraction (sqlite default | mysql via PyMySQL); `connect/query/queryone/replace_snapshot`; `%s` → `?` for sqlite; `query` returns list.
- `get_voters.js` — reference implementation only. - `get_voters.js` — reference implementation only.
- `dashboard.py` + `templates/index.html` + `static/style.css` + `static/app.js` — Flask web dashboard; reads `voters.db`, styled per `DESIGN.md`; `app.js` = debounced live owner search (fetch `/api/search`), degrades to the server-side `?q=` GET form if JS is off. - `dashboard.py` + `templates/index.html` + `static/style.css` + `static/app.js` — Flask web dashboard; reads the store via `db.py`, styled per `DESIGN.md`; `app.js` = debounced live owner search (fetch `/api/search`), degrades to the server-side `?q=` GET form if JS is off.
- `gunicorn.conf.py` — gunicorn production config (bind/workers from `config`, sync worker). `run.sh` — launcher: `./run.sh` = `./venv/bin/gunicorn -c gunicorn.conf.py dashboard:app` from any cwd. - `gunicorn.conf.py` — gunicorn production config (bind/workers from `config`, sync worker). `run.sh` — launcher: `./run.sh` = `./venv/bin/gunicorn -c gunicorn.conf.py dashboard:app` from any cwd.
- `config.py` — loads env/`.env` (python-dotenv) → `TARGET_BP`, `API_NODE`, `DB_PATH`, `MIN_STAKED_VEX`, `PAGE_SIZE`, `DASH_HOST`, `DASH_PORT`, `DASH_WORKERS`, `VEX_STALE_DAYS`, `DATABISNIS_API`, `DASH_LIQUID_TTL`; shared by get_voters & dashboard. - `config.py` — loads env/`.env` (python-dotenv) → `TARGET_BP`, `API_NODE`, `DB_PATH`, `DB_BACKEND`, `DB_HOST`, `DB_PORT`, `DB_USER`, `DB_PASS`, `DB_NAME`, `MIN_STAKED_VEX`, `PAGE_SIZE`, `DASH_HOST`, `DASH_PORT`, `DASH_WORKERS`, `VEX_STALE_DAYS`, `DATABISNIS_API`, `DASH_LIQUID_TTL`; shared by get_voters & dashboard.
- `docker-compose.yml` — mariadb:11 test container (`databisnisid-mariadb`, localhost-only 3306) for the mysql backend.
- `DESIGN.md` — Bugatti austere style guide; the dashboard's CSS maps its tokens (canvas #000000, hairline #262626, weight 400 everywhere, fonts Saira Condensed / EB Garamond / JetBrains Mono). - `DESIGN.md` — Bugatti austere style guide; the dashboard's CSS maps its tokens (canvas #000000, hairline #262626, weight 400 everywhere, fonts Saira Condensed / EB Garamond / JetBrains Mono).
- `voters.db` — SQLite output (daily snapshot, gitignored in spirit). - `voters.db` — SQLite output (daily snapshot, gitignored in spirit).
@@ -41,3 +46,4 @@ Two tools scan the Vexanium blockchain voters table for accounts whose only vote
- Dashboard UI labels are Indonesian uppercase captions (e.g. "DAFTAR PEMILIH", "TOTAL PEMILIH"). - Dashboard UI labels are Indonesian uppercase captions (e.g. "DAFTAR PEMILIH", "TOTAL PEMILIH").
- JS style: 2-space indent, semicolons, single quotes, trailing commas, async/await. - JS style: 2-space indent, semicolons, single quotes, trailing commas, async/await.
- Python style: 4-space indent, stdlib `sqlite3`, `requests`, `flask`, PEP8. - Python style: 4-space indent, stdlib `sqlite3`, `requests`, `flask`, PEP8.
- New SQL shared across backends: `%s` placeholders (never `?`), `ESCAPE '!'` for LIKE (backslash breaks MySQL string literals), no MySQL-only syntax.
+9 -5
View File
@@ -6,9 +6,9 @@ Re-runnable scheduled. Python 3.12, venv `./venv`.
## §C — Constraints ## §C — Constraints
- venv `./venv` exists (py3.12, only pip) — install deps there, ⊥ global - venv `./venv` exists (py3.12, only pip) — install deps there, ⊥ global
- lib: `requests`, `flask`, `python-dotenv`, `gunicorn` ! pip-installed; others stdlib - lib: `requests`, `flask`, `python-dotenv`, `gunicorn` ! pip-installed; `pymysql` utk backend mysql (opt-in); others stdlib
- config via env/`.env` (`config.py`, python-dotenv); default = konstanta lama - config via env/`.env` (`config.py`, python-dotenv); default = konstanta lama
- db: SQLite stdlib `sqlite3` → `DB_PATH` (default `voters.db`); WAL journal mode → pembaca tak terblokir saat snapshot harian - db: `db.py` abstraksi backend — `sqlite` (default, stdlib `sqlite3` → `DB_PATH`, WAL → pembaca tak terblokir) atau `mysql` (MariaDB/MySQL via PyMySQL, `VEX_DB_*`); container uji via `docker-compose.yml`
- api: `API_NODE` (default `https://v2.vexascan.com:2096`, public, flaky → retry) - api: `API_NODE` (default `https://v2.vexascan.com:2096`, public, flaky → retry)
- saldo liquid akun BP: API databisnis (`DATABISNIS_API`, default `https://api.databisnis.id`) — diambil dashboard, cache TTL `DASH_LIQUID_TTL` (default 60s), cooldown kegagalan - saldo liquid akun BP: API databisnis (`DATABISNIS_API`, default `https://api.databisnis.id`) — diambil dashboard, cache TTL `DASH_LIQUID_TTL` (default 60s), cooldown kegagalan
- contract=scope=`vexcore` (≠ `vexio`, EOS convention ⊥) — ⊥ env-able - contract=scope=`vexcore` (≠ `vexio`, EOS convention ⊥) — ⊥ env-able
@@ -25,12 +25,12 @@ Re-runnable scheduled. Python 3.12, venv `./venv`.
api: POST `https://v2.vexascan.com:2096/v1/chain/get_table_rows` api: POST `https://v2.vexascan.com:2096/v1/chain/get_table_rows`
→ body {json:true, code:`vexcore`, scope:`vexcore`, table:`voters`, lower_bound, limit:500} → body {json:true, code:`vexcore`, scope:`vexcore`, table:`voters`, lower_bound, limit:500}
→ rows[owner, proxy, producers[], staked, last_vote_weight, proxied_vote_weight, is_proxy], more, next_key → rows[owner, proxy, producers[], staked, last_vote_weight, proxied_vote_weight, is_proxy], more, next_key
db: `voters.db` → table `voters` (owner PK, weight, staked, scanned_at, last_vote) — ganti tiap skan; weight disimpan (audit/asal `last_vote`) tapi ⊥ ditampilkan web db: `db.py` → table `voters` (owner PK, weight, staked, scanned_at, last_vote) via `connect/query/queryone/replace_snapshot`; sqlite = DROP+CREATE+INSERT (WAL), mysql = CREATE IF NOT EXISTS + DELETE + INSERT satu transaksi; ganti tiap skan; weight disimpan (audit/asal `last_vote`) tapi ⊥ ditampilkan web
cmd: `./venv/bin/python get_voters.py` → stdout summary (id-ID) cmd: `./venv/bin/python get_voters.py` → stdout summary (id-ID)
web: GET `/` (Flask, disajikan gunicorn di produksi) → HTML spec-list, paged 50/halaman, `ORDER BY staked DESC, owner ASC`; ⊥ mutation (read-only) web: GET `/` (Flask, disajikan gunicorn di produksi) → HTML spec-list, paged 50/halaman, `ORDER BY staked DESC, owner ASC`; ⊥ mutation (read-only)
web: GET `/` + `?q=<substring>` → filter owner (server-side, no-JS fallback); pager bawa `q` web: GET `/` + `?q=<substring>` → filter owner (server-side, no-JS fallback); pager bawa `q`
web: GET `/api/search?q=` → JSON `{query,count,cap,results:[{owner,staked,weight,rank,last_vote}]}`, rank global, cap 500 web: GET `/api/search?q=` → JSON `{query,count,cap,results:[{owner,staked,weight,rank,last_vote}]}`, rank global, cap 500
env: `VEX_TARGET_BP`, `VEX_API_NODE`, `VEX_DB_PATH`, `VEX_MIN_STAKED_VEX`, `DASH_PAGE_SIZE`, `DASH_HOST`, `DASH_PORT`, `DASH_WORKERS`, `VEX_STALE_DAYS`, `DATABISNIS_API`, `DASH_LIQUID_TTL` — via `config.py` (`.env`) env: `VEX_TARGET_BP`, `VEX_API_NODE`, `VEX_DB_PATH`, `VEX_DB_BACKEND`, `VEX_DB_HOST`, `VEX_DB_PORT`, `VEX_DB_USER`, `VEX_DB_PASS`, `VEX_DB_NAME`, `VEX_MIN_STAKED_VEX`, `DASH_PAGE_SIZE`, `DASH_HOST`, `DASH_PORT`, `DASH_WORKERS`, `VEX_STALE_DAYS`, `DATABISNIS_API`, `DASH_LIQUID_TTL` — via `config.py` (`.env`)
## §V — Invariants ## §V — Invariants
V1: ∀ fetch → retry ≥3 on timeout/http err (like JS fetchWithRetry) V1: ∀ fetch → retry ≥3 on timeout/http err (like JS fetchWithRetry)
@@ -48,6 +48,7 @@ V12: produksi web disajikan gunicorn (`dashboard:app`, `gunicorn.conf.py`); `DAS
V13: `last_vote` = `2000-01-01 + log2(last_vote_weight / (staked×10000)) × 365.25 hari` (None bila bobot/stake nol); estimasi yang melewati `now` (unstake tanpa revote) diklamp ke `now`; voter dgn `last_vote=None` atau `< now − VEX_STALE_DAYS` dibuang di `normalize` (⊥ disimpan) → DB hanya berisi pemilih segar V13: `last_vote` = `2000-01-01 + log2(last_vote_weight / (staked×10000)) × 365.25 hari` (None bila bobot/stake nol); estimasi yang melewati `now` (unstake tanpa revote) diklamp ke `now`; voter dgn `last_vote=None` atau `< now − VEX_STALE_DAYS` dibuang di `normalize` (⊥ disimpan) → DB hanya berisi pemilih segar
V15: web list = kolom RANK/AKUN/STAKE (VEX)/VOTE TERAKHIR — ⊥ BOBOT SUARA, ⊥ penanda/stat/filter basi; stats = 3 sel (TOTAL PEMILIH/TOTAL VEX/STAKE TERTINGGI); `/api/search` hasil tetap `{owner,staked,weight,rank,last_vote}` (weight disimpan & di-API, ⊥ dirender) V15: web list = kolom RANK/AKUN/STAKE (VEX)/VOTE TERAKHIR — ⊥ BOBOT SUARA, ⊥ penanda/stat/filter basi; stats = 3 sel (TOTAL PEMILIH/TOTAL VEX/STAKE TERTINGGI); `/api/search` hasil tetap `{owner,staked,weight,rank,last_vote}` (weight disimpan & di-API, ⊥ dirender)
V16: saldo liquid akun `TARGET_BP` dari `GET {DATABISNIS_API}/v2/state/get_account` (parse `account.core_liquid_balance`) → sel stat SALDO LIQUID; cache in-memory per worker TTL `DASH_LIQUID_TTL`; gagal fetch → nilai lama (atau `—` bila belum pernah sukses) + `at` ikut diset (cooldown, ⊥ pukulan berulang); read-only, ⊥ tulis DB V16: saldo liquid akun `TARGET_BP` dari `GET {DATABISNIS_API}/v2/state/get_account` (parse `account.core_liquid_balance`) → sel stat SALDO LIQUID; cache in-memory per worker TTL `DASH_LIQUID_TTL`; gagal fetch → nilai lama (atau `—` bila belum pernah sukses) + `at` ikut diset (cooldown, ⊥ pukulan berulang); read-only, ⊥ tulis DB
V17: penyimpanan lewat `db.py` (backend `VEX_DB_BACKEND` = `sqlite` default | `mysql`); query berbagi sintaks pakai placeholder `%s` (diterjemahkan `?` utk sqlite); LIKE escape pakai `ESCAPE '!'` — backslash memutus literal string MySQL (B2); `db.query` → list (fetchall sqlite=list / PyMySQL=tuple diseragamkan, B3); replace snapshot mysql = CREATE IF NOT EXISTS + DELETE + INSERT satu transaksi (MVCC → pembaca dapat snapshot konsisten saat ganti harian, tak kena torn read); oracle utama (test_get_voters/test_dashboard) tetap sqlite hermetik; `test_mariadb.py` opt-in (skip exit 0 bila `VEX_DB_BACKEND != mysql`)
## §T — Tasks ## §T — Tasks
id|status|task|cites id|status|task|cites
@@ -67,7 +68,10 @@ T13|x|produksi web: gunicorn (gunicorn.conf.py, DASH_WORKERS) + WAL di persist|V
T14|x|stale-vote: last_vote di scan (rasio bobot/stake); basi/unverifiable dibuang di normalize (⊥ disimpan)|V13,I.db T14|x|stale-vote: last_vote di scan (rasio bobot/stake); basi/unverifiable dibuang di normalize (⊥ disimpan)|V13,I.db
T15|x|hapus bobot suara + UI basi di web (kolom/stats/penanda/filter `?stale=1`); oracle diperbarui|V15,I.web T15|x|hapus bobot suara + UI basi di web (kolom/stats/penanda/filter `?stale=1`); oracle diperbarui|V15,I.web
T16|x|saldo liquid akun di dashboard: API databisnis + cache TTL + cooldown; sel SALDO LIQUID|V16,I.web,I.env T16|x|saldo liquid akun di dashboard: API databisnis + cache TTL + cooldown; sel SALDO LIQUID|V16,I.web,I.env
T17|x|backend shared db.py (sqlite|mysql, `%s`→`?`, ESCAPE '!'), config VEX_DB_*, refactor fetcher+dashboard, test_mariadb.py opt-in, runbook docker-compose|V17,I.db,I.env
## §B — Bug log ## §B — Bug log
id|date|cause|fix id|date|cause|fix
B1|2026-08-04|node balikin `staked` sbg string utk sebagian baris → TypeError int÷int|V8 B1|2026-08-04|node balikin `staked` sbg string utk sebagian baris → TypeError int÷int|V8
B2|2026-08-05|`ESCAPE '\'` valid di SQLite tapi memutus literal string MySQL (`'\'` = string tak tertutup) → error 1064 di backend mysql|ganti karakter escape ke `!` (netral di kedua backend), `_escape_like` esc `!`,`%`,`_` (V17)
B3|2026-08-05|PyMySQL `fetchall()` → tuple, SQLite → list → asersi `== []` gagal di mysql|`db.query` bungkus `list()` → tipe seragam (V17)
+10
View File
@@ -19,6 +19,16 @@ API_NODE = os.getenv('VEX_API_NODE', 'https://v2.vexascan.com:2096')
# Lokasi basis data SQLite (dipakai fetcher & dashboard) # Lokasi basis data SQLite (dipakai fetcher & dashboard)
DB_PATH = os.getenv('VEX_DB_PATH', 'voters.db') DB_PATH = os.getenv('VEX_DB_PATH', 'voters.db')
# Backend basis data: 'sqlite' (default) atau 'mysql' (MariaDB/MySQL via PyMySQL)
DB_BACKEND = os.getenv('VEX_DB_BACKEND', 'sqlite')
# Koneksi MariaDB/MySQL (dipakai saat DB_BACKEND='mysql')
DB_HOST = os.getenv('VEX_DB_HOST', '127.0.0.1')
DB_PORT = int(os.getenv('VEX_DB_PORT', '3306'))
DB_USER = os.getenv('VEX_DB_USER', '')
DB_PASS = os.getenv('VEX_DB_PASS', '')
DB_NAME = os.getenv('VEX_DB_NAME', '')
# Syarat minimum stake (dalam VEX) untuk disimpan # Syarat minimum stake (dalam VEX) untuk disimpan
MIN_STAKED_VEX = float(os.getenv('VEX_MIN_STAKED_VEX', '1000')) MIN_STAKED_VEX = float(os.getenv('VEX_MIN_STAKED_VEX', '1000'))
+27 -37
View File
@@ -6,15 +6,15 @@ API databisnis dengan cache TTL singkat (V16). Styling per `DESIGN.md`.
Konfigurasi dari env/`.env`. Konfigurasi dari env/`.env`.
""" """
import sqlite3
import time import time
from datetime import datetime from datetime import datetime
import requests import requests
from flask import Flask, abort, jsonify, render_template, request from flask import Flask, abort, jsonify, render_template, request
import db
from config import (DASH_HOST, DASH_LIQUID_TTL, DASH_PORT, DATABISNIS_API, from config import (DASH_HOST, DASH_LIQUID_TTL, DASH_PORT, DATABISNIS_API,
DB_PATH, PAGE_SIZE, TARGET_BP) PAGE_SIZE, TARGET_BP)
SEARCH_CAP = 500 SEARCH_CAP = 500
MAX_Q_LEN = 64 MAX_Q_LEN = 64
@@ -72,10 +72,15 @@ def _get_liquid_vex():
def _escape_like(q): def _escape_like(q):
"""Escape wildcard LIKE agar `%`/`_` dari pengguna tidak jadi pola.""" """Escape wildcard LIKE agar `%`/`_`/`!` dari pengguna tidak jadi pola.
return (q.replace('\\', '\\\\')
.replace('%', '\\%') `!` dipakai sebagai karakter escape, bukan backslash: backslash bermasalah
.replace('_', '\\_')) di literal string MySQL (`'\'` memutus string), sedangkan `!` netral di
kedua backend (sqlite & mysql sama-sama paham `ESCAPE '!'`).
"""
return (q.replace('!', '!!')
.replace('%', '!%')
.replace('_', '!_'))
def _filter_sql(q): def _filter_sql(q):
@@ -83,7 +88,7 @@ def _filter_sql(q):
clauses = [] clauses = []
params = [] params = []
if q: if q:
clauses.append("owner LIKE ? ESCAPE '\\'") clauses.append("owner LIKE %s ESCAPE '!'")
params.append(f'%{_escape_like(q)}%') params.append(f'%{_escape_like(q)}%')
if not clauses: if not clauses:
return '', [] return '', []
@@ -106,40 +111,29 @@ def _db_rows(page, q):
' OR (v2.staked = v.staked AND v2.owner <= v.owner)) AS rn, ' ' OR (v2.staked = v.staked AND v2.owner <= v.owner)) AS rn, '
' v.last_vote ' ' v.last_vote '
'FROM voters v ' + where + ' ' 'FROM voters v ' + where + ' '
'ORDER BY v.staked DESC, v.owner ASC LIMIT ? OFFSET ?' 'ORDER BY v.staked DESC, v.owner ASC LIMIT %s OFFSET %s'
) )
conn = sqlite3.connect(DB_PATH) return db.query(sql, params + [PAGE_SIZE, offset])
try:
return conn.execute(sql, params + [PAGE_SIZE, offset]).fetchall()
finally:
conn.close()
def _count(q): def _count(q):
"""V11: banyak baris cocok filter (atau total bila tanpa filter).""" """V11: banyak baris cocok filter (atau total bila tanpa filter)."""
where, params = _filter_sql(q) where, params = _filter_sql(q)
conn = sqlite3.connect(DB_PATH) return db.queryone(
try: 'SELECT COUNT(*) FROM voters ' + where, params
return conn.execute( )[0]
'SELECT COUNT(*) FROM voters ' + where, params
).fetchone()[0]
finally:
conn.close()
def _stats(): def _stats():
"""Ringkasan agregat untuk band spec (selalu global, ⊥ dipengaruhi search).""" """Ringkasan agregat untuk band spec (selalu global, ⊥ dipengaruhi search)."""
conn = sqlite3.connect(DB_PATH) row = db.queryone(
try: 'SELECT COUNT(*), COALESCE(SUM(staked), 0), COALESCE(MAX(staked), 0) '
total_voters, total_staked, top_staked = conn.execute( 'FROM voters'
'SELECT COUNT(*), COALESCE(SUM(staked), 0), COALESCE(MAX(staked), 0) ' )
'FROM voters' total_voters, total_staked, top_staked = row
).fetchone() scanned = db.queryone(
scanned = conn.execute( 'SELECT COALESCE(MAX(scanned_at), \'\') FROM voters'
'SELECT COALESCE(MAX(scanned_at), \'\') FROM voters' )[0]
).fetchone()[0]
finally:
conn.close()
return total_voters, total_staked, top_staked, scanned return total_voters, total_staked, top_staked, scanned
@@ -185,13 +179,9 @@ def api_search():
' OR (v2.staked = v.staked AND v2.owner <= v.owner)) AS rn, ' ' OR (v2.staked = v.staked AND v2.owner <= v.owner)) AS rn, '
' v.last_vote ' ' v.last_vote '
'FROM voters v ' + where + ' ' 'FROM voters v ' + where + ' '
'ORDER BY v.staked DESC, v.owner ASC LIMIT ?' 'ORDER BY v.staked DESC, v.owner ASC LIMIT %s'
) )
conn = sqlite3.connect(DB_PATH) rows = db.query(sql, params + [SEARCH_CAP])
try:
rows = conn.execute(sql, params + [SEARCH_CAP]).fetchall()
finally:
conn.close()
return jsonify({ return jsonify({
'query': q, 'query': q,
'count': len(rows), 'count': len(rows),
+97
View File
@@ -0,0 +1,97 @@
"""Abstraksi penyimpanan: sqlite (default) atau MariaDB/MySQL (PyMySQL).
Pilih backend via config `VEX_DB_BACKEND` (env `.env`). Kedua backend memakai
skema `voters` yang sama; query berbagi sintaks ditulis dengan placeholder
`%s` yang diterjemahkan ke `?` untuk sqlite. Dashboard read-only (V10).
"""
import sqlite3
from config import (DB_BACKEND, DB_HOST, DB_NAME, DB_PASS, DB_PORT, DB_PATH,
DB_USER)
# Skema voters — sama untuk kedua backend (I.db)
_COLS = (
'owner VARCHAR(13) PRIMARY KEY, '
'weight TEXT NOT NULL, '
'staked DOUBLE NOT NULL, '
'scanned_at TEXT NOT NULL, '
'last_vote TEXT'
)
_SCHEMA = f'CREATE TABLE voters ({_COLS})'
_SCHEMA_IF_NOT_EXISTS = f'CREATE TABLE IF NOT EXISTS voters ({_COLS})'
def _translate(sql):
"""sqlite pakai `?`; mysql pakai `%s`. Ubah `%s` → `?` untuk sqlite."""
return sql.replace('%s', '?') if DB_BACKEND == 'sqlite' else sql
def connect():
"""Buka koneksi backend. sqlite: WAL agar pembaca tak terblokir."""
if DB_BACKEND == 'sqlite':
conn = sqlite3.connect(DB_PATH)
conn.execute('PRAGMA journal_mode=WAL')
return conn
if DB_BACKEND == 'mysql':
import pymysql
return pymysql.connect(
host=DB_HOST, port=DB_PORT, user=DB_USER, password=DB_PASS,
database=DB_NAME, charset='utf8mb4',
)
raise ValueError(f'VEX_DB_BACKEND tak dikenal: {DB_BACKEND!r}')
def query(sql, params=()):
"""Jalankan SELECT → daftar baris (tuple), koneksi dibuka-tutup tiap panggil."""
conn = connect()
try:
cur = conn.cursor()
cur.execute(_translate(sql), params)
return list(cur.fetchall()) # sqlite→list, PyMySQL→tuple: seragamkan
finally:
conn.close()
def queryone(sql, params=()):
"""Jalankan SELECT → baris pertama (tuple) atau None."""
conn = connect()
try:
cur = conn.cursor()
cur.execute(_translate(sql), params)
return cur.fetchone()
finally:
conn.close()
def replace_snapshot(rows, scanned_at):
"""V7: ganti snapshot voters tiap skan (⊥ history/append).
sqlite: DROP+CREATE+INSERT (WAL melindungi pembaca).
mysql: CREATE IF NOT EXISTS + DELETE + INSERT dalam satu transaksi
→ MVCC memberi pembaca snapshot konsisten (⊥ torn read saat ganti harian).
"""
tuples = [(r['owner'], r['weight'], r['staked'], scanned_at, r.get('last_vote'))
for r in rows]
conn = connect()
try:
cur = conn.cursor()
if DB_BACKEND == 'sqlite':
cur.execute('DROP TABLE IF EXISTS voters')
cur.execute(_SCHEMA)
cur.executemany(
'INSERT INTO voters (owner, weight, staked, scanned_at, last_vote) '
'VALUES (?, ?, ?, ?, ?)',
tuples,
)
else:
cur.execute(_SCHEMA_IF_NOT_EXISTS)
cur.execute('DELETE FROM voters')
cur.executemany(
'INSERT INTO voters (owner, weight, staked, scanned_at, last_vote) '
'VALUES (%s, %s, %s, %s, %s)',
tuples,
)
conn.commit()
finally:
conn.close()
+22
View File
@@ -0,0 +1,22 @@
services:
mariadb:
image: mariadb:11
container_name: databisnisid-mariadb
restart: unless-stopped
environment:
MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD:-root}
MARIADB_DATABASE: ${MARIADB_DATABASE:-databisnisid}
MARIADB_USER: ${MARIADB_USER:-databisnis}
MARIADB_PASSWORD: ${MARIADB_PASSWORD:-databisnis}
ports:
- "127.0.0.1:3306:3306"
volumes:
- mariadb_data:/var/lib/mysql
healthcheck:
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
interval: 5s
timeout: 5s
retries: 12
volumes:
mariadb_data:
+7 -26
View File
@@ -5,14 +5,14 @@ dibuang. Berjalan imparsial (repeatable) dan siap dijadwalkan (cron).
""" """
import math import math
import sqlite3
import sys import sys
import time import time
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
import requests import requests
from config import API_NODE, DB_PATH, MIN_STAKED_VEX, TARGET_BP, VEX_STALE_DAYS import db
from config import API_NODE, MIN_STAKED_VEX, TARGET_BP, VEX_STALE_DAYS
# Kontrak & scope sistem Vexanium = vexcore (BUKAN vexio sesuai konvensi EOS) # Kontrak & scope sistem Vexanium = vexcore (BUKAN vexio sesuai konvensi EOS)
CODE = 'vexcore' CODE = 'vexcore'
@@ -145,33 +145,14 @@ def normalize(row, now=None):
def persist(rows): def persist(rows):
"""V7: ganti snapshot — DROP+INSERT tiap skan, ⊥ history/append. """V7: ganti snapshot — tiap skan ⊥ history/append.
WAL: journal_mode=WAL → pembaca (dashboard multi-worker) tak terblokir Delegasi ke `db.replace_snapshot` (backend sqlite/mysql):
saat DROP+INSERT harian. Mode ini persisten setelah diset. sqlite DROP+CREATE dalam WAL → pembaca (dashboard multi-worker) tak
terblokir; mysql DELETE+INSERT dalam satu transaksi (MVCC snapshot).
""" """
scanned_at = datetime.now().isoformat(timespec='seconds') scanned_at = datetime.now().isoformat(timespec='seconds')
conn = sqlite3.connect(DB_PATH) db.replace_snapshot(rows, scanned_at)
try:
conn.execute('PRAGMA journal_mode=WAL')
conn.execute('DROP TABLE IF EXISTS voters') # snapshot bersih
conn.execute(
'CREATE TABLE voters (' # I.db
'owner TEXT PRIMARY KEY, '
'weight TEXT NOT NULL, '
'staked REAL NOT NULL, '
'scanned_at TEXT NOT NULL, '
'last_vote TEXT)'
)
conn.executemany(
'INSERT INTO voters (owner, weight, staked, scanned_at, last_vote) '
'VALUES (?, ?, ?, ?, ?)',
[(r['owner'], r['weight'], r['staked'], scanned_at, r.get('last_vote'))
for r in rows],
)
conn.commit()
finally:
conn.close()
return scanned_at return scanned_at
+1
View File
@@ -2,3 +2,4 @@ requests>=2.31
flask>=3.0 flask>=3.0
python-dotenv>=1.0 python-dotenv>=1.0
gunicorn>=22.0 gunicorn>=22.0
PyMySQL>=1.1
+13 -1
View File
@@ -13,6 +13,7 @@ from datetime import datetime, timedelta, timezone
import requests import requests
import dashboard import dashboard
import db as dbm
def check(tag, cond): def check(tag, cond):
@@ -49,7 +50,7 @@ def main():
rows[5] = (rows[5][0], rows[5][1], 5.0 * 10.0, rows[5][3], fresh) # non-monotonic guard rows[5] = (rows[5][0], rows[5][1], 5.0 * 10.0, rows[5][3], fresh) # non-monotonic guard
make_db(db, rows) make_db(db, rows)
dashboard.DB_PATH = db # arahkan app ke DB uji (read-only tetap dipertahankan) dbm.DB_PATH = db # arahkan app ke DB uji (read-only tetap dipertahankan)
real_get_liquid = dashboard._get_liquid_vex real_get_liquid = dashboard._get_liquid_vex
dashboard._get_liquid_vex = lambda: 50691.4676 # suite utama ⊥ jaringan nyata dashboard._get_liquid_vex = lambda: 50691.4676 # suite utama ⊥ jaringan nyata
client = dashboard.app.test_client() client = dashboard.app.test_client()
@@ -134,6 +135,13 @@ def main():
resp = client.get('/api/search?q=zzzz') resp = client.get('/api/search?q=zzzz')
check('V11 /api/search kosong', resp.get_json()['count'] == 0) check('V11 /api/search kosong', resp.get_json()['count'] == 0)
# wildcard `%`/`_` dari pengguna dibuang (escape `!`, bukan pola):
# tak ada owner berisi `_` atau `%` → 0 hasil bila benar-benar literal
resp = client.get('/api/search?q=acct!_')
check('V17 wildcard `_` di-escape (bukan pola)', resp.get_json()['count'] == 0)
resp = client.get('/api/search?q=acct!%')
check('V17 wildcard `%` di-escape (bukan pola)', resp.get_json()['count'] == 0)
# rank HARUS global, bukan dalam-filter: 'acct00' menyentuh baris stake kecil # rank HARUS global, bukan dalam-filter: 'acct00' menyentuh baris stake kecil
resp = client.get('/api/search?q=acct00') resp = client.get('/api/search?q=acct00')
data = resp.get_json() data = resp.get_json()
@@ -151,6 +159,10 @@ def main():
check('V13 config default VEX_STALE_DAYS', config.VEX_STALE_DAYS == 28) check('V13 config default VEX_STALE_DAYS', config.VEX_STALE_DAYS == 28)
check('V16 config default DATABISNIS_API', config.DATABISNIS_API == 'https://api.databisnis.id') check('V16 config default DATABISNIS_API', config.DATABISNIS_API == 'https://api.databisnis.id')
check('V16 config default DASH_LIQUID_TTL', config.DASH_LIQUID_TTL == 60) check('V16 config default DASH_LIQUID_TTL', config.DASH_LIQUID_TTL == 60)
check('V17 config default DB_BACKEND', config.DB_BACKEND == 'sqlite')
check('V17 config default DB_HOST/PORT', config.DB_HOST == '127.0.0.1' and config.DB_PORT == 3306)
check('V17 config default DB_USER/PASS/NAME',
(config.DB_USER, config.DB_PASS, config.DB_NAME) == ('', '', ''))
# ————— V16: saldo liquid — cache TTL + cooldown kegagalan ————— # ————— V16: saldo liquid — cache TTL + cooldown kegagalan —————
dashboard._get_liquid_vex = real_get_liquid dashboard._get_liquid_vex = real_get_liquid
+4 -3
View File
@@ -13,6 +13,7 @@ from datetime import datetime, timedelta
import requests import requests
import db
import get_voters as gv import get_voters as gv
@@ -69,7 +70,7 @@ def check(tag, cond):
def main(): def main():
tmp = tempfile.mkdtemp() tmp = tempfile.mkdtemp()
gv.DB_PATH = os.path.join(tmp, 'test.db') db.DB_PATH = os.path.join(tmp, 'test.db')
# V1: ulang ≥3x saat node timeout → hits, lalu sukses # V1: ulang ≥3x saat node timeout → hits, lalu sukses
calls = {'n': 0} calls = {'n': 0}
@@ -180,7 +181,7 @@ def main():
{'owner': 'ccc3', 'weight': '222.0', 'staked': 1000.0}, {'owner': 'ccc3', 'weight': '222.0', 'staked': 1000.0},
] ]
gv.persist(set_a) gv.persist(set_a)
conn = sqlite3.connect(gv.DB_PATH) conn = sqlite3.connect(db.DB_PATH)
cnt = conn.execute('SELECT COUNT(*) FROM voters').fetchone()[0] cnt = conn.execute('SELECT COUNT(*) FROM voters').fetchone()[0]
conn.close() conn.close()
check('V7 snapshot pertama = 2', cnt == 2) check('V7 snapshot pertama = 2', cnt == 2)
@@ -190,7 +191,7 @@ def main():
{'owner': 'ddd4', 'weight': '333.0', 'staked': 2000.0}, {'owner': 'ddd4', 'weight': '333.0', 'staked': 2000.0},
] ]
gv.persist(set_b) gv.persist(set_b)
conn = sqlite3.connect(gv.DB_PATH) conn = sqlite3.connect(db.DB_PATH)
rows = conn.execute('SELECT owner, weight, staked, scanned_at FROM voters').fetchall() rows = conn.execute('SELECT owner, weight, staked, scanned_at FROM voters').fetchall()
by = {r[0]: r for r in rows} by = {r[0]: r for r in rows}
cols = [r[1] for r in conn.execute('PRAGMA table_info(voters)').fetchall()] cols = [r[1] for r in conn.execute('PRAGMA table_info(voters)').fetchall()]
+93
View File
@@ -0,0 +1,93 @@
"""Uji integrasi backend MariaDB/MySQL (opt-in, ⊥ dijalankan otomatis).
Oracle utama (test_get_voters.py / test_dashboard.py) tetap pakai sqlite agar
hermetik. Suite ini hanya jalan saat `VEX_DB_BACKEND=mysql` (dengan VEX_DB_*
yang menunjuk container docker-compose):
VEX_DB_BACKEND=mysql VEX_DB_HOST=127.0.0.1 VEX_DB_PORT=3306 \
VEX_DB_USER=databisnis VEX_DB_PASS=databisnis VEX_DB_NAME=databisnisid \
./venv/bin/python test_mariadb.py
Selain itu ia `skip` (exit 0 tanpa menjalankan apa pun). Menyalakan backend
mysql hanya lewat env — jangan pernah menyetel default sqlite ke mysql.
"""
import os
import sys
import get_voters as gv
def check(tag, cond):
if not cond:
raise AssertionError(f'GAGAL: {tag}')
print(f'ok: {tag}')
def main():
if os.getenv('VEX_DB_BACKEND') != 'mysql':
print('skip: VEX_DB_BACKEND != mysql (backend sqlite default, tetap aman)')
return
import db
# ————— snapshot roundtrip: replace_snapshot → query —————
rows_a = [
{'owner': 'aaa1', 'weight': '111.00000000000000000', 'staked': 5000.0, 'last_vote': '2026-07-01T00:00:00'},
{'owner': 'ccc3', 'weight': '222.00000000000000000', 'staked': 3000.0, 'last_vote': '2026-07-02T00:00:00'},
]
scanned_a = gv.persist(rows_a)
got = db.query('SELECT owner, weight, staked, scanned_at, last_vote '
'FROM voters ORDER BY staked DESC')
check('mysql roundtrip: 2 baris sesuai urutan', [r[0] for r in got] == ['aaa1', 'ccc3'])
check('mysql roundtrip: weight verbatim TEXT', got[0][1] == '111.00000000000000000')
check('mysql roundtrip: staked float', isinstance(got[0][2], float) and got[0][2] == 5000.0)
check('mysql roundtrip: scanned_at terisi', got[0][3] == scanned_a)
check('mysql roundtrip: last_vote TEXT', got[0][4] == '2026-07-01T00:00:00')
# ————— snapshot ganti: baris lama hilang, tanpa PK clash —————
rows_b = [
{'owner': 'aaa1', 'weight': '999.00000000000000000', 'staked': 9000.0, 'last_vote': '2026-07-03T00:00:00'},
{'owner': 'ddd4', 'weight': '333.00000000000000000', 'staked': 2000.0, 'last_vote': '2026-07-04T00:00:00'},
]
gv.persist(rows_b)
got = db.query('SELECT owner, weight, staked, last_vote FROM voters ORDER BY owner')
by = {r[0]: r for r in got}
check('mysql ganti snapshot: |set_b|', len(got) == 2)
check('mysql ganti snapshot: ccc3 dihapus', 'ccc3' not in by)
check('mysql ganti snapshot: ddd4 baru', 'ddd4' in by)
check('mysql ganti snapshot: bobot terbaru', by['aaa1'][1] == '999.00000000000000000')
# ————— placeholder `%s` (LIKE) & LIMIT/OFFSET diterjemahkan —————
like = db.query(
"SELECT owner FROM voters WHERE owner LIKE %s ESCAPE '!' "
'ORDER BY owner LIMIT %s OFFSET %s',
['%aa%', 10, 0],
)
check('mysql LIKE %s + LIMIT %s', [r[0] for r in like] == ['aaa1'])
like_none = db.query(
"SELECT owner FROM voters WHERE owner LIKE %s ESCAPE '!'",
['%zzz%'],
)
check('mysql LIKE tak cocok → kosong', like_none == [])
# ————— queryone & COALESCE/agregat —————
cnt = db.queryone('SELECT COUNT(*) FROM voters')[0]
check('mysql COUNT(*)', cnt == 2)
scanned = db.queryone("SELECT COALESCE(MAX(scanned_at), '') FROM voters")[0]
check('mysql MAX(scanned_at) non-empty', isinstance(scanned, str) and scanned != '')
# ————— rank via correlated-count (global, sama semantik sqlite) —————
ranked = db.query(
'SELECT v.owner, (SELECT COUNT(*) FROM voters v2 '
'WHERE v2.staked > v.staked '
' OR (v2.staked = v.staked AND v2.owner <= v.owner)) AS rn '
'FROM voters v ORDER BY v.staked DESC, v.owner ASC'
)
check('mysql rank global', [r[1] for r in ranked] == [1, 2])
print('selesai: semua uji MariaDB lolos')
if __name__ == '__main__':
main()