diff --git a/CLAUDE.md b/CLAUDE.md index cfd9eef..5c7be6d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ The user communicates in Italian in chat — reply to them in Italian. Everythin ## Project status -The migration from FastAPI to Django is in progress on branch `django-migration` (plan: [roadmap.md](roadmap.md), steps A1–A10, B1–B2 done). The suite has 1181 tests: the old stack's under `tests/unit/`, the Django ones under `tests/django/` (`tests/integration/` is an empty package). Open defects of the old stack are in [BUGS.md](BUGS.md); accepted limits in [docs/limiti-noti.md](docs/limiti-noti.md); every closed `B-nn` is in [docs/storico-bug.md](docs/storico-bug.md) (`git log --all --grep 'B-nn'` finds its commit). +The migration from FastAPI to Django is in progress on branch `django-migration` (plan: [roadmap.md](roadmap.md), steps A1–A10, B1–B3 done). The suite has 1187 tests: the old stack's under `tests/unit/`, the Django ones under `tests/django/` (`tests/integration/` needs the local regtest node and skips itself without it). Open defects of the old stack are in [BUGS.md](BUGS.md); accepted limits in [docs/limiti-noti.md](docs/limiti-noti.md); every closed `B-nn` is in [docs/storico-bug.md](docs/storico-bug.md) (`git log --all --grep 'B-nn'` finds its commit). **[docs/](docs/README.md) is the specification, and it is authoritative.** Every verifiable rule there has a stable requirement ID (`RND-04`, `BET-02`, …) with `*Codice:*`/`*Test:*` pointers to what implements it; the index is [docs/come-funziona.md](docs/come-funziona.md) and the ID convention and change workflow are in [docs/processo-di-modifica.md](docs/processo-di-modifica.md). A behaviour change starts in the spec: edit or add the requirement, implement it, cite the ID in the test docstring and the commit message, and ship spec + code + test in one commit. If this file and the spec disagree, the spec wins and this file is the one to fix. Before writing code, read the relevant spec documents and the "Architecture" section below. diff --git a/README.md b/README.md index ec8a965..c438ea2 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ python -m pytest # all tests python -m pytest tests/unit/test_hd.py # one file ``` -1181 unit tests cover HD derivation, PSBT building, the Electrum client, bets, +1187 unit tests cover HD derivation, PSBT building, the Electrum client, bets, deposits, withdrawals, the round/draw engine, RBF fee-bumping, admin config, the pending-inclusive balance calculation, and the SSE push channel. No automated integration tests against a live Electrum connection — mainnet diff --git a/docs/legacy-fastapi-notes.md b/docs/legacy-fastapi-notes.md index bb6aaf8..34c0555 100644 --- a/docs/legacy-fastapi-notes.md +++ b/docs/legacy-fastapi-notes.md @@ -4,7 +4,7 @@ ## Project status (old stack) -All 10 stages of the original build order are code-complete and unit-tested — 1181 tests, the old stack's under `tests/unit/` and the Django ones under `tests/django/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. +All 10 stages of the original build order are code-complete and unit-tested — 1187 tests, the old stack's under `tests/unit/` and the Django ones under `tests/django/` (`tests/integration/` needs the local regtest node and skips itself without it). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. Verified on mainnet with real money: registration + address derivation, deposit crediting (at the time, 1-conf; deposits now need `deposit_confirmations`), a real 10 PLM bet (broadcast → confirmed → change credited back), and one full round cycle (close → draw on a real block hash → 70/30 payout with sat math checked against the broadcast tx → confirmation → close → next round auto-opened). **Withdrawal and the RBF bump path have never been exercised against a live broadcast** — unit-tested only. @@ -30,7 +30,7 @@ PYTHONPATH=. python scripts/decrypt_master_key.py # ops recovery: decrypt+pr PYTHONPATH=. python scripts/encrypt_master_key.py # ops bootstrap: import an externally-generated xprv (--overwrite to replace) PYTHONPATH=. python scripts/electrum_smoke_test.py # manual check: connect, handshake, subscribe to headers, print the tip -python -m pytest # all 1181 tests +python -m pytest # all 1187 tests python -m pytest tests/unit/test_hd.py # one file python -m pytest tests/unit/test_hd.py::test_derivation_is_deterministic # one test ``` @@ -253,6 +253,6 @@ Accepted **by design** — distinct from the audit findings above (all fixed), w - **Admin auth is one shared bearer token** (`ADMIN_TOKEN`) with no per-admin identity: `audit_log` records *what* changed (config edits as `config_updated`, with before/after) but never *who* did it. It gates the user list, privkey export, password resets and history, so a leak is high-blast-radius. - **No rate limiting on bet, withdrawal, admin or SSE.** Login has a per-username + per-IP failure throttle and registration a per-IP quota of 5 accounts/hour (`app/auth/rate_limit.py`: `RateLimiter` for failed guesses at a secret, `RollingQuota` for "how many of these may one source create" — B-33, B-58); everything else is unlimited. - **`/guida` is a placeholder** (`app/static/guida.html`) — the link works, the content is "coming soon". `/report-bug` is *not*: it is fully implemented and translated, with admin triage (see Frontends). -- **No integration tests against a live Electrum connection.** `tests/integration/` is empty; live verification has all been manual (`scripts/electrum_smoke_test.py`, ad hoc scripts, real mainnet txs). +- **No integration tests against a live Electrum connection.** the old stack has none (since B3 `tests/integration/` holds smoke tests against a local regtest, not against the old code); live verification has all been manual (`scripts/electrum_smoke_test.py`, ad hoc scripts, real mainnet txs). - **Single-process assumptions**: the SSE broadcaster and the per-user locks are in-process only. A multi-worker deployment needs a shared channel and a DB/Redis lock. The round-uniqueness invariant is *not* in this category — it's a DB index. diff --git a/docs/setup.md b/docs/setup.md index 0c6d60d..82941bf 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -23,6 +23,19 @@ docker compose up -d --build 5. Verifica a mano su telefono e desktop: [checklist-verifica-sito.md](checklist-verifica-sito.md). Test automatici: `python -m pytest`. 6. Spegnere: `docker compose down` (i dati restano in `data/db`). +### Regtest per i test di integrazione (da B3) + +Nodo e Fulcrum in regtest stanno nella cartella `plm-regtest` accanto al repository (`docker compose up -d` lì); il sito non li avvia. I test sotto `tests/integration/` si agganciano a loro e **si saltano** se non rispondono. Per default cercano l'RPC su `http://127.0.0.1:18332` (credenziali da `plm-regtest/node/palladium.conf`) e Fulcrum su `127.0.0.1:50001`; si cambiano con `PLM_REGTEST_RPC_URL`, `PLM_REGTEST_RPC_USER`, `PLM_REGTEST_RPC_PASSWORD`, `PLM_REGTEST_CONF` e `PLM_REGTEST_ELECTRUM`. + +```bash +python scripts/regtest.py status # catena, altezza, saldo del portafoglio del nodo, Fulcrum +python scripts/regtest.py mine 10 # mina 10 blocchi +python scripts/regtest.py fund rplm1… 5 # manda 5 PLM a un indirizzo e mina un blocco +python -m pytest tests/integration # i test sulla catena vera +``` + +Lo script e i test rifiutano di agire su una catena che non sia regtest. + --- > **Nota (migrazione a Django).** Il resto di questa guida descrive il vecchio stack FastAPI, non più nel `docker-compose.yml`. diff --git a/plm_engine/regtest.py b/plm_engine/regtest.py new file mode 100644 index 0000000..4b7998e --- /dev/null +++ b/plm_engine/regtest.py @@ -0,0 +1,117 @@ +"""A handle on a local regtest node, for tests and the dev scripts (ADR-58, roadmap B3). Never used by the site. + +The node is the one of `plm-regtest` (palladiumcore in `-regtest`, with plm-fulcrum in front of it). Nothing here +starts it: bring it up with `docker compose up -d` in that folder. Where to find it is configuration, all optional: + +* `PLM_REGTEST_RPC_URL` (default `http://127.0.0.1:18332`), with `PLM_REGTEST_RPC_USER` / `PLM_REGTEST_RPC_PASSWORD`, + or else the `rpcuser`/`rpcpassword` of `PLM_REGTEST_CONF` (default `../plm-regtest/node/palladium.conf`); +* `PLM_REGTEST_ELECTRUM` as `host:port` of the plain-TCP Fulcrum port (default `127.0.0.1:50001`). +""" + +import base64 +import json +import os +import socket +import urllib.error +import urllib.request +from dataclasses import dataclass +from pathlib import Path + +DEFAULT_RPC_URL = "http://127.0.0.1:18332" +DEFAULT_ELECTRUM = "127.0.0.1:50001" +DEFAULT_CONF = Path(__file__).resolve().parents[2] / "plm-regtest" / "node" / "palladium.conf" +COINBASE_MATURITY = 100 + + +class RegtestError(Exception): + pass + + +def _conf_credentials(path: Path) -> tuple[str, str] | None: + try: + lines = path.read_text().splitlines() + except OSError: + return None + values = dict(line.split("=", 1) for line in lines if "=" in line and not line.lstrip().startswith("#")) + if "rpcuser" in values and "rpcpassword" in values: + return values["rpcuser"].strip(), values["rpcpassword"].strip() + return None + + +@dataclass +class RegtestNode: + url: str + user: str + password: str + + @classmethod + def from_environment(cls) -> "RegtestNode": + url = os.environ.get("PLM_REGTEST_RPC_URL", DEFAULT_RPC_URL) + user, password = os.environ.get("PLM_REGTEST_RPC_USER"), os.environ.get("PLM_REGTEST_RPC_PASSWORD") + if not (user and password): + found = _conf_credentials(Path(os.environ.get("PLM_REGTEST_CONF", DEFAULT_CONF))) + if found is None: + raise RegtestError("no RPC credentials: set PLM_REGTEST_RPC_USER and PLM_REGTEST_RPC_PASSWORD") + user, password = found + return cls(url, user, password) + + def rpc(self, method: str, *params): + body = json.dumps({"jsonrpc": "1.0", "id": "plm", "method": method, "params": list(params)}).encode() + token = base64.b64encode(f"{self.user}:{self.password}".encode()).decode() + request = urllib.request.Request(self.url, body, {"Content-Type": "text/plain", "Authorization": f"Basic {token}"}) + try: + with urllib.request.urlopen(request, timeout=30) as reply: + payload = json.load(reply) + except urllib.error.HTTPError as error: # bitcoind answers 500 with a JSON error body + payload = json.load(error) + except (urllib.error.URLError, OSError) as error: + raise RegtestError(f"the regtest node is not reachable at {self.url}: {error}") from error + if payload.get("error"): + raise RegtestError(f"{method}: {payload['error']}") + return payload["result"] + + # -- what a test needs ------------------------------------------------------------------- + + def info(self) -> dict: + info = self.rpc("getblockchaininfo") + if info["chain"] != "regtest": + raise RegtestError(f"this node is on {info['chain']!r}, not regtest: refusing to touch it") + return info + + def height(self) -> int: + return self.info()["blocks"] + + def mine(self, blocks: int = 1, address: str | None = None) -> list[str]: + """Mine `blocks` blocks (to `address`, or to a fresh address of the node's wallet) and return their hashes.""" + self.info() + return self.rpc("generatetoaddress", blocks, address or self.rpc("getnewaddress")) + + def balance(self) -> float: + return float(self.rpc("getbalance")) + + def ensure_funds(self, plm: float) -> None: + """Make sure the node's wallet can spend `plm`, mining past the coinbase maturity if it cannot.""" + self.info() + while self.balance() < plm: + self.mine(COINBASE_MATURITY + 1) + + def fund(self, address: str, plm: float, confirm: bool = True) -> str: + """Send `plm` to `address` from the node's wallet; with `confirm`, mine one block on top. Returns the txid.""" + self.ensure_funds(plm + 1) + txid = self.rpc("sendtoaddress", address, plm) + if confirm: + self.mine(1) + return txid + + +def electrum_address() -> tuple[str, int]: + host, _, port = os.environ.get("PLM_REGTEST_ELECTRUM", DEFAULT_ELECTRUM).rpartition(":") + return host, int(port) + + +def electrum_reachable(timeout: float = 2.0) -> bool: + try: + with socket.create_connection(electrum_address(), timeout=timeout): + return True + except OSError: + return False diff --git a/roadmap.md b/roadmap.md index 448ec76..013d154 100644 --- a/roadmap.md +++ b/roadmap.md @@ -41,7 +41,7 @@ Uscita: flusso completo su regtest; un riavvio durante il broadcast, due richies - [x] **B1 · Pacchetto `plm_engine`** (le primitive dell'estrazione sono già state spostate in A4). Spostamento dei moduli senza database (derivazione HD, `psbt_builder`, client Electrum, `merkle`, `draw`) e dei loro test, senza cambiare il comportamento. *Requisiti:* ADR-59. *Test:* i test esistenti dei moduli passano invariati. - [x] **B2 · Profilo di rete.** `PLM_NETWORK=regtest|mainnet` (hrp, prefissi, genesis, powLimit), conferma esplicita per `mainnet`, validazione degli indirizzi con l'hrp del profilo. *Requisiti:* ADR-58, ELC-30. *Test:* indirizzi e genesis per ciascun profilo. -- [ ] **B3 · Ambiente regtest e fixture.** Compose con nodo e Fulcrum regtest (profilo `regtest`), script che genera blocchi e fondi, fixture di integrazione che si salta se il regtest non c'è. *Test:* smoke di connessione e generazione di un blocco. +- [x] **B3 · Ambiente regtest e fixture.** Il nodo (`palladiumcore` in `-regtest`) e Fulcrum sono quelli della cartella `plm-regtest` accanto al repository (`docker compose up -d` lì): non si duplicano nel compose del sito, ci si aggancia. `plm_engine/regtest.py` (client RPC con rifiuto di toccare una catena diversa da regtest), `scripts/regtest.py` (`status`, `mine`, `fund`) e le fixture di `tests/integration/`, che si saltano se il nodo o Fulcrum non rispondono. *Test:* `tests/integration/test_regtest_smoke.py` (blocco minato, indirizzi e genesis del profilo, tip di Fulcrum, fondi visti via Electrum). - [ ] **B4 · Modelli di catena.** UTXO, transazioni con tutte le versioni TXID, depositi, assegnazione dell'indice HD; la chiave di firma è accessibile solo al processo worker. *Requisiti:* DAT, TX. *Test:* vincoli, derivazione identica a parità di input. - [ ] **B5 · Listener e depositi reali.** Listener Electrum sull'ORM con rotazione, header validati, quorum e soglie di conferma al posto del deposito simulato. *Requisiti:* DEP, ELC. *Test:* unit con server finto + integrazione su regtest. - [ ] **B6 · Prenotazione UTXO e puntata reale.** Prenotazione atomica degli UTXO nel database, identificativo idempotente, «salva operazione e transazione firmata prima del broadcast», verdetto del broadcast (accettato, rifiutato, incerto). *Requisiti:* BET, TX-26. *Test:* due richieste concorrenti non spendono gli stessi input; timeout ambiguo riconciliato. diff --git a/scripts/regtest.py b/scripts/regtest.py new file mode 100644 index 0000000..26eda16 --- /dev/null +++ b/scripts/regtest.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Drive the local regtest node (ADR-58, roadmap B3): `status`, `mine [N]`, `fund ADDRESS PLM`. + +The node is started separately (`docker compose up -d` in plm-regtest); see plm_engine/regtest.py for where +it is looked for. It refuses to act on any chain that is not regtest. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from plm_engine.regtest import RegtestError, RegtestNode, electrum_address, electrum_reachable # noqa: E402 + + +def main(argv: list[str]) -> int: + command, args = (argv[1] if len(argv) > 1 else "status"), argv[2:] + try: + node = RegtestNode.from_environment() + if command == "status": + info = node.info() + host, port = electrum_address() + print(f"chain {info['chain']}, height {info['blocks']}, wallet balance {node.balance()} PLM") + print(f"electrum {host}:{port}: {'reachable' if electrum_reachable() else 'NOT reachable'}") + elif command == "mine": + hashes = node.mine(int(args[0]) if args else 1) + print(f"mined {len(hashes)} block(s), height now {node.height()}") + elif command == "fund" and len(args) == 2: + print(f"sent {args[1]} PLM, txid {node.fund(args[0], float(args[1]))}") + else: + print(__doc__) + return 2 + except RegtestError as error: + print(f"error: {error}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py new file mode 100644 index 0000000..40fadb2 --- /dev/null +++ b/tests/integration/conftest.py @@ -0,0 +1,52 @@ +"""Integration fixtures against the local regtest (ADR-58, roadmap B3): the node of `plm-regtest` with Fulcrum. + +Nothing here starts them. When the node or Fulcrum is not reachable, every test that asks for them is skipped, so +the suite stays green on a machine without Docker. Bring them up with `docker compose up -d` in `plm-regtest`; +`plm_engine/regtest.py` says how to point at another place. +""" + +import pytest +from embit.bip32 import HDKey + +from plm_engine import hd +from plm_engine.electrum.client import ElectrumClient +from plm_engine.network import REGTEST +from plm_engine.regtest import RegtestError, RegtestNode, electrum_address, electrum_reachable + + +@pytest.fixture(scope="session") +def regtest_node() -> RegtestNode: + try: + node = RegtestNode.from_environment() + node.info() + except RegtestError as error: + pytest.skip(f"no regtest node: {error}") + return node + + +@pytest.fixture(scope="session") +def electrum_endpoint(regtest_node) -> tuple[str, int]: + if not electrum_reachable(): + pytest.skip("no Fulcrum on the regtest Electrum port") + return electrum_address() + + +@pytest.fixture +async def electrum(electrum_endpoint): + """A connected client that checks the regtest genesis at the handshake (ELC-30).""" + host, port = electrum_endpoint + client = ElectrumClient(host, port, use_ssl=False, genesis_hash=REGTEST.genesis_hash) + await client.connect() + try: + yield client + finally: + await client.close() + + +@pytest.fixture +def fresh_address(): + """A regtest address nobody has used: the account of a random master key, user index 0.""" + import os + + master = HDKey.from_seed(os.urandom(32), version=REGTEST.embit["xprv"]) + return hd.user_address(hd.account_key(master, REGTEST), 0, REGTEST) diff --git a/tests/integration/test_regtest_smoke.py b/tests/integration/test_regtest_smoke.py new file mode 100644 index 0000000..f2171a0 --- /dev/null +++ b/tests/integration/test_regtest_smoke.py @@ -0,0 +1,45 @@ +"""B3 — smoke of the local regtest: the node, Fulcrum and our profile agree (ADR-58, ELC-30).""" + +import asyncio + +from plm_engine.electrum.scripthash import address_to_scripthash +from plm_engine.network import REGTEST + + +def test_the_node_is_on_regtest_and_mines_a_block(regtest_node): + before = regtest_node.height() + hashes = regtest_node.mine(1) + assert len(hashes) == 1 and regtest_node.height() == before + 1 + + +def test_the_node_accepts_our_regtest_addresses(regtest_node, fresh_address): + assert fresh_address.startswith("rplm1") + assert regtest_node.rpc("validateaddress", fresh_address)["isvalid"] is True + + +def test_the_node_has_the_genesis_of_our_profile(regtest_node): + assert regtest_node.rpc("getblockhash", 0) == REGTEST.genesis_hash + + +async def test_fulcrum_follows_the_node_tip(regtest_node, electrum): + """Fulcrum indexes asynchronously: give it a moment after a new block.""" + regtest_node.mine(1) + expected = regtest_node.height() + for _ in range(40): + tip = (await electrum.subscribe_headers())["height"] + if tip == expected: + break + await asyncio.sleep(0.25) + assert tip == expected + + +async def test_funds_sent_to_an_address_are_seen_through_electrum(regtest_node, electrum, fresh_address): + regtest_node.fund(fresh_address, 1.5) + scripthash = address_to_scripthash(fresh_address) + utxos = [] + for _ in range(40): + utxos = await electrum.listunspent(scripthash) + if utxos: + break + await asyncio.sleep(0.25) + assert [u["value"] for u in utxos] == [150_000_000]