diff --git a/BUGS.md b/BUGS.md index 0d9d6e1..c1afedf 100644 --- a/BUGS.md +++ b/BUGS.md @@ -166,15 +166,6 @@ winner's own bet is gone the payout is held (`payout_blocked`) and the lottery p ## High -### B-86 — A connected but stalled server is never replaced - -`ping` keeps succeeding while the server stops following the chain; nothing compares -tip progress with the other servers, and a restart goes back to the primary. - -Fix: if the tip has not advanced for 360s and any other server reports a higher tip, -rotate. `GET /health/ready` and `/admin` expose connection state, endpoint and -last-header age (B-92). - ### B-87 — The RBF bump can leave a dust change output `bump_fee` only requires change > fee delta. A 294-sat change (full-balance diff --git a/CLAUDE.md b/CLAUDE.md index 03f3678..d809e4e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ The user communicates in Italian in chat — reply to them in Italian. Everythin ## Project status -All 10 stages of the original build order are code-complete and unit-tested — 576 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. +All 10 stages of the original build order are code-complete and unit-tested — 583 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. Verified on mainnet with real money: registration + address derivation, deposit crediting (1-conf), a real 10 PLM bet (broadcast → confirmed → change credited back), and one full round cycle (close → draw on a real block hash → 70/30 payout with sat math checked against the broadcast tx → confirmation → close → next round auto-opened). **Withdrawal and the RBF bump path have never been exercised against a live broadcast** — unit-tested only. @@ -49,7 +49,7 @@ PYTHONPATH=. python scripts/decrypt_master_key.py # ops recovery: decrypt+pr PYTHONPATH=. python scripts/encrypt_master_key.py # ops bootstrap: import an externally-generated xprv (--overwrite to replace) PYTHONPATH=. python scripts/electrum_smoke_test.py # manual check: connect, handshake, subscribe to headers, print the tip -python -m pytest # all 576 tests +python -m pytest # all 583 tests python -m pytest tests/unit/test_hd.py # one file python -m pytest tests/unit/test_hd.py::test_derivation_is_deterministic # one test ``` @@ -156,7 +156,7 @@ One connection serves everything — deposit credits, broadcasts, confirmations, - **Rotation.** `ELECTRUM_HOST`/`PORT` is primary, `ELECTRUM_FALLBACK_SERVERS` a comma-separated `host:port[:notls][:pin=]` list (`client.py:parse_endpoints` rejects malformed entries at startup, not during the outage when the fallback is needed; `pin=` fixes the server's DER certificate hash, `ELECTRUM_CERT_PIN` does the same for the primary, B-84). The server **refuses to start with fewer than 2 fallbacks** (`config.validate_quorum_servers`, B-84; `ALLOW_NO_QUORUM=true` overrides it, development only). After any failed or dropped session the next server is tried immediately; the backoff (1s doubling to 30s) only kicks in once every server has had a turn. - **Bounded requests** (`_REQUEST_TIMEOUT_SECONDS` = 15s); a timeout tears the connection down. Unbounded waits used to hang `POST /bets` *while holding the per-user lock*, and could stall the confirmation poller permanently. -- **The drop is observable**: `client.wait_closed()` resolves when the read loop dies, and `_run_once` races it against the notification consumers and a 60s `server.ping`. Without it the listener sat on queues nobody would ever fill while `listener.client` still looked alive. +- **The drop is observable**: `client.wait_closed()` resolves when the read loop dies, and `_run_once` races it against the notification consumers and a 60s `server.ping`. A server that keeps answering pings but stopped following the chain is caught too (B-86): at every ping, if no header has been *accepted* for 360s (`_STALL_SECONDS`), `_check_not_stalled` asks the other servers for their tip and ends the session with `ServerStalledError` if any is higher, so `run()` rotates; none higher means a quiet chain and the session goes on. Without it the listener sat on queues nobody would ever fill while `listener.client` still looked alive. - **Headers are validated, not trusted** (`_apply_header`, async since B-74): the tip never regresses, a header must meet the difficulty target it claims *and* that target must be no easier than `draw.POW_LIMIT` (PLM's powLimit; `tests/unit/conftest.py` swaps in a regtest limit so tests can mine headers, and the tests of the limit itself put `MAINNET_POW_LIMIT` back), and every advance must be linked to the tip: a single block by its `prev_hash`, a jump by downloading the headers in between (`blockchain.block.headers`, ≤2016 per request) from the active server and checking each. Failure raises `HeaderValidationError`, which ends the session like a dropped connection and rotates away — that header is the draw's only entropy, so a fabricated one picks the winner. The listener keeps the last 10 headers (`_window`, built from the first header plus its 9 predecessors); a header that does not join the tip is a reorg, walked back to its fork point inside the window and followed (`_follow_reorg`), a fork older than the window ends the session. A followed reorg writes `chain_reorg_detected` *before* the window moves (`_record_reorg`, B-85), and a depth of 2 or more pauses the lottery with `paused_reason = 'reorg'` in the same transaction; a header at the tip's own height with a different hash is a reorg too, not ignored. One more header is refused *without* ending the session (B-64): one carrying no `hex` at all (nothing to validate or draw from, and applying the height alone would break the `tip_height`/`tip_header_hex` pairing). `_run_once` separately refuses to publish the client while *no* tip is known, so the ignore-don't-kill choice can't reopen B-63. - **A quorum corroborates every money-moving decision** (`_corroborate_majority`, 10s per server, asking only the *other* endpoints — never the active one, which is what a MITM controls): `corroborate_header` before a block seeds the draw (B-28), `corroborate_utxo_spent` before a UTXO missing from one `listunspent` is written off as externally spent (B-29), and `corroborate_utxo_credit` before a new outpoint credits a balance — same outpoint, same amount, confirmed (B-59). Balances move in both directions, so both directions need the same quorum — and since B-83 that includes `corroborate_utxo_reinstatement` before a UTXO written off as externally spent is put back (same outpoint, amount, confirmed; `deposits/service.py:find_reinstate_candidates` only reads, `reinstate_reappeared_utxos` writes what the quorum agreed on), and `confirmation_proven` before a tx is marked confirmed: `tx/confirmation.py:poll_once` takes the height the active server reports only as a *claim*, and the row confirms when a strict majority of at least 2 of the *other* servers return a `get_merkle` proof that recomputes to the merkle root of the block's header (`electrum/merkle.py`; the header is the tip we hold, or for an older block the one the quorum agrees on and that meets its own target). With no `network` nothing confirms. The one decision still on the active server's word is the pool-coin listing for a payout (B-82). It needs **at least 2 answers** and a strict majority of those that answered (B-84): fewer → returns **False**, since one voice decides nothing independently and an unreachable network proves nothing. Only `ALLOW_NO_QUORUM=true` keeps the old reduced rule (no other server → True, otherwise a majority of whoever answers). A failed credit corroboration only *delays*: `find_new_credit_candidates` re-offers the outpoint on the next refresh or `DepositReconciler` sweep. diff --git a/README.md b/README.md index 1649aac..3aba92a 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ python -m pytest # all tests python -m pytest tests/unit/test_hd.py # one file ``` -576 unit tests cover HD derivation, PSBT building, the Electrum client, bets, +583 unit tests cover HD derivation, PSBT building, the Electrum client, bets, deposits, withdrawals, the round/draw engine, RBF fee-bumping, admin config, the pending-inclusive balance calculation, and the SSE push channel. No automated integration tests against a live Electrum connection — mainnet diff --git a/app/electrum/listener.py b/app/electrum/listener.py index 39076b3..a560f19 100644 --- a/app/electrum/listener.py +++ b/app/electrum/listener.py @@ -1,5 +1,6 @@ import asyncio import logging +import time from collections import Counter from collections.abc import Callable @@ -55,6 +56,14 @@ _RESUBSCRIBE_CONCURRENCY = 20 # a hostile server. Also the deepest fork it will follow. _HEADER_WINDOW = 10 +# B-86 (ELC-21): a connected server whose last accepted header is older than this is asked +# about by comparing its tip with the other servers' at every ping. Three block times of +# 120s: late enough that an ordinary slow stretch of the chain does not trigger the check. +_STALL_SECONDS = 360 + +# The clock the stall check reads — a module attribute so a test can move time. +_monotonic = time.monotonic + # A reorganisation at least this deep pauses the lottery (B-85, ELC-24): one block is # ordinary on any chain, two or more is worth a look before a new round opens. _REORG_PAUSE_DEPTH = 2 @@ -63,6 +72,11 @@ _REORG_PAUSE_DEPTH = 2 _MAX_HEADERS_PER_REQUEST = 2016 +class ServerStalledError(Exception): + """The active server stopped following the chain while others report a higher tip + (B-86, ELC-21). Ends the session like a dropped connection, so run() rotates.""" + + class ElectrumListener: """Long-lived background task: keeps one Electrum connection open, subscribes every user's address (plus any address added later via add_address), and @@ -103,6 +117,8 @@ class ElectrumListener: # B-74 (ELC-22): height -> header of the last _HEADER_WINDOW accepted headers, tip # included. Survives reconnections, not restarts. self._window: dict[int, str] = {} + # B-86: when the last header was accepted (monotonic seconds), None until one is. + self._last_header_at: float | None = None self.client: ElectrumClient | None = None @property @@ -266,6 +282,29 @@ class ElectrumListener: while True: await asyncio.sleep(_PING_INTERVAL_SECONDS) await client.ping() # raises (and so ends the session) on timeout or a dead socket + await self._check_not_stalled() + + async def _check_not_stalled(self) -> None: + """B-86 (ELC-21): `server.ping` keeps succeeding on a server that has stopped + following the chain, so liveness alone never replaced one. When the last accepted + header is older than _STALL_SECONDS, ask every *other* server for its tip + (`blockchain.headers.subscribe`, with ELC-14's timeouts): if any reports a higher one + than we know, this session ends and the listener moves to the next server. If none + does, the network simply produced no block and the session goes on.""" + if self._last_header_at is None or _monotonic() - self._last_header_at <= _STALL_SECONDS: + return + + async def _tip_height(client: ElectrumClient) -> int | None: + header = await client.subscribe_headers() + height = header.get("height") if isinstance(header, dict) else None + return height if isinstance(height, int) else None + + heights = [h for h in await self._ask_other_servers(_tip_height) if h is not None] + if any(height > self.tip_height for height in heights): + raise ServerStalledError( + f"no header accepted for {_monotonic() - self._last_header_at:.0f}s while another server " + f"reports tip {max(heights)} (ours: {self.tip_height})" + ) async def _subscribe_all_users(self) -> None: """B-31: subscribes with bounded concurrency (_RESUBSCRIBE_CONCURRENCY at @@ -422,6 +461,7 @@ class ElectrumListener: for old in sorted(self._window)[:-_HEADER_WINDOW]: del self._window[old] self.tip_height, self.tip_header_hex = entries[-1] + self._last_header_at = _monotonic() async def _adopt_first_header(self, height: int, header_hex: str, client: ElectrumClient | None) -> None: """The process's first header has nothing to join, so it is taken on its work alone diff --git a/docs/architettura.md b/docs/architettura.md index e7316b1..e928069 100644 --- a/docs/architettura.md +++ b/docs/architettura.md @@ -537,10 +537,9 @@ implementare» non esistono ancora nel codice (vedi [ARC-60] e [ARC-22]). *Codice:* `app/electrum/listener.py:ElectrumListener.run` · `app/electrum/listener.py:ElectrumListener._apply_header` · + `app/electrum/listener.py:ElectrumListener._check_not_stalled` · `app/electrum/client.py:ElectrumClient.connect` · *Test:* `tests/unit/test_electrum_listener.py`, `tests/unit/test_electrum_client.py` - *Stato:* da implementare (B-86) · *Oggi:* il listener non confronta la - propria punta con gli altri server. - **[ARC-27]** **`RoundScheduler`.** A ogni giro (5s) chiama `open_new_round_if_needed` e poi avanza il round attivo, in qualunque stato sia, `drawing` compreso: per un round in `drawing` il giro rientra diff --git a/docs/connessione-electrum.md b/docs/connessione-electrum.md index 0066fdb..1416ca8 100644 --- a/docs/connessione-electrum.md +++ b/docs/connessione-electrum.md @@ -120,10 +120,8 @@ Le difese sono cinque. quella nota, la sessione termina e si passa al server successivo nell'elenco ([ELC-02]). Se nessuno riporta un'altezza maggiore (la rete non ha prodotto blocchi) la sessione continua. - *Codice:* `app/electrum/listener.py:ElectrumListener._keepalive` · *Test:* — - *Stato:* da implementare (B-86) · *Oggi:* nulla confronta l'avanzamento - della punta con gli altri server; un server fermo che risponde ai ping - resta il server attivo indefinitamente. + *Codice:* `app/electrum/listener.py:ElectrumListener._keepalive`, `app/electrum/listener.py:ElectrumListener._check_not_stalled` · + *Test:* `tests/unit/test_electrum_listener.py` ## 4. Header validati, non creduti diff --git a/docs/storico-bug.md b/docs/storico-bug.md index 05a07a5..abb7065 100644 --- a/docs/storico-bug.md +++ b/docs/storico-bug.md @@ -234,7 +234,7 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)). | B-83 | Conferme e reintegri si basano solo sul server attivo | Alta | *da registrare* | `test_confirmation_proof.py`, `test_confirmation.py` | ELC, TX, DEP | | B-84 | Il quorum può ridursi a un solo server o a nessuno senza avvisi; certificati TLS non verificati | Alta | *da registrare* | `test_electrum_client.py`, `test_electrum_listener.py` | ELC, SEC | | B-85 | Le riorganizzazioni della catena non vengono né seguite né segnalate | Alta | *da registrare* | `test_electrum_listener.py` | ELC, RND | -| B-86 | Un server connesso ma fermo non viene mai sostituito | Alta | **aperto** | — | ELC, OPS | +| B-86 | Un server connesso ma fermo non viene mai sostituito | Alta | *da registrare* | `test_electrum_listener.py` | ELC, OPS | | B-87 | L'RBF può lasciare un resto sotto la soglia di dust | Alta | **aperto** | — | TX | | B-88 | Le transazioni SQLite non sono quelle che il codice presuppone; `synchronous=NORMAL` può perdere gli ultimi commit | Alta | *da registrare* | `test_sqlite_transactions.py`, `test_db_base.py` | DAT, ARC | | B-89 | Problemi della chiave rilevati solo al primo uso; gli script accettano qualunque xprv e scrivono file leggibili da tutti | Alta | **aperto** | — | SEC, OPS | @@ -252,9 +252,9 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)). - **Corretti:** B-01 … B-70, tranne B-16, che è chiuso nella sostanza (il crash è eliminato e `/guida` serve un segnaposto) ma lascia la guida vera - come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76, B-77, B-78, B-83, B-74, B-85. + come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76, B-77, B-78, B-83, B-74, B-85, B-86. - **Aperti:** B-71, B-72 (gravità bassa, operativi o di documentazione) e B-73 … B-98 dalla revisione del 2026-09-28 tranne i corretti (4 critici, - 5 alti, 6 medi), da implementare nell'ordine di + 4 alti, 6 medi), da implementare nell'ordine di [piano-correzioni.md](piano-correzioni.md). - **Prossimo numero libero:** `B-99`. diff --git a/tests/unit/test_electrum_listener.py b/tests/unit/test_electrum_listener.py index ab4c332..7a4c29d 100644 --- a/tests/unit/test_electrum_listener.py +++ b/tests/unit/test_electrum_listener.py @@ -1254,3 +1254,138 @@ async def test_the_window_does_not_move_when_the_reorg_cannot_be_recorded(sessio await listener._apply_header({"height": 101, "hex": client.chain[101]}, client) assert listener.tip_height == 101 assert len(await _reorg_events(session_factory)) == 1 + + +# --- B-86 (ELC-21): a connected server that stopped following the chain is replaced ------- + + +class _TipClient: + """What a *other* server answers to a headers subscription: its tip.""" + + def __init__(self, height: int | None, fail: bool = False): + self._height = height + self._fail = fail + + async def connect(self): + if self._fail: + raise OSError("unreachable") + + async def subscribe_headers(self): + return {"height": self._height, "hex": "00"} + + async def close(self): + pass + + +def _stall_listener(session_factory, monkeypatch, other_heights: dict[str, int | None]): + """A listener on `_ENDPOINTS[0]` with a tip at 100 accepted at t=0 and a movable clock; + the other two servers answer `other_heights` (None = unreachable).""" + from app.electrum import listener as listener_module + + clock = {"now": 0.0} + monkeypatch.setattr(listener_module, "_monotonic", lambda: clock["now"]) + + def factory(endpoint): + height = other_heights[endpoint.host] + return _TipClient(height, fail=height is None) + + listener = ElectrumListener(factory, session_factory, _ENDPOINTS) + listener._extend_window([(100, _mine_chain(100, 1)[100])]) + return listener, clock + + +async def test_a_stalled_server_is_replaced_when_another_reports_a_higher_tip(session_factory, monkeypatch): + """B-86, ELC-21: the active server keeps answering pings but its last accepted header is + older than 360s while another server is already at 101 — the session must end so run() + rotates. Nothing used to compare the tips, so a stalled server stayed active for ever.""" + from app.electrum.listener import ServerStalledError + + listener, clock = _stall_listener( + session_factory, monkeypatch, {"first.example": None, "second.example": 101, "third.example": 100} + ) + clock["now"] = 361.0 + with pytest.raises(ServerStalledError): + await listener._check_not_stalled() + + +async def test_a_quiet_chain_does_not_rotate_the_session(session_factory, monkeypatch): + """ELC-21: no block for 360s but nobody reports a higher tip — the network produced + nothing, and the session goes on.""" + listener, clock = _stall_listener( + session_factory, monkeypatch, {"first.example": None, "second.example": 100, "third.example": 99} + ) + clock["now"] = 10_000.0 + await listener._check_not_stalled() # must not raise + + +async def test_the_other_servers_are_not_asked_before_360_seconds(session_factory, monkeypatch): + """ELC-21: the check costs a connection to every other server, so it only starts once + the last header is older than 360s — exactly 360 is still fine.""" + listener, clock = _stall_listener( + session_factory, monkeypatch, {"first.example": None, "second.example": 500, "third.example": 500} + ) + clock["now"] = 360.0 + await listener._check_not_stalled() # must not raise although others are ahead + + +async def test_an_unreachable_other_server_does_not_count_as_ahead(session_factory, monkeypatch): + listener, clock = _stall_listener( + session_factory, monkeypatch, {"first.example": None, "second.example": None, "third.example": None} + ) + clock["now"] = 1_000.0 + await listener._check_not_stalled() # no answers, nothing to rotate to + + +async def test_an_accepted_header_restarts_the_stall_clock(session_factory, monkeypatch): + """ELC-21: it is the age of the last *accepted* header that counts.""" + listener, clock = _stall_listener( + session_factory, monkeypatch, {"first.example": None, "second.example": 500, "third.example": 500} + ) + clock["now"] = 400.0 + listener._extend_window([(101, _mine_chain(101, 1)[101])]) + clock["now"] = 600.0 # 200s since the header at 101 + await listener._check_not_stalled() + + +async def test_the_keepalive_ends_the_session_of_a_stalled_server(session_factory, monkeypatch): + """B-86: wired into the ping loop — after a successful ping the stall check runs, and its + error is what ends the session (`_run_once` then returns and `run` rotates).""" + from app.electrum import listener as listener_module + from app.electrum.listener import ServerStalledError + + monkeypatch.setattr(listener_module, "_PING_INTERVAL_SECONDS", 0) + listener, clock = _stall_listener( + session_factory, monkeypatch, {"first.example": None, "second.example": 101, "third.example": 100} + ) + clock["now"] = 400.0 + pinged = [] + + class _Active: + async def ping(self): + pinged.append(True) + + with pytest.raises(ServerStalledError): + await asyncio.wait_for(listener._keepalive(_Active()), timeout=2) + assert pinged == [True] + + +async def test_a_run_once_session_ends_when_the_active_server_stalls(session_factory, monkeypatch): + """B-86, end to end through _run_once: connected, header accepted at t=0, then the clock + passes 360s with another server ahead — the session returns by itself.""" + from app.electrum import listener as listener_module + + monkeypatch.setattr(listener_module, "_PING_INTERVAL_SECONDS", 0.01) + clock = {"now": 0.0} + monkeypatch.setattr(listener_module, "_monotonic", lambda: clock["now"]) + chain = _mine_chain(91, 10) + active = _FakeConnectClient({"height": 100, "hex": chain[100]}, chain) + + def factory(endpoint): + return active if endpoint == _ENDPOINTS[0] else _TipClient(101) + + listener = ElectrumListener(factory, session_factory, _ENDPOINTS) + task = asyncio.create_task(listener._run_once(_ENDPOINTS[0])) + await _wait_until(lambda: listener.client is not None) + clock["now"] = 361.0 + assert await asyncio.wait_for(task, timeout=3) is True + assert listener.client is None