Rotate away from a connected server that stopped following the chain (B-86, ELC-21, ARC-26)

`server.ping` keeps succeeding on a server that is stalled, and nothing compared its tip
with the others', so it stayed the active server for ever.

- The listener records when the last header was accepted (monotonic clock, a module
  attribute so tests can move it).
- After every successful ping, ElectrumListener._check_not_stalled: once the last accepted
  header is older than 360s the other servers are asked for their tip
  (blockchain.headers.subscribe, ELC-14's timeouts); if any reports a higher height than
  ours the session ends with ServerStalledError and run() rotates to the next server. If
  none does the chain is just quiet and the session goes on.

No existing test changed. ELC-29 and /admin/electrum (the connection state and the age of
the last header, B-92) are untouched.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

BUGS.md

@@ -166,15 +166,6 @@ winner's own bet is gone the payout is held (`payout_blocked`) and the lottery p
 
 ## High
 
-### B-86 — A connected but stalled server is never replaced
-
-`ping` keeps succeeding while the server stops following the chain; nothing compares
-tip progress with the other servers, and a restart goes back to the primary.
-
-Fix: if the tip has not advanced for 360s and any other server reports a higher tip,
-rotate. `GET /health/ready` and `/admin` expose connection state, endpoint and
-last-header age (B-92).
-
 ### B-87 — The RBF bump can leave a dust change output
 
 `bump_fee` only requires change > fee delta. A 294-sat change (full-balance

CLAUDE.md

@@ -8,7 +8,7 @@ The user communicates in Italian in chat — reply to them in Italian. Everythin
 
 ## Project status
 
-All 10 stages of the original build order are code-complete and unit-tested — 576 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below.
+All 10 stages of the original build order are code-complete and unit-tested — 583 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below.
 
 Verified on mainnet with real money: registration + address derivation, deposit crediting (1-conf), a real 10 PLM bet (broadcast → confirmed → change credited back), and one full round cycle (close → draw on a real block hash → 70/30 payout with sat math checked against the broadcast tx → confirmation → close → next round auto-opened). **Withdrawal and the RBF bump path have never been exercised against a live broadcast** — unit-tested only.
 
@@ -49,7 +49,7 @@ PYTHONPATH=. python scripts/decrypt_master_key.py     # ops recovery: decrypt+pr
 PYTHONPATH=. python scripts/encrypt_master_key.py     # ops bootstrap: import an externally-generated xprv (--overwrite to replace)
 PYTHONPATH=. python scripts/electrum_smoke_test.py    # manual check: connect, handshake, subscribe to headers, print the tip
 
-python -m pytest                                      # all 576 tests
+python -m pytest                                      # all 583 tests
 python -m pytest tests/unit/test_hd.py                # one file
 python -m pytest tests/unit/test_hd.py::test_derivation_is_deterministic   # one test
 ```
@@ -156,7 +156,7 @@ One connection serves everything — deposit credits, broadcasts, confirmations,
 
 - **Rotation.** `ELECTRUM_HOST`/`PORT` is primary, `ELECTRUM_FALLBACK_SERVERS` a comma-separated `host:port[:notls][:pin=<sha256>]` list (`client.py:parse_endpoints` rejects malformed entries at startup, not during the outage when the fallback is needed; `pin=` fixes the server's DER certificate hash, `ELECTRUM_CERT_PIN` does the same for the primary, B-84). The server **refuses to start with fewer than 2 fallbacks** (`config.validate_quorum_servers`, B-84; `ALLOW_NO_QUORUM=true` overrides it, development only). After any failed or dropped session the next server is tried immediately; the backoff (1s doubling to 30s) only kicks in once every server has had a turn.
 - **Bounded requests** (`_REQUEST_TIMEOUT_SECONDS` = 15s); a timeout tears the connection down. Unbounded waits used to hang `POST /bets` *while holding the per-user lock*, and could stall the confirmation poller permanently.
-- **The drop is observable**: `client.wait_closed()` resolves when the read loop dies, and `_run_once` races it against the notification consumers and a 60s `server.ping`. Without it the listener sat on queues nobody would ever fill while `listener.client` still looked alive.
+- **The drop is observable**: `client.wait_closed()` resolves when the read loop dies, and `_run_once` races it against the notification consumers and a 60s `server.ping`. A server that keeps answering pings but stopped following the chain is caught too (B-86): at every ping, if no header has been *accepted* for 360s (`_STALL_SECONDS`), `_check_not_stalled` asks the other servers for their tip and ends the session with `ServerStalledError` if any is higher, so `run()` rotates; none higher means a quiet chain and the session goes on. Without it the listener sat on queues nobody would ever fill while `listener.client` still looked alive.
 - **Headers are validated, not trusted** (`_apply_header`, async since B-74): the tip never regresses, a header must meet the difficulty target it claims *and* that target must be no easier than `draw.POW_LIMIT` (PLM's powLimit; `tests/unit/conftest.py` swaps in a regtest limit so tests can mine headers, and the tests of the limit itself put `MAINNET_POW_LIMIT` back), and every advance must be linked to the tip: a single block by its `prev_hash`, a jump by downloading the headers in between (`blockchain.block.headers`, ≤2016 per request) from the active server and checking each. Failure raises `HeaderValidationError`, which ends the session like a dropped connection and rotates away — that header is the draw's only entropy, so a fabricated one picks the winner. The listener keeps the last 10 headers (`_window`, built from the first header plus its 9 predecessors); a header that does not join the tip is a reorg, walked back to its fork point inside the window and followed (`_follow_reorg`), a fork older than the window ends the session. A followed reorg writes `chain_reorg_detected` *before* the window moves (`_record_reorg`, B-85), and a depth of 2 or more pauses the lottery with `paused_reason = 'reorg'` in the same transaction; a header at the tip's own height with a different hash is a reorg too, not ignored. One more header is refused *without* ending the session (B-64): one carrying no `hex` at all (nothing to validate or draw from, and applying the height alone would break the `tip_height`/`tip_header_hex` pairing). `_run_once` separately refuses to publish the client while *no* tip is known, so the ignore-don't-kill choice can't reopen B-63.
 - **A quorum corroborates every money-moving decision** (`_corroborate_majority`, 10s per server, asking only the *other* endpoints — never the active one, which is what a MITM controls): `corroborate_header` before a block seeds the draw (B-28), `corroborate_utxo_spent` before a UTXO missing from one `listunspent` is written off as externally spent (B-29), and `corroborate_utxo_credit` before a new outpoint credits a balance — same outpoint, same amount, confirmed (B-59). Balances move in both directions, so both directions need the same quorum — and since B-83 that includes `corroborate_utxo_reinstatement` before a UTXO written off as externally spent is put back (same outpoint, amount, confirmed; `deposits/service.py:find_reinstate_candidates` only reads, `reinstate_reappeared_utxos` writes what the quorum agreed on), and `confirmation_proven` before a tx is marked confirmed: `tx/confirmation.py:poll_once` takes the height the active server reports only as a *claim*, and the row confirms when a strict majority of at least 2 of the *other* servers return a `get_merkle` proof that recomputes to the merkle root of the block's header (`electrum/merkle.py`; the header is the tip we hold, or for an older block the one the quorum agrees on and that meets its own target). With no `network` nothing confirms. The one decision still on the active server's word is the pool-coin listing for a payout (B-82). It needs **at least 2 answers** and a strict majority of those that answered (B-84): fewer → returns **False**, since one voice decides nothing independently and an unreachable network proves nothing. Only `ALLOW_NO_QUORUM=true` keeps the old reduced rule (no other server → True, otherwise a majority of whoever answers). A failed credit corroboration only *delays*: `find_new_credit_candidates` re-offers the outpoint on the next refresh or `DepositReconciler` sweep.
 

README.md

@@ -62,7 +62,7 @@ python -m pytest                    # all tests
 python -m pytest tests/unit/test_hd.py   # one file
 ```
 
-576 unit tests cover HD derivation, PSBT building, the Electrum client, bets,
+583 unit tests cover HD derivation, PSBT building, the Electrum client, bets,
 deposits, withdrawals, the round/draw engine, RBF fee-bumping, admin config,
 the pending-inclusive balance calculation, and the SSE push channel. No
 automated integration tests against a live Electrum connection — mainnet

app/electrum/listener.py

@@ -1,5 +1,6 @@
 import asyncio
 import logging
+import time
 from collections import Counter
 from collections.abc import Callable
 
@@ -55,6 +56,14 @@ _RESUBSCRIBE_CONCURRENCY = 20
 # a hostile server. Also the deepest fork it will follow.
 _HEADER_WINDOW = 10
 
+# B-86 (ELC-21): a connected server whose last accepted header is older than this is asked
+# about by comparing its tip with the other servers' at every ping. Three block times of
+# 120s: late enough that an ordinary slow stretch of the chain does not trigger the check.
+_STALL_SECONDS = 360
+
+# The clock the stall check reads — a module attribute so a test can move time.
+_monotonic = time.monotonic
+
 # A reorganisation at least this deep pauses the lottery (B-85, ELC-24): one block is
 # ordinary on any chain, two or more is worth a look before a new round opens.
 _REORG_PAUSE_DEPTH = 2
@@ -63,6 +72,11 @@ _REORG_PAUSE_DEPTH = 2
 _MAX_HEADERS_PER_REQUEST = 2016
 
 
+class ServerStalledError(Exception):
+    """The active server stopped following the chain while others report a higher tip
+    (B-86, ELC-21). Ends the session like a dropped connection, so run() rotates."""
+
+
 class ElectrumListener:
     """Long-lived background task: keeps one Electrum connection open, subscribes
     every user's address (plus any address added later via add_address), and
@@ -103,6 +117,8 @@ class ElectrumListener:
         # B-74 (ELC-22): height -> header of the last _HEADER_WINDOW accepted headers, tip
         # included. Survives reconnections, not restarts.
         self._window: dict[int, str] = {}
+        # B-86: when the last header was accepted (monotonic seconds), None until one is.
+        self._last_header_at: float | None = None
         self.client: ElectrumClient | None = None
 
     @property
@@ -266,6 +282,29 @@ class ElectrumListener:
         while True:
             await asyncio.sleep(_PING_INTERVAL_SECONDS)
             await client.ping()  # raises (and so ends the session) on timeout or a dead socket
+            await self._check_not_stalled()
+
+    async def _check_not_stalled(self) -> None:
+        """B-86 (ELC-21): `server.ping` keeps succeeding on a server that has stopped
+        following the chain, so liveness alone never replaced one. When the last accepted
+        header is older than _STALL_SECONDS, ask every *other* server for its tip
+        (`blockchain.headers.subscribe`, with ELC-14's timeouts): if any reports a higher one
+        than we know, this session ends and the listener moves to the next server. If none
+        does, the network simply produced no block and the session goes on."""
+        if self._last_header_at is None or _monotonic() - self._last_header_at <= _STALL_SECONDS:
+            return
+
+        async def _tip_height(client: ElectrumClient) -> int | None:
+            header = await client.subscribe_headers()
+            height = header.get("height") if isinstance(header, dict) else None
+            return height if isinstance(height, int) else None
+
+        heights = [h for h in await self._ask_other_servers(_tip_height) if h is not None]
+        if any(height > self.tip_height for height in heights):
+            raise ServerStalledError(
+                f"no header accepted for {_monotonic() - self._last_header_at:.0f}s while another server "
+                f"reports tip {max(heights)} (ours: {self.tip_height})"
+            )
 
     async def _subscribe_all_users(self) -> None:
         """B-31: subscribes with bounded concurrency (_RESUBSCRIBE_CONCURRENCY at
@@ -422,6 +461,7 @@ class ElectrumListener:
         for old in sorted(self._window)[:-_HEADER_WINDOW]:
             del self._window[old]
         self.tip_height, self.tip_header_hex = entries[-1]
+        self._last_header_at = _monotonic()
 
     async def _adopt_first_header(self, height: int, header_hex: str, client: ElectrumClient | None) -> None:
         """The process's first header has nothing to join, so it is taken on its work alone

docs/architettura.md

@@ -537,10 +537,9 @@ implementare» non esistono ancora nel codice (vedi [ARC-60] e [ARC-22]).
 
   *Codice:* `app/electrum/listener.py:ElectrumListener.run` ·
   `app/electrum/listener.py:ElectrumListener._apply_header` ·
+  `app/electrum/listener.py:ElectrumListener._check_not_stalled` ·
   `app/electrum/client.py:ElectrumClient.connect` ·
   *Test:* `tests/unit/test_electrum_listener.py`, `tests/unit/test_electrum_client.py`
-  *Stato:* da implementare (B-86) · *Oggi:* il listener non confronta la
-  propria punta con gli altri server.
 - **[ARC-27]** **`RoundScheduler`.** A ogni giro (5s) chiama
   `open_new_round_if_needed` e poi avanza il round attivo, in qualunque stato
   sia, `drawing` compreso: per un round in `drawing` il giro rientra

docs/connessione-electrum.md

@@ -120,10 +120,8 @@ Le difese sono cinque.
   quella nota, la sessione termina e si passa al server successivo
   nell'elenco ([ELC-02]). Se nessuno riporta un'altezza maggiore (la rete
   non ha prodotto blocchi) la sessione continua.
-  *Codice:* `app/electrum/listener.py:ElectrumListener._keepalive` · *Test:* —
-  *Stato:* da implementare (B-86) · *Oggi:* nulla confronta l'avanzamento
-  della punta con gli altri server; un server fermo che risponde ai ping
-  resta il server attivo indefinitamente.
+  *Codice:* `app/electrum/listener.py:ElectrumListener._keepalive`, `app/electrum/listener.py:ElectrumListener._check_not_stalled` ·
+  *Test:* `tests/unit/test_electrum_listener.py`
 
 ## 4. Header validati, non creduti
 

docs/storico-bug.md

@@ -234,7 +234,7 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)).
 | B-83 | Conferme e reintegri si basano solo sul server attivo | Alta | *da registrare* | `test_confirmation_proof.py`, `test_confirmation.py` | ELC, TX, DEP |
 | B-84 | Il quorum può ridursi a un solo server o a nessuno senza avvisi; certificati TLS non verificati | Alta | *da registrare* | `test_electrum_client.py`, `test_electrum_listener.py` | ELC, SEC |
 | B-85 | Le riorganizzazioni della catena non vengono né seguite né segnalate | Alta | *da registrare* | `test_electrum_listener.py` | ELC, RND |
-| B-86 | Un server connesso ma fermo non viene mai sostituito | Alta | **aperto** | — | ELC, OPS |
+| B-86 | Un server connesso ma fermo non viene mai sostituito | Alta | *da registrare* | `test_electrum_listener.py` | ELC, OPS |
 | B-87 | L'RBF può lasciare un resto sotto la soglia di dust | Alta | **aperto** | — | TX |
 | B-88 | Le transazioni SQLite non sono quelle che il codice presuppone; `synchronous=NORMAL` può perdere gli ultimi commit | Alta | *da registrare* | `test_sqlite_transactions.py`, `test_db_base.py` | DAT, ARC |
 | B-89 | Problemi della chiave rilevati solo al primo uso; gli script accettano qualunque xprv e scrivono file leggibili da tutti | Alta | **aperto** | — | SEC, OPS |
@@ -252,9 +252,9 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)).
 
 - **Corretti:** B-01 … B-70, tranne B-16, che è chiuso nella sostanza (il
   crash è eliminato e `/guida` serve un segnaposto) ma lascia la guida vera
-  come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76, B-77, B-78, B-83, B-74, B-85.
+  come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76, B-77, B-78, B-83, B-74, B-85, B-86.
 - **Aperti:** B-71, B-72 (gravità bassa, operativi o di documentazione) e
   B-73 … B-98 dalla revisione del 2026-09-28 tranne i corretti (4 critici,
-  5 alti, 6 medi), da implementare nell'ordine di
+  4 alti, 6 medi), da implementare nell'ordine di
   [piano-correzioni.md](piano-correzioni.md).
 - **Prossimo numero libero:** `B-99`.

tests/unit/test_electrum_listener.py

@@ -1254,3 +1254,138 @@ async def test_the_window_does_not_move_when_the_reorg_cannot_be_recorded(sessio
     await listener._apply_header({"height": 101, "hex": client.chain[101]}, client)
     assert listener.tip_height == 101
     assert len(await _reorg_events(session_factory)) == 1
+
+
+# --- B-86 (ELC-21): a connected server that stopped following the chain is replaced -------
+
+
+class _TipClient:
+    """What a *other* server answers to a headers subscription: its tip."""
+
+    def __init__(self, height: int | None, fail: bool = False):
+        self._height = height
+        self._fail = fail
+
+    async def connect(self):
+        if self._fail:
+            raise OSError("unreachable")
+
+    async def subscribe_headers(self):
+        return {"height": self._height, "hex": "00"}
+
+    async def close(self):
+        pass
+
+
+def _stall_listener(session_factory, monkeypatch, other_heights: dict[str, int | None]):
+    """A listener on `_ENDPOINTS[0]` with a tip at 100 accepted at t=0 and a movable clock;
+    the other two servers answer `other_heights` (None = unreachable)."""
+    from app.electrum import listener as listener_module
+
+    clock = {"now": 0.0}
+    monkeypatch.setattr(listener_module, "_monotonic", lambda: clock["now"])
+
+    def factory(endpoint):
+        height = other_heights[endpoint.host]
+        return _TipClient(height, fail=height is None)
+
+    listener = ElectrumListener(factory, session_factory, _ENDPOINTS)
+    listener._extend_window([(100, _mine_chain(100, 1)[100])])
+    return listener, clock
+
+
+async def test_a_stalled_server_is_replaced_when_another_reports_a_higher_tip(session_factory, monkeypatch):
+    """B-86, ELC-21: the active server keeps answering pings but its last accepted header is
+    older than 360s while another server is already at 101 — the session must end so run()
+    rotates. Nothing used to compare the tips, so a stalled server stayed active for ever."""
+    from app.electrum.listener import ServerStalledError
+
+    listener, clock = _stall_listener(
+        session_factory, monkeypatch, {"first.example": None, "second.example": 101, "third.example": 100}
+    )
+    clock["now"] = 361.0
+    with pytest.raises(ServerStalledError):
+        await listener._check_not_stalled()
+
+
+async def test_a_quiet_chain_does_not_rotate_the_session(session_factory, monkeypatch):
+    """ELC-21: no block for 360s but nobody reports a higher tip — the network produced
+    nothing, and the session goes on."""
+    listener, clock = _stall_listener(
+        session_factory, monkeypatch, {"first.example": None, "second.example": 100, "third.example": 99}
+    )
+    clock["now"] = 10_000.0
+    await listener._check_not_stalled()  # must not raise
+
+
+async def test_the_other_servers_are_not_asked_before_360_seconds(session_factory, monkeypatch):
+    """ELC-21: the check costs a connection to every other server, so it only starts once
+    the last header is older than 360s — exactly 360 is still fine."""
+    listener, clock = _stall_listener(
+        session_factory, monkeypatch, {"first.example": None, "second.example": 500, "third.example": 500}
+    )
+    clock["now"] = 360.0
+    await listener._check_not_stalled()  # must not raise although others are ahead
+
+
+async def test_an_unreachable_other_server_does_not_count_as_ahead(session_factory, monkeypatch):
+    listener, clock = _stall_listener(
+        session_factory, monkeypatch, {"first.example": None, "second.example": None, "third.example": None}
+    )
+    clock["now"] = 1_000.0
+    await listener._check_not_stalled()  # no answers, nothing to rotate to
+
+
+async def test_an_accepted_header_restarts_the_stall_clock(session_factory, monkeypatch):
+    """ELC-21: it is the age of the last *accepted* header that counts."""
+    listener, clock = _stall_listener(
+        session_factory, monkeypatch, {"first.example": None, "second.example": 500, "third.example": 500}
+    )
+    clock["now"] = 400.0
+    listener._extend_window([(101, _mine_chain(101, 1)[101])])
+    clock["now"] = 600.0  # 200s since the header at 101
+    await listener._check_not_stalled()
+
+
+async def test_the_keepalive_ends_the_session_of_a_stalled_server(session_factory, monkeypatch):
+    """B-86: wired into the ping loop — after a successful ping the stall check runs, and its
+    error is what ends the session (`_run_once` then returns and `run` rotates)."""
+    from app.electrum import listener as listener_module
+    from app.electrum.listener import ServerStalledError
+
+    monkeypatch.setattr(listener_module, "_PING_INTERVAL_SECONDS", 0)
+    listener, clock = _stall_listener(
+        session_factory, monkeypatch, {"first.example": None, "second.example": 101, "third.example": 100}
+    )
+    clock["now"] = 400.0
+    pinged = []
+
+    class _Active:
+        async def ping(self):
+            pinged.append(True)
+
+    with pytest.raises(ServerStalledError):
+        await asyncio.wait_for(listener._keepalive(_Active()), timeout=2)
+    assert pinged == [True]
+
+
+async def test_a_run_once_session_ends_when_the_active_server_stalls(session_factory, monkeypatch):
+    """B-86, end to end through _run_once: connected, header accepted at t=0, then the clock
+    passes 360s with another server ahead — the session returns by itself."""
+    from app.electrum import listener as listener_module
+
+    monkeypatch.setattr(listener_module, "_PING_INTERVAL_SECONDS", 0.01)
+    clock = {"now": 0.0}
+    monkeypatch.setattr(listener_module, "_monotonic", lambda: clock["now"])
+    chain = _mine_chain(91, 10)
+    active = _FakeConnectClient({"height": 100, "hex": chain[100]}, chain)
+
+    def factory(endpoint):
+        return active if endpoint == _ENDPOINTS[0] else _TipClient(101)
+
+    listener = ElectrumListener(factory, session_factory, _ENDPOINTS)
+    task = asyncio.create_task(listener._run_once(_ENDPOINTS[0]))
+    await _wait_until(lambda: listener.client is not None)
+    clock["now"] = 361.0
+    assert await asyncio.wait_for(task, timeout=3) is True
+    assert listener.client is None