Survive a dropped Electrum connection, and fall back to other servers

A dropped connection used to hang the whole platform permanently, and three
defects composed to do it (BUGS.md B-01):

The read loop's death was invisible. When the socket closed, _read_loop broke out
and finished, but _run_once was blocked on gather() over two notification
consumers waiting on queues nobody would ever fill again — it never returned and
never raised, so the reconnect-with-backoff logic was unreachable.
client.wait_closed() now resolves when the loop ends for any reason, and
_run_once races it against the consumers and a keepalive with
asyncio.wait(FIRST_COMPLETED).

Nothing had a timeout. request() registered a future, wrote to a half-closed
socket (drain() often doesn't raise) and awaited a reply that would never come.
That hung a POST /bets *while holding the per-user lock*, and could stop the
confirmation poller for good. Every request is now bounded at 15s, and a timeout
tears the connection down rather than leaving a server that owes us a reply in
rotation.

There was no keepalive, so on a quiet instance the normal way this connection
dies is an idle-timeout drop by the server (~10 minutes for many). A server.ping
every 60s makes that observable within a minute.

listener.client is also cleared before reconnecting, so callers stop treating a
dead connection as live.

On top of the finding, the listener now rotates over a list of servers:
ELECTRUM_FALLBACK_SERVERS holds comma-separated host:port[:notls] extras, tried
after the primary. Everything the platform does goes through this one connection
— deposit credits, broadcasts, confirmations, the chain tip the draw waits on —
which made a single hardcoded server its biggest point of failure. A failed or
dropped session moves to the next server immediately and only sleeps on the
backoff once every server has had a turn, so one dead server costs one attempt
instead of an outage, while a genuinely offline network still backs off. A
malformed entry fails at startup, not during the outage when the fallback is what
you need.

Also fixes B-19: header handling refuses a height below the current tip and
applies height and hex together, since _wait_for_next_block waits for
tip_height > tip_at_close (a regression silently added a block to the draw's
wait) and that hex is the draw's entropy source, so a mismatched pair would be
worse than a stale one.

Verified in the live deployment: the log shows the endpoint list, then "Electrum
connected to santantonio.sytes.net:50002", and the connection holds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

app/electrum/client.py

@@ -2,6 +2,67 @@ import asyncio
 import itertools
 import json
 import ssl
+from typing import NamedTuple
+
+
+class ElectrumEndpoint(NamedTuple):
+    """One server to connect to. The listener rotates over a list of these so a
+    single unreachable or misbehaving server doesn't take the platform down —
+    every consumer of PLM chain data goes through this one connection."""
+
+    host: str
+    port: int
+    use_ssl: bool = True
+
+    def __str__(self) -> str:
+        return f"{self.host}:{self.port}{'' if self.use_ssl else ' (plaintext)'}"
+
+
+def parse_endpoints(
+    primary_host: str, primary_port: int, primary_use_ssl: bool, fallback_spec: str
+) -> list[ElectrumEndpoint]:
+    """Build the connection rotation: the primary server first, then whatever
+    ELECTRUM_FALLBACK_SERVERS lists.
+
+    `fallback_spec` is comma-separated, each entry `host:port` (TLS, the normal
+    case) or `host:port:notls`. Malformed entries raise ValueError rather than
+    being skipped: a typo in a fallback server is something to fix at startup,
+    not to discover during an outage, when the fallback is what's needed.
+    Duplicates are dropped, keeping first position.
+    """
+    endpoints = [ElectrumEndpoint(primary_host, primary_port, primary_use_ssl)]
+    for raw in fallback_spec.split(","):
+        entry = raw.strip()
+        if not entry:
+            continue
+        parts = entry.split(":")
+        if len(parts) not in (2, 3):
+            raise ValueError(f"invalid Electrum server {entry!r}: expected host:port[:notls]")
+        host, port = parts[0].strip(), parts[1].strip()
+        if not host or not port.isdigit():
+            raise ValueError(f"invalid Electrum server {entry!r}: expected host:port[:notls]")
+        use_ssl = True
+        if len(parts) == 3:
+            flag = parts[2].strip().lower()
+            if flag not in ("ssl", "tls", "notls", "plain"):
+                raise ValueError(f"invalid TLS flag {flag!r} in Electrum server {entry!r}")
+            use_ssl = flag in ("ssl", "tls")
+        endpoints.append(ElectrumEndpoint(host, int(port), use_ssl))
+
+    deduped: list[ElectrumEndpoint] = []
+    for endpoint in endpoints:
+        if endpoint not in deduped:
+            deduped.append(endpoint)
+    return deduped
+
+
+
+# Every request is bounded: without this, a half-open socket (the peer vanished
+# without a FIN, or the read loop died — see _read_loop) leaves `await future`
+# hanging forever, and with it whatever was awaiting the call. That used to be
+# unbounded, which meant a POST /bets could hang while holding the per-user lock
+# and the confirmation poller could stop polling permanently.
+_REQUEST_TIMEOUT_SECONDS = 15
 
 
 class ElectrumError(Exception):
@@ -15,6 +76,11 @@ class ElectrumClient:
     arrive under the *same* method name as the subscribe call, multiplexed for every
     scripthash subscribed — callers read `notifications(method)` and, for scripthash
     pushes, dispatch on `params[0]` (the scripthash) themselves.
+
+    A dead connection is observable rather than silent: `wait_closed()` resolves as
+    soon as the read loop terminates for any reason, which is what lets
+    ElectrumListener notice the drop and reconnect instead of waiting forever on
+    notification queues nobody will ever fill again.
     """
 
     def __init__(self, host: str, port: int, use_ssl: bool = True):
@@ -27,6 +93,7 @@ class ElectrumClient:
         self._pending: dict[int, asyncio.Future] = {}
         self._subscriptions: dict[str, asyncio.Queue] = {}
         self._read_task: asyncio.Task | None = None
+        self._closed = asyncio.Event()
 
     async def connect(self) -> None:
         # Electrum servers commonly present self-signed certs; the protocol's trust
@@ -42,25 +109,51 @@ class ElectrumClient:
         await self.request("server.version", ["plm-lottery", "1.4"])
 
     async def close(self) -> None:
+        self._closed.set()
         if self._read_task is not None:
             self._read_task.cancel()
         if self._writer is not None:
             self._writer.close()
             try:
                 await asyncio.wait_for(self._writer.wait_closed(), timeout=2)
-            except (ssl.SSLError, TimeoutError, asyncio.TimeoutError):
+            except (ssl.SSLError, TimeoutError, asyncio.TimeoutError, ConnectionError):
                 pass  # some Electrum servers don't send a clean TLS close_notify
 
+    async def wait_closed(self) -> None:
+        """Resolves once this connection is gone — read loop finished (peer closed,
+        protocol error, TLS failure) or close() was called. ElectrumListener races
+        this against its notification consumers so a drop triggers a reconnect."""
+        await self._closed.wait()
+
     async def request(self, method: str, params: list | None = None) -> object:
-        if self._writer is None:
+        if self._writer is None or self._closed.is_set():
             raise ElectrumError("not connected")
         request_id = next(self._id_counter)
-        future: asyncio.Future = asyncio.get_event_loop().create_future()
+        future: asyncio.Future = asyncio.get_running_loop().create_future()
         self._pending[request_id] = future
         payload = json.dumps({"id": request_id, "method": method, "params": params or []}) + "\n"
-        self._writer.write(payload.encode())
-        await self._writer.drain()
-        return await future
+        try:
+            self._writer.write(payload.encode())
+            await self._writer.drain()
+        except (ConnectionError, ssl.SSLError, OSError) as exc:
+            self._pending.pop(request_id, None)
+            self._closed.set()
+            raise ElectrumError(f"write failed: {exc}") from exc
+        try:
+            return await asyncio.wait_for(future, timeout=_REQUEST_TIMEOUT_SECONDS)
+        except (TimeoutError, asyncio.TimeoutError) as exc:
+            self._pending.pop(request_id, None)
+            # A server that owes us a reply and never sends one is indistinguishable
+            # from a dead socket, and retrying on the same connection would keep
+            # hitting it — tear it down so the listener reconnects.
+            self._closed.set()
+            raise ElectrumError(f"{method} timed out after {_REQUEST_TIMEOUT_SECONDS}s") from exc
+
+    async def ping(self) -> None:
+        """Keepalive: Electrum servers drop idle connections (commonly after ~10
+        minutes), which on a quiet instance would otherwise be the normal way this
+        connection dies. Called periodically by ElectrumListener."""
+        await self.request("server.ping")
 
     def notifications(self, method: str) -> asyncio.Queue:
         return self._subscriptions.setdefault(method, asyncio.Queue())
@@ -92,6 +185,12 @@ class ElectrumClient:
                 message = json.loads(line)
                 self._dispatch(message)
         finally:
+            # Whatever ended this loop — clean EOF, protocol error, TLS failure — the
+            # connection is unusable from here on. Setting this is what makes the
+            # death observable to wait_closed(), and so to the listener's reconnect
+            # logic; without it the listener waited on notification queues nobody
+            # would ever fill again, forever (B-01).
+            self._closed.set()
             error = ElectrumError("connection closed")
             for future in self._pending.values():
                 if not future.done():