Survive a dropped Electrum connection, and fall back to other servers

A dropped connection used to hang the whole platform permanently, and three
defects composed to do it (BUGS.md B-01):

The read loop's death was invisible. When the socket closed, _read_loop broke out
and finished, but _run_once was blocked on gather() over two notification
consumers waiting on queues nobody would ever fill again — it never returned and
never raised, so the reconnect-with-backoff logic was unreachable.
client.wait_closed() now resolves when the loop ends for any reason, and
_run_once races it against the consumers and a keepalive with
asyncio.wait(FIRST_COMPLETED).

Nothing had a timeout. request() registered a future, wrote to a half-closed
socket (drain() often doesn't raise) and awaited a reply that would never come.
That hung a POST /bets *while holding the per-user lock*, and could stop the
confirmation poller for good. Every request is now bounded at 15s, and a timeout
tears the connection down rather than leaving a server that owes us a reply in
rotation.

There was no keepalive, so on a quiet instance the normal way this connection
dies is an idle-timeout drop by the server (~10 minutes for many). A server.ping
every 60s makes that observable within a minute.

listener.client is also cleared before reconnecting, so callers stop treating a
dead connection as live.

On top of the finding, the listener now rotates over a list of servers:
ELECTRUM_FALLBACK_SERVERS holds comma-separated host:port[:notls] extras, tried
after the primary. Everything the platform does goes through this one connection
— deposit credits, broadcasts, confirmations, the chain tip the draw waits on —
which made a single hardcoded server its biggest point of failure. A failed or
dropped session moves to the next server immediately and only sleeps on the
backoff once every server has had a turn, so one dead server costs one attempt
instead of an outage, while a genuinely offline network still backs off. A
malformed entry fails at startup, not during the outage when the fallback is what
you need.

Also fixes B-19: header handling refuses a height below the current tip and
applies height and hex together, since _wait_for_next_block waits for
tip_height > tip_at_close (a regression silently added a block to the draw's
wait) and that hex is the draw's entropy source, so a mismatched pair would be
worse than a stale one.

Verified in the live deployment: the log shows the endpoint list, then "Electrum
connected to santantonio.sytes.net:50002", and the connection holds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

app/main.py

@@ -24,35 +24,51 @@ from app.auth.routes import router as auth_router
 from app.api.errors import ApiError, http_error
 from app.config import settings, validate_runtime_secrets
 from app.db.base import AsyncSessionLocal
-from app.electrum.client import ElectrumClient
+from app.electrum.client import ElectrumClient, ElectrumEndpoint, parse_endpoints
 from app.electrum.listener import ElectrumListener
 from app.rounds.scheduler import RoundScheduler
 from app.tx.broadcast import RbfBumper
 from app.tx.confirmation import ConfirmationPoller
 from app.tx.locks import UserLocks
+from app.tx.reconcile import PendingTransactionReconciler
 
 logger = logging.getLogger(__name__)
 
 
-def _make_electrum_client() -> ElectrumClient:
-    return ElectrumClient(settings.electrum_host, settings.electrum_port, settings.electrum_use_ssl)
+def _make_electrum_client(endpoint: ElectrumEndpoint) -> ElectrumClient:
+    return ElectrumClient(endpoint.host, endpoint.port, endpoint.use_ssl)
 
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
-    listener = ElectrumListener(_make_electrum_client, AsyncSessionLocal)
+    # Refuses to serve rather than starting up half-configured — see B-15 in BUGS.md.
+    validate_runtime_secrets()
+
+    endpoints = parse_endpoints(
+        settings.electrum_host,
+        settings.electrum_port,
+        settings.electrum_use_ssl,
+        settings.electrum_fallback_servers,
+    )
+    logger.info("Electrum endpoints (in rotation order): %s", ", ".join(str(e) for e in endpoints))
+
+    listener = ElectrumListener(_make_electrum_client, AsyncSessionLocal, endpoints)
     app.state.electrum_listener = listener
     app.state.user_locks = UserLocks()
 
     scheduler = RoundScheduler(AsyncSessionLocal, listener)
     poller = ConfirmationPoller(AsyncSessionLocal, lambda: listener.client)
     bumper = RbfBumper(AsyncSessionLocal, lambda: listener.client)
+    # Resolves in-flight transactions against the chain — the piece that lets the
+    # system recover on its own from a broadcast that never confirmed (B-04/B-08).
+    reconciler = PendingTransactionReconciler(AsyncSessionLocal, lambda: listener.client)
 
     tasks = [
         asyncio.create_task(listener.run()),
         asyncio.create_task(scheduler.run()),
         asyncio.create_task(poller.run()),
         asyncio.create_task(bumper.run()),
+        asyncio.create_task(reconciler.run()),
     ]
     try:
         yield
@@ -75,6 +91,9 @@ app.include_router(rounds_router)
 
 @app.exception_handler(Exception)
 async def log_unhandled_exception(request: Request, exc: Exception) -> JSONResponse:
+    """Answers with the same structured `detail` shape as every deliberate failure
+    (app/api/errors.py) so clients never have to special-case unexpected errors.
+    The exception itself stays in logs/app.log only — never in the response body."""
     logger.exception("Unhandled error on %s %s", request.method, request.url.path)
     return JSONResponse(
         status_code=500,
@@ -100,6 +119,9 @@ async def admin_panel() -> FileResponse:
     return FileResponse("app/static/admin.html", headers=_NO_STORE_HEADERS)
 
 
+_USER_GUIDE_PATH = Path("docs/guida-utente.md")
+
+
 @app.get("/guida", include_in_schema=False)
 async def user_guide() -> FileResponse:
     """Serves docs/guida-utente.md as plain text so it opens inline in the
@@ -116,6 +138,3 @@ async def user_guide() -> FileResponse:
 
 
 app.mount("/", StaticFiles(directory="app/static", html=True), name="static")
-_USER_GUIDE_PATH = Path("docs/guida-utente.md")
-
-