Survive a dropped Electrum connection, and fall back to other servers

A dropped connection used to hang the whole platform permanently, and three
defects composed to do it (BUGS.md B-01):

The read loop's death was invisible. When the socket closed, _read_loop broke out
and finished, but _run_once was blocked on gather() over two notification
consumers waiting on queues nobody would ever fill again — it never returned and
never raised, so the reconnect-with-backoff logic was unreachable.
client.wait_closed() now resolves when the loop ends for any reason, and
_run_once races it against the consumers and a keepalive with
asyncio.wait(FIRST_COMPLETED).

Nothing had a timeout. request() registered a future, wrote to a half-closed
socket (drain() often doesn't raise) and awaited a reply that would never come.
That hung a POST /bets *while holding the per-user lock*, and could stop the
confirmation poller for good. Every request is now bounded at 15s, and a timeout
tears the connection down rather than leaving a server that owes us a reply in
rotation.

There was no keepalive, so on a quiet instance the normal way this connection
dies is an idle-timeout drop by the server (~10 minutes for many). A server.ping
every 60s makes that observable within a minute.

listener.client is also cleared before reconnecting, so callers stop treating a
dead connection as live.

On top of the finding, the listener now rotates over a list of servers:
ELECTRUM_FALLBACK_SERVERS holds comma-separated host:port[:notls] extras, tried
after the primary. Everything the platform does goes through this one connection
— deposit credits, broadcasts, confirmations, the chain tip the draw waits on —
which made a single hardcoded server its biggest point of failure. A failed or
dropped session moves to the next server immediately and only sleeps on the
backoff once every server has had a turn, so one dead server costs one attempt
instead of an outage, while a genuinely offline network still backs off. A
malformed entry fails at startup, not during the outage when the fallback is what
you need.

Also fixes B-19: header handling refuses a height below the current tip and
applies height and hex together, since _wait_for_next_block waits for
tip_height > tip_at_close (a regression silently added a block to the draw's
wait) and that hex is the draw's entropy source, so a mismatched pair would be
worse than a stale one.

Verified in the live deployment: the log shows the endpoint list, then "Electrum
connected to santantonio.sytes.net:50002", and the connection holds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 00:34:59 +02:00
co-authored by Claude Opus 5
parent 25f4a1c6b6
commit 7c4e9983ea
9 changed files with 516 additions and 42 deletions
+23 -1
View File
@@ -11,6 +11,13 @@ poterla avviare (in locale o via Docker). Per come avviarla poi ogni volta, vedi
- Un server Electrum raggiungibile per la rete PLM. Il server di bootstrap per lo
sviluppo è `santantonio.sytes.net:50002` (SSL) — va bene per i test, ma in
produzione conviene usarne uno di cui ci si fida o gestirne uno proprio.
- **Consigliato in produzione: più di un server.** Tutto passa da questa singola
connessione (accredito depositi, invio transazioni, conferme, altezza della
catena su cui si basa l'estrazione), quindi un solo server è il principale
punto di rottura della piattaforma. Elencane altri in
`ELECTRUM_FALLBACK_SERVERS` (vedi sotto): l'app li prova a rotazione, così un
server irraggiungibile costa un solo tentativo di riconnessione invece di un
disservizio.
## 2. Creare il file `.env`
@@ -29,7 +36,22 @@ cp .env.example .env
| `ADMIN_TOKEN` | Token bearer richiesto sugli endpoint admin (header `X-Admin-Token`). | `python -c "import secrets; print(secrets.token_urlsafe(32))"` |
Le altre chiavi di `.env` (`DATABASE_URL`, `ELECTRUM_HOST`/`PORT`/`USE_SSL`,
`MASTER_KEY_PATH`) hanno default sensati in `.env.example`. Nota: `.env`
`MASTER_KEY_PATH`) hanno default sensati in `.env.example`.
`ELECTRUM_FALLBACK_SERVERS` elenca i server di riserva, separati da virgola, nel
formato `host:porta` (TLS, il caso normale) oppure `host:porta:notls`. Esempio:
```
ELECTRUM_FALLBACK_SERVERS=nodo2.example.net:50002,nodo3.example.net:50001:notls
```
Vengono provati a rotazione dopo il primario. Attenzione: un valore scritto male
**blocca l'avvio** dell'app — è voluto, meglio accorgersene subito che durante il
disservizio in cui il fallback serve davvero.
`JWT_SECRET` e `XPRV_ENCRYPTION_KEY` vengono verificati all'avvio: se sono vuoti
(o `JWT_SECRET` è più corto di 32 caratteri) il container si rifiuta di partire con
un errore esplicito, invece di avviarsi e rompersi al primo login. Nota: `.env`
contiene solo segreti e configurazione di infrastruttura — i parametri di
business (bet amount, durata round, fee, ecc.) si configurano dal pannello
admin dopo l'avvio, non qui — vedi [guida-admin.md](guida-admin.md).