poll_once and reconcile.py's existence check both called
blockchain.transaction.get(txid, verbose=True). Several Electrum server
implementations and versions reject the verbose flag outright
("verbose transactions are currently unsupported"), which would have
meant no confirmations and no reconciliation ever running against such
a server, read as a plain transport error. reconcile.py additionally
decided whether to abandon a transaction - releasing its funds - by
substring-matching the error text ("missing", "not found", ...), which
only works against ElectrumX's specific wording.
Both now ask blockchain.scripthash.get_history for the address that
owns every input of the transaction (a user's own address for a
bet/withdrawal, the pool address for a payout) and look for the txid in
the result: present with height > 0 means confirmed, present with
height <= 0 means still in the mempool, absent means the server
doesn't know it. get_history is a plain, universally-supported Electrum
method, and "not in the list" replaces the old substring-matching
entirely - no more guessing at error wording to decide whether to
release funds. History is cached per scripthash within one pass, since
every "payout" row shares the same pool address.
New app/tx/pending_address.py factors out own_address_for (the
address derivation was previously duplicated informally inside
tx/broadcast.py's signing context) so confirmation.py and reconcile.py
share one definition instead of two that could compute different
addresses for the same row.
tests/unit/test_confirmation.py and test_reconcile.py needed real User
rows and a master-key bootstrap they didn't have before, since address
derivation is now exercised for real rather than assumed. Suite grows
from 217 to 222 tests. BUGS.md moves B-41 to Previously fixed - no
Medium-severity finding remains open.
_run_once awaited _subscribe_all_users() inline, before starting the
header/scripthash consumer tasks, and that method subscribed one user
at a time. At thousands of users that's thousands of sequential
round-trips during which nothing else ran: tip_height was frozen and
an in-flight draw's _wait_for_next_block made zero progress for the
entire resubscribe - a reconnect (which the listener already treats as
routine, not exceptional) could stall the lottery for minutes.
_subscribe_all_users now fans out with bounded concurrency
(asyncio.Semaphore, 20 at a time) instead of a serial loop, and one
user's failure no longer stops the rest. _run_once now starts it as
its own background task, created after the consumer tasks rather than
awaited before them, so tip updates and already-subscribed users'
notifications keep flowing throughout - its own completion is
deliberately not raced against the session-ending tasks (unlike them,
it's expected to finish normally), and its failure is logged the same
way address_for_new_user's background task is (B-30).
Left out: decoupling the listunspent refresh from the subscribe call
itself (the third part of the proposed fix) - the periodic
DepositReconciler (B-30) already provides a backstop for a slow or
delayed initial refresh, so the added complexity wasn't worth it here.
Suite grows from 182 to 185 tests, including an end-to-end test
against _run_once proving a new tip is processed while a slow
resubscribe is still in flight. BUGS.md moves B-31 to Previously
fixed.
Deposits were credited exclusively by scripthash-change notifications,
with nothing re-verifying a user's balance against the chain if a
subscription was ever silently lost. address_for_new_user's subscribe
was fire-and-forget: the task wasn't retained, so it could be
garbage-collected mid-flight, and any failure (including self.client
turning None between the check and the task running) vanished into
asyncio's default unretrieved-exception handler instead of being
logged anywhere. On an otherwise healthy, long-lived connection there
may be no reconnect for days to re-subscribe everyone, so a user in
that state never saw their deposits.
address_for_new_user now retains the task and logs its exception if
it fails. New app/deposits/reconcile.py adds DepositReconciler, a
periodic sweep (every 5 minutes, gated on the Electrum client being
connected, same shape as tx/reconcile.py) that round-robins over every
user and calls the listener's own refresh_user (renamed from
_refresh_user since it's now called from outside the class) - so the
notification-driven and periodic paths can never behave differently.
Deliberately sweeps every user rather than only ones missing from the
in-memory scripthash map, since that map can't tell "never subscribed"
apart from "subscribed, but the server stopped delivering
notifications for it". Wired into app/main.py's lifespan alongside the
other three background reconcilers.
Suite grows from 176 to 182 tests. BUGS.md moves B-30 to Previously
fixed.
detect_external_spends marked a UTXO spent_txid='external-spend'
irreversibly the moment it was missing from one listunspent reply, on
one server, with no way to undo it. A rotated-to server that's broken
or behind, or an empty reply, could zero a user's balance permanently.
Split into three functions in deposits/service.py: find_utxos_missing_
from (read-only candidate detection, and refuses to flag anything at
all when listunspent comes back entirely empty for a funded address -
that reads as a broken response, not a full sweep), mark_utxos_spent_
externally (persistence only, once a candidate is already confirmed),
and reinstate_reappeared_utxos (undoes the mark if the outpoint
reappears as unspent later).
ElectrumListener gains corroborate_utxo_spent, sharing the same
majority-quorum logic corroborate_header already uses for B-28: before
a candidate is marked, the other configured servers are asked whether
they also see it as spent. _refresh_user now reads candidates, then
corroborates each one with no DB session held open across those
network calls (same shape as B-18/B-25), then persists.
Suite grows from 165 to 176 tests. BUGS.md moves B-29 to Previously
fixed.
A single hostile Electrum server, or a MITM on the one active
connection, could fabricate the block header the draw's entropy comes
from and so pick the winner of every round: headers were accepted with
no proof-of-work check and no link to the previous tip.
app/rounds/draw.py adds header_meets_its_own_target (rejects a header
whose hash doesn't satisfy the difficulty target it claims) and
header_prev_hash. electrum/listener.py's _apply_header now rejects a
header failing either check by raising HeaderValidationError, which
ends the session the same way a dropped connection would so the
listener rotates to the next configured server.
ElectrumListener gains corroborate_header: before the draw uses a
block, it's independently checked against the other configured servers
and needs a majority to agree. rounds/scheduler.py's
_wait_for_next_block now calls this and, on failure, logs why and
waits for a further block instead of ever using an uncorroborated
header.
Certificate/hostname verification stays disabled, so this doesn't
cover an attacker able to MITM every configured server at once -
BUGS.md notes that as not covered.
Suite grows from 151 to 165 tests. BUGS.md moves B-28 to Previously
fixed.
credit_confirmed_utxos only ever credited new UTXOs; a UTXO spent by
something other than the app's own bet/withdrawal/payout flow (e.g. someone
using the raw derived privkey directly) never got its spent_txid set, so
cached_balance_sats kept counting it forever. detect_external_spends mirrors
the same listunspent refresh in the other direction: anything still marked
unspent in our DB but missing from the address's current unspent set gets
spent_txid="external-spend", an audit_log entry, and an immediate balance
recompute — wired into the same ElectrumListener._refresh_user call that
already runs on every scripthash notification and on listener (re)connect.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A dropped connection used to hang the whole platform permanently, and three
defects composed to do it (BUGS.md B-01):
The read loop's death was invisible. When the socket closed, _read_loop broke out
and finished, but _run_once was blocked on gather() over two notification
consumers waiting on queues nobody would ever fill again — it never returned and
never raised, so the reconnect-with-backoff logic was unreachable.
client.wait_closed() now resolves when the loop ends for any reason, and
_run_once races it against the consumers and a keepalive with
asyncio.wait(FIRST_COMPLETED).
Nothing had a timeout. request() registered a future, wrote to a half-closed
socket (drain() often doesn't raise) and awaited a reply that would never come.
That hung a POST /bets *while holding the per-user lock*, and could stop the
confirmation poller for good. Every request is now bounded at 15s, and a timeout
tears the connection down rather than leaving a server that owes us a reply in
rotation.
There was no keepalive, so on a quiet instance the normal way this connection
dies is an idle-timeout drop by the server (~10 minutes for many). A server.ping
every 60s makes that observable within a minute.
listener.client is also cleared before reconnecting, so callers stop treating a
dead connection as live.
On top of the finding, the listener now rotates over a list of servers:
ELECTRUM_FALLBACK_SERVERS holds comma-separated host:port[:notls] extras, tried
after the primary. Everything the platform does goes through this one connection
— deposit credits, broadcasts, confirmations, the chain tip the draw waits on —
which made a single hardcoded server its biggest point of failure. A failed or
dropped session moves to the next server immediately and only sleeps on the
backoff once every server has had a turn, so one dead server costs one attempt
instead of an outage, while a genuinely offline network still backs off. A
malformed entry fails at startup, not during the outage when the fallback is what
you need.
Also fixes B-19: header handling refuses a height below the current tip and
applies height and hex together, since _wait_for_next_block waits for
tip_height > tip_at_close (a regression silently added a block to the draw's
wait) and that hex is the draw's entropy source, so a mismatched pair would be
worse than a stale one.
Verified in the live deployment: the log shows the endpoint list, then "Electrum
connected to santantonio.sytes.net:50002", and the connection holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Frontend dashboards previously found out about state changes only on their
next poll tick (up to 15s, or 3s during a draw) — this adds a push channel
so updates land as soon as they happen instead.
- app/rounds/events.py: a small in-process pub/sub (RoundEventBroadcaster).
The message carries no payload — it's just a "something changed, go
refetch" signal, so it needs no auth and no knowledge of who's allowed to
see what; personalization stays entirely in the existing REST endpoints.
- GET /rounds/stream: an SSE endpoint exposing that channel, with keep-alive
comments so it survives idle periods behind a reverse proxy, and a
defensive MAX_SUBSCRIBERS cap (well above the ~100 concurrent users
expected) — past it, the endpoint returns 503 instead of opening a stream,
and callers just keep working off polling.
- publish() calls added at every point that actually changes what a
dashboard would want to know: new round opened, round status transitions
(closing/drawing/paying_out/closed), a bet or withdrawal broadcast, any
pending tx confirming (bet/withdrawal/payout), a deposit credited, and a
new block tip arriving (the exact moment the "drawing" phase is waiting on).
- Caddyfile: excludes /rounds/stream from gzip encoding, since compression
would buffer output and defeat the point of a live stream.
Single-process only by design for now (no cross-worker fan-out) and the
notification is a generic broadcast rather than a per-user channel — both
are deliberate scope decisions for the current ~100-user, single-container
deployment, not oversights. Polling is left fully in place as a fallback;
this is purely additive.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Minimal Electrum protocol client (client.py) plus a scripthash
subscription listener (listener.py) that watches user deposit addresses
for confirmed UTXOs, with the address->scripthash conversion helper and
a manual smoke-test script against the dev bootstrap server.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
davide
