a4b160e2405b091310053ec14fa9db86d874d32e
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  davideandClaude Sonnet 5.5
|
a4b160e240 |
Abandon a transaction only on proof, and keep watching failed rows (B-78, TX-15, TX-17, TX-23, TX-26, ELC-28, DAT-33, DAT-36, DAT-69, DAT-72, AUD-11, AUD-12, AUD-31, AUD-39, API-32)
reconcile_once abandoned a row when the *active* server's history lacked its txid:
after 120s for `building` (which a restart onto a different server makes likely),
after 6h for `pending`, and all at once after a forward clock jump. It re-checked the
status but not the txid, so it raced the bumper, and a `failed` row was never looked at
again.
- The reconciler asks every server (ElectrumListener.get_histories, active + others) for
the history of the row's address and looks for every txid of the row. Present on any
server -> promoted (adopting the version found). Absent is only "no server of the
quorum knows it" and needs the active server plus the quorum minimum of the others to
have answered; even then nothing is released without proof:
(a) an input spent in a block by a transaction that is none of the row's versions,
agreed by a strict majority of a quorum of the other servers (confirmed_spender), or
(b) a rebroadcast of the latest raw tx to every server (rebroadcast_everywhere)
refused by all of them, explicitly, for missing/spent inputs.
Accepted or "already known" by any server means alive; a fee refusal is not proof
(`building` -> `pending` for the bumper, `pending` untouched); too few answers or
anything else leaves the row alone. failure_reason names the proof.
- The write re-reads status *and* current_txid under begin_immediate (TX-26).
- pending_transactions.failed_at (set by _abandon and _release_failed_withdrawal) starts
a 14-day watch: a failed row that a majority of the other servers (at least the quorum
minimum) list in a block becomes `confirmed`, writes pending_tx_resurrected once and
pauses the lottery with paused_reason `resurrected_tx`; linked rows are not corrected.
The confirmation is by quorum of histories for now; merkle proofs come with B-83.
- round_config.paused_reason (migration b7d2e41c9a05, with the failed_at one): `operator`
from POST /admin/pause (which accepts an optional `note` and never overwrites an
automatic reason), cleared by resume, which logs the reason it lifted; exposed on
/admin/config and GET /rounds/current. rounds/pause.py:pause_automatically is the
system's way in.
- PendingTransactionReconciler takes the listener as `network`; with none it can promote
but never abandon.
Tests changed because the spec changed: the three reconciler tests that expect an
abandonment (and the one in test_tx_versions) now supply a network that proves the
transaction is gone.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
||
|
davideandClaude Sonnet 5.5
|
2b6ddb019b |
Never roll back an ambiguous broadcast as if it had been refused (B-76, TX-04, TX-05, TX-20, TX-21, BET-16, BET-17, BET-23, WDR-11, WDR-12, WDR-16, API-13)
place_bet and request_withdrawal released everything on any exception from the broadcast, including the 15s timeout and a connection dropped after the write. The tx may have propagated: a bet's sats landed in the pool with no participant, a withdrawal left while its row said failed and its coins were spendable again. - client: ElectrumRpcError (subclass of ElectrumError) is raised only for a reply carrying a JSON-RPC `error`; timeouts, drops and write errors stay plain ElectrumError. - tx/outcome.py:broadcast_with_verdict sorts a broadcast into accepted / refused / uncertain. A refusal is verified first: ElectrumListener.transaction_known asks blockchain.transaction.get of the active server and every other one (10s each) — any server returning the tx means accepted; the active server plus MIN_QUORUM_RESPONSES others all denying it means refused (ALLOW_NO_QUORUM keeps the reduced rule); too few answers is uncertain. "already known" wording is only a shortcut that skips the check. With no network to ask, a refusal is uncertain. - place_bet / request_withdrawal / _trigger_payout use it: only a verified refusal releases; uncertain leaves the rows building for the reconciler, publishes SSE and POST /bets and POST /withdrawals answer 202 with status building; "already known" promotes a payout (payout_sent) like any accepted broadcast. - An accepted transaction is fanned out to every other server (ElectrumListener.fan_out_transaction, best effort, off the caller's path). The RBF replacement in bump_fee is left for B-77, which rewrites that function. Tests changed because the spec changed: the existing "refused broadcast" tests raised a bare RuntimeError (now an uncertain outcome) and are made explicit refusals (rpc_error) with a network that denies the tx; direct route calls pass a Response; FakePayoutListener gains the network surface. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
13cb66cc06 |
Require a real Electrum quorum and allow certificate pinning (B-84, ELC-01, ELC-03, ELC-14, ELC-19, ARC-20, OPS-12)
An empty ELECTRUM_FALLBACK_SERVERS (the .env.example default) silently made every corroboration pass, and with fallbacks the majority was taken over whoever answered, so one reachable fallback decided alone. TLS certificates were never checked. - _corroborate_majority passes only with at least 2 answers (MIN_QUORUM_RESPONSES) and a strict majority of the answers; no other server configured, or fewer than 2 answering, fails. ALLOW_NO_QUORUM=true keeps the old reduced rule. - config.validate_quorum_servers, called from the lifespan after parse_endpoints, refuses to start with fewer than 2 fallback servers unless ALLOW_NO_QUORUM=true (which logs a warning at every start). - Optional pinning: `host:port[:mode]:pin=<sha256>` per fallback and ELECTRUM_CERT_PIN for the primary; ElectrumClient compares the SHA-256 of the DER certificate after the TLS handshake and a mismatch fails like an unreachable server. A pin on a plaintext entry is a startup error. - .env.example now ships the four bootstrap servers and documents both new variables; docs/setup.md, spec, storico and BUGS.md updated. - Tests changed because the spec changed: the two "no other servers configured -> True" corroboration tests now assert that outcome only under allow_no_quorum. Note: a dev .env with fewer than 2 fallback servers will now refuse to start. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
daec4749c2 |
Survive oversized Electrum replies and ignore dust deposits (B-75, ELC-20, DEP-01, DAT-10)
A reply line over asyncio's default 64 KiB StreamReader limit raised out of _read_loop and closed the only connection; the pool address's get_history passes that after ~800 transactions, and dust sent to any public address does it on purpose. - ElectrumClient opens the stream with a 16 MiB limit and frames lines itself: an oversized line is discarded up to its newline, a non-JSON line is dropped, and the request whose id is readable in the first/last 256 bytes fails at once; the connection stays up either way. - Coins below DUST_LIMIT_SATS are never offered as credit candidates, never credited, and select_utxos never selects them. - Spec (ELC-20, DEP-01, DAT-10, ARC), storico, BUGS.md and test counts updated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
c0314e2bf0 |
Never swap the tip's hash sideways, and never split it from its height (B-64)
_apply_header's linkage check only fires on a single-block advance, so a header at the height we already held one for was applied on nothing but its own self-consistency — and that check, as header_meets_its_own_target's own docstring says, a server can satisfy with a self-declared easy target. So the one value the draw is seeded from could be replaced under us at the current height, by a reorg at the tip or by a single server disagreeing with the rest, with no check able to speak to it. Separately, a header carrying no hex set tip_header_hex back to None while advancing tip_height, leaving the two describing different blocks — the exact pairing that function exists to keep. Both are now refused without ending the session, unlike the fabrication cases above them: neither is evidence of a hostile server, and rotating away would cost us the one connection that also credits deposits and broadcasts transactions. - A same-height header is ignored (logged when it actually differs). The hash committed to for a height is not swapped under us; if ours turns out to be the orphan, corroborate_header already refuses to seed a draw from it and the draw waits for a further block. - A hex-less header is ignored outright: nothing to validate, nothing to draw from. A server that only ever pushed heights now freezes the draw — visibly, via B-36's draw_stalled — instead of costing us the connection. Because ignoring is not fatal, _run_once additionally refuses to publish the client when the initial header leaves the tip still unknown, so this cannot reopen B-63's window from the other side. The two _run_once tests are bounded with asyncio.wait_for: without their guard that call waits on session tasks nothing ends, and a regression must fail rather than hang the suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
8a0ebecfcc |
Never let a draw be seeded by a block that predates the close (B-63)
ElectrumListener._run_once assigned self.client before subscribe_headers() returned, so there was a window — one round-trip wide, at process start — where the connection looked alive while tip_height was still its initial 0. "client is not None" is what every consumer reads as "the chain is reachable", RoundScheduler._tick included, and a round closing inside that window recorded tip_at_close = 0. The very first header we then learned about — the current tip, a block mined *before* the round closed, whose hash was already public while bets were still open — satisfied tip_height > tip_at_close and became the draw's entropy. The draw's whole guarantee is that its seed did not exist yet when betting stopped. Two changes, defending different things: - The client is published only once the first header has been applied, so "client is not None" now means "reachable *and* we know where the chain is". During the window consumers see no connection, which is honest: a bet gets the same 503 it already gets while disconnected, and the background tasks skip a cycle as they already do. - _wait_for_next_block treats a baseline of 0 as *unknown*, not as height zero: it adopts the first height it learns as the baseline, waits for a block strictly after it, and records draw_baseline_tip_unknown so the extra block of waiting is explainable from /admin. Unreachable via the listener now, but it is the local statement of what the draw requires, and nothing else in that function would notice if the invariant stopped holding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
6246b13247 |
Corroborate deposit credits, not just external spends (B-59)
A candidate external spend has needed a quorum since B-29, but `value` and `height` for a *credit* came from the single active connection and went straight into utxo_events. One hostile or broken server could therefore inflate a user's displayed balance with outpoints that don't exist. It never spends anyone else's coins — a bet or withdrawal built on a phantom UTXO is refused at broadcast and rolled back — but it wedges the balance display and burns build attempts, and on a custodial platform a balance that isn't real is a support incident either way. Balances move in both directions; both directions now need the same quorum. corroborate_utxo_credit asks the other configured servers whether they report the same outpoint, for the same amount, confirmed. The height itself isn't compared: a server still catching up reports height 0 and simply doesn't agree, which is the same answer, while two honest servers can't disagree on the height of a genuinely confirmed outpoint. refresh_user gains the phase that shape already implied: find_new_credit_ candidates (new, confirmed, not already held) inside the first session, corroboration outside any session, then credit_confirmed_utxos over what survived. Only new outpoints are corroborated — re-checking what we already hold would open a connection to every other server on every refresh for an answer that can no longer change anything. A failed corroboration delays a credit, it never loses one: the next scripthash notification or DepositReconciler sweep (300s) re-offers the same outpoint, and the withholding is logged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
davide
|
4124dc08e6 |
Check confirmation/existence via scripthash history, not verbose replies (B-41)
poll_once and reconcile.py's existence check both called
blockchain.transaction.get(txid, verbose=True). Several Electrum server
implementations and versions reject the verbose flag outright
("verbose transactions are currently unsupported"), which would have
meant no confirmations and no reconciliation ever running against such
a server, read as a plain transport error. reconcile.py additionally
decided whether to abandon a transaction - releasing its funds - by
substring-matching the error text ("missing", "not found", ...), which
only works against ElectrumX's specific wording.
Both now ask blockchain.scripthash.get_history for the address that
owns every input of the transaction (a user's own address for a
bet/withdrawal, the pool address for a payout) and look for the txid in
the result: present with height > 0 means confirmed, present with
height <= 0 means still in the mempool, absent means the server
doesn't know it. get_history is a plain, universally-supported Electrum
method, and "not in the list" replaces the old substring-matching
entirely - no more guessing at error wording to decide whether to
release funds. History is cached per scripthash within one pass, since
every "payout" row shares the same pool address.
New app/tx/pending_address.py factors out own_address_for (the
address derivation was previously duplicated informally inside
tx/broadcast.py's signing context) so confirmation.py and reconcile.py
share one definition instead of two that could compute different
addresses for the same row.
tests/unit/test_confirmation.py and test_reconcile.py needed real User
rows and a master-key bootstrap they didn't have before, since address
derivation is now exercised for real rather than assumed. Suite grows
from 217 to 222 tests. BUGS.md moves B-41 to Previously fixed - no
Medium-severity finding remains open.
|
||
|
davide
|
12df04178e |
Resubscribe concurrently and in the background on reconnect (B-31)
_run_once awaited _subscribe_all_users() inline, before starting the header/scripthash consumer tasks, and that method subscribed one user at a time. At thousands of users that's thousands of sequential round-trips during which nothing else ran: tip_height was frozen and an in-flight draw's _wait_for_next_block made zero progress for the entire resubscribe - a reconnect (which the listener already treats as routine, not exceptional) could stall the lottery for minutes. _subscribe_all_users now fans out with bounded concurrency (asyncio.Semaphore, 20 at a time) instead of a serial loop, and one user's failure no longer stops the rest. _run_once now starts it as its own background task, created after the consumer tasks rather than awaited before them, so tip updates and already-subscribed users' notifications keep flowing throughout - its own completion is deliberately not raced against the session-ending tasks (unlike them, it's expected to finish normally), and its failure is logged the same way address_for_new_user's background task is (B-30). Left out: decoupling the listunspent refresh from the subscribe call itself (the third part of the proposed fix) - the periodic DepositReconciler (B-30) already provides a backstop for a slow or delayed initial refresh, so the added complexity wasn't worth it here. Suite grows from 182 to 185 tests, including an end-to-end test against _run_once proving a new tip is processed while a slow resubscribe is still in flight. BUGS.md moves B-31 to Previously fixed. |
||
|
davide
|
63df38d30b |
Add a periodic deposit reconciler, and stop losing subscribe tasks (B-30)
Deposits were credited exclusively by scripthash-change notifications, with nothing re-verifying a user's balance against the chain if a subscription was ever silently lost. address_for_new_user's subscribe was fire-and-forget: the task wasn't retained, so it could be garbage-collected mid-flight, and any failure (including self.client turning None between the check and the task running) vanished into asyncio's default unretrieved-exception handler instead of being logged anywhere. On an otherwise healthy, long-lived connection there may be no reconnect for days to re-subscribe everyone, so a user in that state never saw their deposits. address_for_new_user now retains the task and logs its exception if it fails. New app/deposits/reconcile.py adds DepositReconciler, a periodic sweep (every 5 minutes, gated on the Electrum client being connected, same shape as tx/reconcile.py) that round-robins over every user and calls the listener's own refresh_user (renamed from _refresh_user since it's now called from outside the class) - so the notification-driven and periodic paths can never behave differently. Deliberately sweeps every user rather than only ones missing from the in-memory scripthash map, since that map can't tell "never subscribed" apart from "subscribed, but the server stopped delivering notifications for it". Wired into app/main.py's lifespan alongside the other three background reconcilers. Suite grows from 176 to 182 tests. BUGS.md moves B-30 to Previously fixed. |
||
|
davide
|
e8fdea0389 |
Corroborate an external spend before marking a UTXO gone (B-29)
detect_external_spends marked a UTXO spent_txid='external-spend' irreversibly the moment it was missing from one listunspent reply, on one server, with no way to undo it. A rotated-to server that's broken or behind, or an empty reply, could zero a user's balance permanently. Split into three functions in deposits/service.py: find_utxos_missing_ from (read-only candidate detection, and refuses to flag anything at all when listunspent comes back entirely empty for a funded address - that reads as a broken response, not a full sweep), mark_utxos_spent_ externally (persistence only, once a candidate is already confirmed), and reinstate_reappeared_utxos (undoes the mark if the outpoint reappears as unspent later). ElectrumListener gains corroborate_utxo_spent, sharing the same majority-quorum logic corroborate_header already uses for B-28: before a candidate is marked, the other configured servers are asked whether they also see it as spent. _refresh_user now reads candidates, then corroborates each one with no DB session held open across those network calls (same shape as B-18/B-25), then persists. Suite grows from 165 to 176 tests. BUGS.md moves B-29 to Previously fixed. |
||
|
davide
|
0ce0562fd7 |
Validate Electrum headers and corroborate the draw's block (B-28)
A single hostile Electrum server, or a MITM on the one active connection, could fabricate the block header the draw's entropy comes from and so pick the winner of every round: headers were accepted with no proof-of-work check and no link to the previous tip. app/rounds/draw.py adds header_meets_its_own_target (rejects a header whose hash doesn't satisfy the difficulty target it claims) and header_prev_hash. electrum/listener.py's _apply_header now rejects a header failing either check by raising HeaderValidationError, which ends the session the same way a dropped connection would so the listener rotates to the next configured server. ElectrumListener gains corroborate_header: before the draw uses a block, it's independently checked against the other configured servers and needs a majority to agree. rounds/scheduler.py's _wait_for_next_block now calls this and, on failure, logs why and waits for a further block instead of ever using an uncorroborated header. Certificate/hostname verification stays disabled, so this doesn't cover an attacker able to MITM every configured server at once - BUGS.md notes that as not covered. Suite grows from 151 to 165 tests. BUGS.md moves B-28 to Previously fixed. |
||
|
davideandClaude Sonnet 5
|
f1a1145cda |
Detect UTXOs spent outside the platform and correct the cached balance
credit_confirmed_utxos only ever credited new UTXOs; a UTXO spent by something other than the app's own bet/withdrawal/payout flow (e.g. someone using the raw derived privkey directly) never got its spent_txid set, so cached_balance_sats kept counting it forever. detect_external_spends mirrors the same listunspent refresh in the other direction: anything still marked unspent in our DB but missing from the address's current unspent set gets spent_txid="external-spend", an audit_log entry, and an immediate balance recompute — wired into the same ElectrumListener._refresh_user call that already runs on every scripthash notification and on listener (re)connect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
7c4e9983ea |
Survive a dropped Electrum connection, and fall back to other servers
A dropped connection used to hang the whole platform permanently, and three defects composed to do it (BUGS.md B-01): The read loop's death was invisible. When the socket closed, _read_loop broke out and finished, but _run_once was blocked on gather() over two notification consumers waiting on queues nobody would ever fill again — it never returned and never raised, so the reconnect-with-backoff logic was unreachable. client.wait_closed() now resolves when the loop ends for any reason, and _run_once races it against the consumers and a keepalive with asyncio.wait(FIRST_COMPLETED). Nothing had a timeout. request() registered a future, wrote to a half-closed socket (drain() often doesn't raise) and awaited a reply that would never come. That hung a POST /bets *while holding the per-user lock*, and could stop the confirmation poller for good. Every request is now bounded at 15s, and a timeout tears the connection down rather than leaving a server that owes us a reply in rotation. There was no keepalive, so on a quiet instance the normal way this connection dies is an idle-timeout drop by the server (~10 minutes for many). A server.ping every 60s makes that observable within a minute. listener.client is also cleared before reconnecting, so callers stop treating a dead connection as live. On top of the finding, the listener now rotates over a list of servers: ELECTRUM_FALLBACK_SERVERS holds comma-separated host:port[:notls] extras, tried after the primary. Everything the platform does goes through this one connection — deposit credits, broadcasts, confirmations, the chain tip the draw waits on — which made a single hardcoded server its biggest point of failure. A failed or dropped session moves to the next server immediately and only sleeps on the backoff once every server has had a turn, so one dead server costs one attempt instead of an outage, while a genuinely offline network still backs off. A malformed entry fails at startup, not during the outage when the fallback is what you need. Also fixes B-19: header handling refuses a height below the current tip and applies height and hex together, since _wait_for_next_block waits for tip_height > tip_at_close (a regression silently added a block to the draw's wait) and that hex is the draw's entropy source, so a mismatched pair would be worse than a stale one. Verified in the live deployment: the log shows the endpoint list, then "Electrum connected to santantonio.sytes.net:50002", and the connection holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5
|
f229f91632 |
Add server-push (SSE) notifications for round/bet/balance state changes
Frontend dashboards previously found out about state changes only on their next poll tick (up to 15s, or 3s during a draw) — this adds a push channel so updates land as soon as they happen instead. - app/rounds/events.py: a small in-process pub/sub (RoundEventBroadcaster). The message carries no payload — it's just a "something changed, go refetch" signal, so it needs no auth and no knowledge of who's allowed to see what; personalization stays entirely in the existing REST endpoints. - GET /rounds/stream: an SSE endpoint exposing that channel, with keep-alive comments so it survives idle periods behind a reverse proxy, and a defensive MAX_SUBSCRIBERS cap (well above the ~100 concurrent users expected) — past it, the endpoint returns 503 instead of opening a stream, and callers just keep working off polling. - publish() calls added at every point that actually changes what a dashboard would want to know: new round opened, round status transitions (closing/drawing/paying_out/closed), a bet or withdrawal broadcast, any pending tx confirming (bet/withdrawal/payout), a deposit credited, and a new block tip arriving (the exact moment the "drawing" phase is waiting on). - Caddyfile: excludes /rounds/stream from gzip encoding, since compression would buffer output and defeat the point of a live stream. Single-process only by design for now (no cross-worker fan-out) and the notification is a generic broadcast rather than a per-user channel — both are deliberate scope decisions for the current ~100-user, single-container deployment, not oversights. Polling is left fully in place as a fallback; this is purely additive. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5
|
a21e058cdd |
Add Electrum SPV client and address listener
Minimal Electrum protocol client (client.py) plus a scripthash subscription listener (listener.py) that watches user deposit addresses for confirmed UTXOs, with the address->scripthash conversion helper and a manual smoke-test script against the dev bootstrap server. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |