24e5fc3d8b329e161a97887ac4250f98754922d7
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  davideandClaude Sonnet 5.5
|
40ac490826 |
Safe restore from a backup: START_PAUSED and the derivation-gap check (B-90, REG-15, OPS-27)
START_PAUSED=true pauses the lottery as the operator before the scheduler exists. On the first Electrum session the 20 derivation indexes past the highest assigned one are looked up on every server; if any has history, registration stays closed (503 registration_unavailable) and restore_gap_detected is written, so a restored database cannot hand a lost user's address to a new one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
2c0e71a74c |
Refuse an Electrum server that is not on the PLM chain (B-99, ELC-30)
PLM shares Bitcoin's genesis and powLimit, so a Bitcoin server passed every header check. connect() now asks server.features and requires PLM's genesis. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
54dbffb552 |
Credit deposits at deposit_confirmations and verify the round's coins with the quorum before paying (B-82, DEP-03, DEP-04, DEP-09, DEP-12, DEP-14, ELC-17, ELC-27, RND-21, RND-22, RND-25, RND-39, DAT-10, DAT-11, DAT-24, DAT-62, DAT-67, DAT-68, AUD-12, AUD-44, AUD-45, API-30, API-31, API-36, ADR-06, ADR-52)
A user could bet and then double-spend the 1-conference deposit that funded the bet with a private chain: the bet left the chain but its participant stayed `confirmed`, and the payout failed "insufficient pool UTXOs" for ever with everyone else's money stuck. Coinbase outputs were also credited at 1 confirmation (maturity is 120). Deposits - `round_config.deposit_confirmations` (default 6, 1-100, editable from /admin) is the threshold for a deposit; an output of a platform transaction (any txid of a pending_transactions row, RBF versions included) keeps 1; a coinbase output needs 120, recognised from the raw transaction, which must hash to the txid (`is_coinbase`, cached). `read_deposit_policy` / `classify_new_deposits` run between the read and the write, so no session is open across the lookup. - The quorum must also place the coin deep enough on our own tip (ELC-17). - Confirmed outputs short of their threshold are pending deposits (memory only, replaced at every refresh): shown in `pending_balance_sats` / `has_pending` (also for DEP-13's withdrawal message), never spendable. No scripthash notification announces a maturing deposit, so each new block refreshes only the users that have one, one task per user (DEP-09 now says so). Payout - `_trigger_payout` verifies the round's confirmed participants' bet outputs with the other servers (`verify_pool_outpoints`: strict majority of >= 2 answers per outpoint, a mempool-only coin or a tie gives no verdict) and builds from exactly those outpoints (RND-22), never from whatever else sits at the address. No quorum: `payout_failed`, `pool verification unavailable`, nothing changes. Bets that are gone become `reorged_out`, one `participant_reorged_out` row each, `pool_amount_sats` rewritten (only downward) in the same transaction, and the 70/30 split is computed on what is left. - Winner's own bet gone: nothing is built; `rounds.payout_blocked_at` (Alembic f2c64a8d1e93, with `deposit_confirmations`, plain add_column) is set, `payout_blocked` and the pause with `paused_reason = 'payout_blocked'` are written in one transaction; `POST /admin/resume` clears the mark and the next attempt verifies again. /admin/rounds returns it. - Rounds opened before B-80 (pool = the shared reserve address) keep the old active-server `listunspent` choice, since that address holds no single round's bets. Ambiguities resolved: the winner's row stays `confirmed` while held (AUD-45 says every participant whose bet is gone becomes `reorged_out`, but OPS-30/DAT-68 need the verification to repeat after the resume, which only a `confirmed` row allows); `/`'s deposit hint no longer says "1 confirmation" (7 languages, same keys); guides updated. Tests changed because the spec changed: listener tests that credited height-100 coins at tip 0 now set a tip past every threshold; the payout fixtures seed confirmed participants (the new source of the inputs) instead of an active-server `listunspent`, FakeNetwork answers the verification, and test_round_pool_address asserts the quorum is asked about the round's address. New: thresholds at exactly 6/5, configured value, coinbase at 119/120 and a mismatching or missing raw tx, platform outputs at 1, dust, quorum height, waiting-deposit display and disappearance, block-triggered refresh, resume, verification verdicts, gone bets excluded with the split recomputed, winner gone held and paused, held payouts untouched until the resume, re-verification after it, no quorum, a payout with 1 and with 400 participants against reserve+fee+dust, admin config bounds and migration. Mutation-checked: threshold 1, ignored coinbase, no height bound, no verification and no winner hold each fail tests. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
db24287ea5 |
Make the draw a commit-reveal at a fixed, confirmed, quorum-agreed height with a public proof (B-73, RND-14 … RND-20, RND-33 … RND-37, DAT-61 … DAT-66, API-25, API-45, AUD-24 … AUD-27, AUD-37, ELC-15, SEC-56, ADR-24, ADR-25)
The seed was the hash of the first block seen after the close, so a miner holding tickets
could withhold blocks until one made them win, the active Electrum server could pick which
real block was used by not announcing another (a failed corroboration even moved the draw to
the next block), and the participant order was the server-only `broadcast_at` with no
tie-break, so nobody could recompute the result.
- Open: every round gets a 32-byte `server_seed`, stored only Fernet-encrypted with the
master-xprv key (`server_seed_enc`), with its published `server_seed_hash`;
`draw_confirmations` is copied from round_config. The plaintext column is written only by
the reveal.
- Entering `drawing` (one transaction, `draw_committed` audit row): participants ordered by
bet txid, each `draw_position`, `participants_hash` (sha256 of `txid:vout:address\n`
lines; new `bet_vout`), `draw_height = tip_at_close + 1`. A tip of 0 keeps the round
`closing` (`draw_baseline_tip_unknown`, once per round). All persisted, so a resumed draw
reads rather than recomputes (the `_tick` branch itself is B-81).
- `_complete_draw` / `_wait_for_draw_block` replace `_wait_for_next_block`: wait for
`draw_confirmations` blocks from `draw_height`, then `ElectrumListener.agreed_block_hash`
asks every server, the active one included, for the header at that exact height (valid
work, >= 2 other answers, strict majority of all); no agreement means the same height is
asked again and `draw_header_corroboration_failed` is written once per distinct reported
hash. `draw_stalled` now counts from the last tip advance, or from when the quorum started
to fail. Seed = sha256(server_seed || block_hash || participants_hash), index = seed mod n.
- The winner, the seed and the reveal (`server_seed`, `winner_drawn` payload) are written in
one transaction; a round closing with no participants reveals too. A round opened before
this change (no commitment) is given a secret as it enters `drawing`.
- `GET /rounds/{id}/proof` (public, 404 `round_not_found`, translated in all 7 languages) and
the new fields on `GET /rounds/current`; `draw_confirmations` in `/admin/config` and the
/admin form; `server_seed_hash`/`draw_height` in `/admin/rounds`, never the secret.
- Alembic d3e8b15a7c42 (plain add_column / create_index, expression indexes untouched).
- Docs: the B-73 `*Stato:*`/`*Oggi:*` lines are gone, guides and CLAUDE.md describe the new
draw. Ambiguities resolved in api.md: the proof's participants also carry `amount_sats`
(RND-37 lists it); `seed` is decimal in the proof and hex in the audit row, as specified.
Tests changed because the spec changed: ElectrumListener.corroborate_header (asked only the
other servers about the block the active one announced) is replaced by agreed_block_hash and
its quorum tests ported; draw_winner takes the secret and the list fingerprint; the four
_wait_for_next_block scheduler tests are replaced by _wait_for_draw_block ones (fixed height,
confirmations, same-height retry, stall clock). New: test_draw_proof.py recomputes the winner
independently from the public proof, checks the plaintext secret is nowhere in the database,
the API or the log before the reveal, and that a second pass over a `drawing` round yields
the same winner. Mutation-checked: ordering by broadcast_at, an early reveal and asking the
tip instead of the draw height each fail tests.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
||
|
davideandClaude Sonnet 5.5
|
82d078e2d6 |
Rotate away from a connected server that stopped following the chain (B-86, ELC-21, ARC-26)
`server.ping` keeps succeeding on a server that is stalled, and nothing compared its tip with the others', so it stayed the active server for ever. - The listener records when the last header was accepted (monotonic clock, a module attribute so tests can move it). - After every successful ping, ElectrumListener._check_not_stalled: once the last accepted header is older than 360s the other servers are asked for their tip (blockchain.headers.subscribe, ELC-14's timeouts); if any reports a higher height than ours the session ends with ServerStalledError and run() rotates to the next server. If none does the chain is just quiet and the session goes on. No existing test changed. ELC-29 and /admin/electrum (the connection state and the age of the last header, B-92) are untouched. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
50b68d2957 |
Follow and record chain reorganisations, pausing on deep ones (B-85, ELC-09, ELC-22, ELC-23, ELC-24, AUD-38, DAT-69)
A reorganisation was neither recorded nor acted upon, and a competing header at the tip's own height was ignored for as long as the tip stood, leaving the listener on an orphan. - A header at the height we hold with a different hash is followed through the same fork-point search as any other reorg (ELC-09) instead of being ignored; it still has to bring valid work and a connected chain. - ElectrumListener._record_reorg writes `chain_reorg_detected` (depth, fork height, old and new tip height and hash) before the window moves, so a failed write ends the session and the next server's announcement of the same fork is recorded then. Depth 1 is a transaction of its own; depth >= 2 also pauses the lottery with paused_reason 'reorg' and writes `lottery_paused` in the same transaction (begin_immediate first). - Docs: ELC-09/22/23/24, AUD-03/11/38, DAT-30/69, OPS-30, RND-04, ADR-26/52 lose their B-85 "da implementare" lines. Tests changed because the spec changed: test_apply_header_ignores_a_competing_header_at_ the_current_tip_height encoded the B-64 behaviour that ELC-09 now reverses; it is replaced by test_a_competing_header_at_the_tip_height_is_a_reorg_to_follow and the depth tests. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
ca50ef62f3 |
Require real proof of work in headers and link every advance to the tip (B-74, ELC-11, ELC-12, ELC-22, ELC-23)
A header only had to meet the target it declared for itself, so a fabricated one with `bits = 0x207fffff` passed half the time, a jump of several blocks was accepted without being linked to the tip (a fake header at a huge height locked `tip_height` and froze every draw), and an ordinary one-block reorganisation closed the session on every server. - draw.py: MAINNET_POW_LIMIT (00000000ffff...ffff); header_meets_its_own_target also refuses a claimed target easier than POW_LIMIT. - ElectrumListener._apply_header is async and takes the active client. A single block must chain by prev_hash; a jump downloads the headers in between (blockchain.block.headers, at most 2016 per request, ElectrumClient.get_block_headers) and each must carry valid work and chain, otherwise HeaderValidationError ends the session. - The listener keeps the last 10 headers (`_window`), built from the process's first header and its 9 predecessors (which must chain into it). A header that does not join the tip is walked back to its fork point inside the window and the server's branch is followed instead of ending the session; a fork older than the window, or a branch that does not end at the announced header, still ends it. The audit row and the pause for deep reorgs are B-85; a competing header at the tip's own height is still ignored until then. - _header_at also serves older heights from the window. Tests changed because the spec changed: the mainnet powLimit cannot be mined in a test, so tests/unit/conftest.py swaps in a regtest limit at import time (the new tests put the real one back); the _apply_header tests are async; the two _run_once tests serve the first header's 9 predecessors; test_apply_header_skips_linkage_check_across_a_height_gap is replaced by the jump tests, since that behaviour is exactly what the finding removed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
a3c51719bd |
Confirm transactions by merkle proof and reinstate coins only on a quorum (B-83, TX-06, TX-22, TX-23, ELC-25, ELC-26, DEP-02, DEP-08, AUD-18, ADR-07)
poll_once marked a transaction confirmed on one server's `height > 0`, and reinstate_reappeared_utxos restored a coin written off as spent outside the platform on the active server's word alone, so a hostile active server could report a never-relayed payout as confirmed (winner unpaid, round closed) or put a spent coin back into a balance. - electrum/merkle.py: pure merkle-branch verification against a block header. - ElectrumListener.confirmation_proven(txid, height): the *other* servers are asked for blockchain.transaction.get_merkle; a strict majority of at least the quorum minimum must return a proof that recomputes to the merkle root of that block's header (block height and position checked too). The header is the tip the listener holds, or for an older block the one a majority of the quorum agrees on and that meets its own target; a height above our tip cannot be proven. Quorum rules are _corroborate_majority's. - poll_once takes the listener as `network`: the active server's history only nominates a candidate (txid, height); the row confirms only when the proof holds, and stays pending otherwise. With no network nothing confirms. ConfirmationPoller wires it. - The reconciler's late-confirmation check of failed rows (B-78) now puts each claimed (txid, height) to the same proof instead of counting histories. - refresh_user splits reinstatement like a credit: find_reinstate_candidates (read-only; same outpoint, recorded amount, confirmed) -> corroborate_utxo_reinstatement (same quorum as a credit) -> reinstate_reappeared_utxos writes only what was corroborated, rechecking the mark under the lock. Nothing is committed in the read phase any more. - Broadcast fan-out, the third part of the finding, shipped with B-76. Tests changed because the spec changed: every poll_once call now passes a network that proves (FakeNetwork) — the confirmation tests are otherwise unchanged; the failed-row "quorum does not confirm" test now asserts the claim goes to the proof. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
a4b160e240 |
Abandon a transaction only on proof, and keep watching failed rows (B-78, TX-15, TX-17, TX-23, TX-26, ELC-28, DAT-33, DAT-36, DAT-69, DAT-72, AUD-11, AUD-12, AUD-31, AUD-39, API-32)
reconcile_once abandoned a row when the *active* server's history lacked its txid:
after 120s for `building` (which a restart onto a different server makes likely),
after 6h for `pending`, and all at once after a forward clock jump. It re-checked the
status but not the txid, so it raced the bumper, and a `failed` row was never looked at
again.
- The reconciler asks every server (ElectrumListener.get_histories, active + others) for
the history of the row's address and looks for every txid of the row. Present on any
server -> promoted (adopting the version found). Absent is only "no server of the
quorum knows it" and needs the active server plus the quorum minimum of the others to
have answered; even then nothing is released without proof:
(a) an input spent in a block by a transaction that is none of the row's versions,
agreed by a strict majority of a quorum of the other servers (confirmed_spender), or
(b) a rebroadcast of the latest raw tx to every server (rebroadcast_everywhere)
refused by all of them, explicitly, for missing/spent inputs.
Accepted or "already known" by any server means alive; a fee refusal is not proof
(`building` -> `pending` for the bumper, `pending` untouched); too few answers or
anything else leaves the row alone. failure_reason names the proof.
- The write re-reads status *and* current_txid under begin_immediate (TX-26).
- pending_transactions.failed_at (set by _abandon and _release_failed_withdrawal) starts
a 14-day watch: a failed row that a majority of the other servers (at least the quorum
minimum) list in a block becomes `confirmed`, writes pending_tx_resurrected once and
pauses the lottery with paused_reason `resurrected_tx`; linked rows are not corrected.
The confirmation is by quorum of histories for now; merkle proofs come with B-83.
- round_config.paused_reason (migration b7d2e41c9a05, with the failed_at one): `operator`
from POST /admin/pause (which accepts an optional `note` and never overwrites an
automatic reason), cleared by resume, which logs the reason it lifted; exposed on
/admin/config and GET /rounds/current. rounds/pause.py:pause_automatically is the
system's way in.
- PendingTransactionReconciler takes the listener as `network`; with none it can promote
but never abandon.
Tests changed because the spec changed: the three reconciler tests that expect an
abandonment (and the one in test_tx_versions) now supply a network that proves the
transaction is gone.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
||
|
davideandClaude Sonnet 5.5
|
2b6ddb019b |
Never roll back an ambiguous broadcast as if it had been refused (B-76, TX-04, TX-05, TX-20, TX-21, BET-16, BET-17, BET-23, WDR-11, WDR-12, WDR-16, API-13)
place_bet and request_withdrawal released everything on any exception from the broadcast, including the 15s timeout and a connection dropped after the write. The tx may have propagated: a bet's sats landed in the pool with no participant, a withdrawal left while its row said failed and its coins were spendable again. - client: ElectrumRpcError (subclass of ElectrumError) is raised only for a reply carrying a JSON-RPC `error`; timeouts, drops and write errors stay plain ElectrumError. - tx/outcome.py:broadcast_with_verdict sorts a broadcast into accepted / refused / uncertain. A refusal is verified first: ElectrumListener.transaction_known asks blockchain.transaction.get of the active server and every other one (10s each) — any server returning the tx means accepted; the active server plus MIN_QUORUM_RESPONSES others all denying it means refused (ALLOW_NO_QUORUM keeps the reduced rule); too few answers is uncertain. "already known" wording is only a shortcut that skips the check. With no network to ask, a refusal is uncertain. - place_bet / request_withdrawal / _trigger_payout use it: only a verified refusal releases; uncertain leaves the rows building for the reconciler, publishes SSE and POST /bets and POST /withdrawals answer 202 with status building; "already known" promotes a payout (payout_sent) like any accepted broadcast. - An accepted transaction is fanned out to every other server (ElectrumListener.fan_out_transaction, best effort, off the caller's path). The RBF replacement in bump_fee is left for B-77, which rewrites that function. Tests changed because the spec changed: the existing "refused broadcast" tests raised a bare RuntimeError (now an uncertain outcome) and are made explicit refusals (rpc_error) with a network that denies the tx; direct route calls pass a Response; FakePayoutListener gains the network surface. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
13cb66cc06 |
Require a real Electrum quorum and allow certificate pinning (B-84, ELC-01, ELC-03, ELC-14, ELC-19, ARC-20, OPS-12)
An empty ELECTRUM_FALLBACK_SERVERS (the .env.example default) silently made every corroboration pass, and with fallbacks the majority was taken over whoever answered, so one reachable fallback decided alone. TLS certificates were never checked. - _corroborate_majority passes only with at least 2 answers (MIN_QUORUM_RESPONSES) and a strict majority of the answers; no other server configured, or fewer than 2 answering, fails. ALLOW_NO_QUORUM=true keeps the old reduced rule. - config.validate_quorum_servers, called from the lifespan after parse_endpoints, refuses to start with fewer than 2 fallback servers unless ALLOW_NO_QUORUM=true (which logs a warning at every start). - Optional pinning: `host:port[:mode]:pin=<sha256>` per fallback and ELECTRUM_CERT_PIN for the primary; ElectrumClient compares the SHA-256 of the DER certificate after the TLS handshake and a mismatch fails like an unreachable server. A pin on a plaintext entry is a startup error. - .env.example now ships the four bootstrap servers and documents both new variables; docs/setup.md, spec, storico and BUGS.md updated. - Tests changed because the spec changed: the two "no other servers configured -> True" corroboration tests now assert that outcome only under allow_no_quorum. Note: a dev .env with fewer than 2 fallback servers will now refuse to start. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
daec4749c2 |
Survive oversized Electrum replies and ignore dust deposits (B-75, ELC-20, DEP-01, DAT-10)
A reply line over asyncio's default 64 KiB StreamReader limit raised out of _read_loop and closed the only connection; the pool address's get_history passes that after ~800 transactions, and dust sent to any public address does it on purpose. - ElectrumClient opens the stream with a 16 MiB limit and frames lines itself: an oversized line is discarded up to its newline, a non-JSON line is dropped, and the request whose id is readable in the first/last 256 bytes fails at once; the connection stays up either way. - Coins below DUST_LIMIT_SATS are never offered as credit candidates, never credited, and select_utxos never selects them. - Spec (ELC-20, DEP-01, DAT-10, ARC), storico, BUGS.md and test counts updated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
c0314e2bf0 |
Never swap the tip's hash sideways, and never split it from its height (B-64)
_apply_header's linkage check only fires on a single-block advance, so a header at the height we already held one for was applied on nothing but its own self-consistency — and that check, as header_meets_its_own_target's own docstring says, a server can satisfy with a self-declared easy target. So the one value the draw is seeded from could be replaced under us at the current height, by a reorg at the tip or by a single server disagreeing with the rest, with no check able to speak to it. Separately, a header carrying no hex set tip_header_hex back to None while advancing tip_height, leaving the two describing different blocks — the exact pairing that function exists to keep. Both are now refused without ending the session, unlike the fabrication cases above them: neither is evidence of a hostile server, and rotating away would cost us the one connection that also credits deposits and broadcasts transactions. - A same-height header is ignored (logged when it actually differs). The hash committed to for a height is not swapped under us; if ours turns out to be the orphan, corroborate_header already refuses to seed a draw from it and the draw waits for a further block. - A hex-less header is ignored outright: nothing to validate, nothing to draw from. A server that only ever pushed heights now freezes the draw — visibly, via B-36's draw_stalled — instead of costing us the connection. Because ignoring is not fatal, _run_once additionally refuses to publish the client when the initial header leaves the tip still unknown, so this cannot reopen B-63's window from the other side. The two _run_once tests are bounded with asyncio.wait_for: without their guard that call waits on session tasks nothing ends, and a regression must fail rather than hang the suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
8a0ebecfcc |
Never let a draw be seeded by a block that predates the close (B-63)
ElectrumListener._run_once assigned self.client before subscribe_headers() returned, so there was a window — one round-trip wide, at process start — where the connection looked alive while tip_height was still its initial 0. "client is not None" is what every consumer reads as "the chain is reachable", RoundScheduler._tick included, and a round closing inside that window recorded tip_at_close = 0. The very first header we then learned about — the current tip, a block mined *before* the round closed, whose hash was already public while bets were still open — satisfied tip_height > tip_at_close and became the draw's entropy. The draw's whole guarantee is that its seed did not exist yet when betting stopped. Two changes, defending different things: - The client is published only once the first header has been applied, so "client is not None" now means "reachable *and* we know where the chain is". During the window consumers see no connection, which is honest: a bet gets the same 503 it already gets while disconnected, and the background tasks skip a cycle as they already do. - _wait_for_next_block treats a baseline of 0 as *unknown*, not as height zero: it adopts the first height it learns as the baseline, waits for a block strictly after it, and records draw_baseline_tip_unknown so the extra block of waiting is explainable from /admin. Unreachable via the listener now, but it is the local statement of what the draw requires, and nothing else in that function would notice if the invariant stopped holding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
6246b13247 |
Corroborate deposit credits, not just external spends (B-59)
A candidate external spend has needed a quorum since B-29, but `value` and `height` for a *credit* came from the single active connection and went straight into utxo_events. One hostile or broken server could therefore inflate a user's displayed balance with outpoints that don't exist. It never spends anyone else's coins — a bet or withdrawal built on a phantom UTXO is refused at broadcast and rolled back — but it wedges the balance display and burns build attempts, and on a custodial platform a balance that isn't real is a support incident either way. Balances move in both directions; both directions now need the same quorum. corroborate_utxo_credit asks the other configured servers whether they report the same outpoint, for the same amount, confirmed. The height itself isn't compared: a server still catching up reports height 0 and simply doesn't agree, which is the same answer, while two honest servers can't disagree on the height of a genuinely confirmed outpoint. refresh_user gains the phase that shape already implied: find_new_credit_ candidates (new, confirmed, not already held) inside the first session, corroboration outside any session, then credit_confirmed_utxos over what survived. Only new outpoints are corroborated — re-checking what we already hold would open a connection to every other server on every refresh for an answer that can no longer change anything. A failed corroboration delays a credit, it never loses one: the next scripthash notification or DepositReconciler sweep (300s) re-offers the same outpoint, and the withholding is logged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
davide
|
4124dc08e6 |
Check confirmation/existence via scripthash history, not verbose replies (B-41)
poll_once and reconcile.py's existence check both called
blockchain.transaction.get(txid, verbose=True). Several Electrum server
implementations and versions reject the verbose flag outright
("verbose transactions are currently unsupported"), which would have
meant no confirmations and no reconciliation ever running against such
a server, read as a plain transport error. reconcile.py additionally
decided whether to abandon a transaction - releasing its funds - by
substring-matching the error text ("missing", "not found", ...), which
only works against ElectrumX's specific wording.
Both now ask blockchain.scripthash.get_history for the address that
owns every input of the transaction (a user's own address for a
bet/withdrawal, the pool address for a payout) and look for the txid in
the result: present with height > 0 means confirmed, present with
height <= 0 means still in the mempool, absent means the server
doesn't know it. get_history is a plain, universally-supported Electrum
method, and "not in the list" replaces the old substring-matching
entirely - no more guessing at error wording to decide whether to
release funds. History is cached per scripthash within one pass, since
every "payout" row shares the same pool address.
New app/tx/pending_address.py factors out own_address_for (the
address derivation was previously duplicated informally inside
tx/broadcast.py's signing context) so confirmation.py and reconcile.py
share one definition instead of two that could compute different
addresses for the same row.
tests/unit/test_confirmation.py and test_reconcile.py needed real User
rows and a master-key bootstrap they didn't have before, since address
derivation is now exercised for real rather than assumed. Suite grows
from 217 to 222 tests. BUGS.md moves B-41 to Previously fixed - no
Medium-severity finding remains open.
|
||
|
davide
|
12df04178e |
Resubscribe concurrently and in the background on reconnect (B-31)
_run_once awaited _subscribe_all_users() inline, before starting the header/scripthash consumer tasks, and that method subscribed one user at a time. At thousands of users that's thousands of sequential round-trips during which nothing else ran: tip_height was frozen and an in-flight draw's _wait_for_next_block made zero progress for the entire resubscribe - a reconnect (which the listener already treats as routine, not exceptional) could stall the lottery for minutes. _subscribe_all_users now fans out with bounded concurrency (asyncio.Semaphore, 20 at a time) instead of a serial loop, and one user's failure no longer stops the rest. _run_once now starts it as its own background task, created after the consumer tasks rather than awaited before them, so tip updates and already-subscribed users' notifications keep flowing throughout - its own completion is deliberately not raced against the session-ending tasks (unlike them, it's expected to finish normally), and its failure is logged the same way address_for_new_user's background task is (B-30). Left out: decoupling the listunspent refresh from the subscribe call itself (the third part of the proposed fix) - the periodic DepositReconciler (B-30) already provides a backstop for a slow or delayed initial refresh, so the added complexity wasn't worth it here. Suite grows from 182 to 185 tests, including an end-to-end test against _run_once proving a new tip is processed while a slow resubscribe is still in flight. BUGS.md moves B-31 to Previously fixed. |
||
|
davide
|
63df38d30b |
Add a periodic deposit reconciler, and stop losing subscribe tasks (B-30)
Deposits were credited exclusively by scripthash-change notifications, with nothing re-verifying a user's balance against the chain if a subscription was ever silently lost. address_for_new_user's subscribe was fire-and-forget: the task wasn't retained, so it could be garbage-collected mid-flight, and any failure (including self.client turning None between the check and the task running) vanished into asyncio's default unretrieved-exception handler instead of being logged anywhere. On an otherwise healthy, long-lived connection there may be no reconnect for days to re-subscribe everyone, so a user in that state never saw their deposits. address_for_new_user now retains the task and logs its exception if it fails. New app/deposits/reconcile.py adds DepositReconciler, a periodic sweep (every 5 minutes, gated on the Electrum client being connected, same shape as tx/reconcile.py) that round-robins over every user and calls the listener's own refresh_user (renamed from _refresh_user since it's now called from outside the class) - so the notification-driven and periodic paths can never behave differently. Deliberately sweeps every user rather than only ones missing from the in-memory scripthash map, since that map can't tell "never subscribed" apart from "subscribed, but the server stopped delivering notifications for it". Wired into app/main.py's lifespan alongside the other three background reconcilers. Suite grows from 176 to 182 tests. BUGS.md moves B-30 to Previously fixed. |
||
|
davide
|
e8fdea0389 |
Corroborate an external spend before marking a UTXO gone (B-29)
detect_external_spends marked a UTXO spent_txid='external-spend' irreversibly the moment it was missing from one listunspent reply, on one server, with no way to undo it. A rotated-to server that's broken or behind, or an empty reply, could zero a user's balance permanently. Split into three functions in deposits/service.py: find_utxos_missing_ from (read-only candidate detection, and refuses to flag anything at all when listunspent comes back entirely empty for a funded address - that reads as a broken response, not a full sweep), mark_utxos_spent_ externally (persistence only, once a candidate is already confirmed), and reinstate_reappeared_utxos (undoes the mark if the outpoint reappears as unspent later). ElectrumListener gains corroborate_utxo_spent, sharing the same majority-quorum logic corroborate_header already uses for B-28: before a candidate is marked, the other configured servers are asked whether they also see it as spent. _refresh_user now reads candidates, then corroborates each one with no DB session held open across those network calls (same shape as B-18/B-25), then persists. Suite grows from 165 to 176 tests. BUGS.md moves B-29 to Previously fixed. |
||
|
davide
|
0ce0562fd7 |
Validate Electrum headers and corroborate the draw's block (B-28)
A single hostile Electrum server, or a MITM on the one active connection, could fabricate the block header the draw's entropy comes from and so pick the winner of every round: headers were accepted with no proof-of-work check and no link to the previous tip. app/rounds/draw.py adds header_meets_its_own_target (rejects a header whose hash doesn't satisfy the difficulty target it claims) and header_prev_hash. electrum/listener.py's _apply_header now rejects a header failing either check by raising HeaderValidationError, which ends the session the same way a dropped connection would so the listener rotates to the next configured server. ElectrumListener gains corroborate_header: before the draw uses a block, it's independently checked against the other configured servers and needs a majority to agree. rounds/scheduler.py's _wait_for_next_block now calls this and, on failure, logs why and waits for a further block instead of ever using an uncorroborated header. Certificate/hostname verification stays disabled, so this doesn't cover an attacker able to MITM every configured server at once - BUGS.md notes that as not covered. Suite grows from 151 to 165 tests. BUGS.md moves B-28 to Previously fixed. |
||
|
davideandClaude Sonnet 5
|
f1a1145cda |
Detect UTXOs spent outside the platform and correct the cached balance
credit_confirmed_utxos only ever credited new UTXOs; a UTXO spent by something other than the app's own bet/withdrawal/payout flow (e.g. someone using the raw derived privkey directly) never got its spent_txid set, so cached_balance_sats kept counting it forever. detect_external_spends mirrors the same listunspent refresh in the other direction: anything still marked unspent in our DB but missing from the address's current unspent set gets spent_txid="external-spend", an audit_log entry, and an immediate balance recompute — wired into the same ElectrumListener._refresh_user call that already runs on every scripthash notification and on listener (re)connect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
7c4e9983ea |
Survive a dropped Electrum connection, and fall back to other servers
A dropped connection used to hang the whole platform permanently, and three defects composed to do it (BUGS.md B-01): The read loop's death was invisible. When the socket closed, _read_loop broke out and finished, but _run_once was blocked on gather() over two notification consumers waiting on queues nobody would ever fill again — it never returned and never raised, so the reconnect-with-backoff logic was unreachable. client.wait_closed() now resolves when the loop ends for any reason, and _run_once races it against the consumers and a keepalive with asyncio.wait(FIRST_COMPLETED). Nothing had a timeout. request() registered a future, wrote to a half-closed socket (drain() often doesn't raise) and awaited a reply that would never come. That hung a POST /bets *while holding the per-user lock*, and could stop the confirmation poller for good. Every request is now bounded at 15s, and a timeout tears the connection down rather than leaving a server that owes us a reply in rotation. There was no keepalive, so on a quiet instance the normal way this connection dies is an idle-timeout drop by the server (~10 minutes for many). A server.ping every 60s makes that observable within a minute. listener.client is also cleared before reconnecting, so callers stop treating a dead connection as live. On top of the finding, the listener now rotates over a list of servers: ELECTRUM_FALLBACK_SERVERS holds comma-separated host:port[:notls] extras, tried after the primary. Everything the platform does goes through this one connection — deposit credits, broadcasts, confirmations, the chain tip the draw waits on — which made a single hardcoded server its biggest point of failure. A failed or dropped session moves to the next server immediately and only sleeps on the backoff once every server has had a turn, so one dead server costs one attempt instead of an outage, while a genuinely offline network still backs off. A malformed entry fails at startup, not during the outage when the fallback is what you need. Also fixes B-19: header handling refuses a height below the current tip and applies height and hex together, since _wait_for_next_block waits for tip_height > tip_at_close (a regression silently added a block to the draw's wait) and that hex is the draw's entropy source, so a mismatched pair would be worse than a stale one. Verified in the live deployment: the log shows the endpoint list, then "Electrum connected to santantonio.sytes.net:50002", and the connection holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5
|
f229f91632 |
Add server-push (SSE) notifications for round/bet/balance state changes
Frontend dashboards previously found out about state changes only on their next poll tick (up to 15s, or 3s during a draw) — this adds a push channel so updates land as soon as they happen instead. - app/rounds/events.py: a small in-process pub/sub (RoundEventBroadcaster). The message carries no payload — it's just a "something changed, go refetch" signal, so it needs no auth and no knowledge of who's allowed to see what; personalization stays entirely in the existing REST endpoints. - GET /rounds/stream: an SSE endpoint exposing that channel, with keep-alive comments so it survives idle periods behind a reverse proxy, and a defensive MAX_SUBSCRIBERS cap (well above the ~100 concurrent users expected) — past it, the endpoint returns 503 instead of opening a stream, and callers just keep working off polling. - publish() calls added at every point that actually changes what a dashboard would want to know: new round opened, round status transitions (closing/drawing/paying_out/closed), a bet or withdrawal broadcast, any pending tx confirming (bet/withdrawal/payout), a deposit credited, and a new block tip arriving (the exact moment the "drawing" phase is waiting on). - Caddyfile: excludes /rounds/stream from gzip encoding, since compression would buffer output and defeat the point of a live stream. Single-process only by design for now (no cross-worker fan-out) and the notification is a generic broadcast rather than a per-user channel — both are deliberate scope decisions for the current ~100-user, single-container deployment, not oversights. Polling is left fully in place as a fallback; this is purely additive. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5
|
a21e058cdd |
Add Electrum SPV client and address listener
Minimal Electrum protocol client (client.py) plus a scripthash subscription listener (listener.py) that watches user deposit addresses for confirmed UTXOs, with the address->scripthash conversion helper and a manual smoke-test script against the dev bootstrap server. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |