50b68d295778b680d6cbfb236e2613f8d9557fd1
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  davideandClaude Sonnet 5.5
|
a3c51719bd |
Confirm transactions by merkle proof and reinstate coins only on a quorum (B-83, TX-06, TX-22, TX-23, ELC-25, ELC-26, DEP-02, DEP-08, AUD-18, ADR-07)
poll_once marked a transaction confirmed on one server's `height > 0`, and reinstate_reappeared_utxos restored a coin written off as spent outside the platform on the active server's word alone, so a hostile active server could report a never-relayed payout as confirmed (winner unpaid, round closed) or put a spent coin back into a balance. - electrum/merkle.py: pure merkle-branch verification against a block header. - ElectrumListener.confirmation_proven(txid, height): the *other* servers are asked for blockchain.transaction.get_merkle; a strict majority of at least the quorum minimum must return a proof that recomputes to the merkle root of that block's header (block height and position checked too). The header is the tip the listener holds, or for an older block the one a majority of the quorum agrees on and that meets its own target; a height above our tip cannot be proven. Quorum rules are _corroborate_majority's. - poll_once takes the listener as `network`: the active server's history only nominates a candidate (txid, height); the row confirms only when the proof holds, and stays pending otherwise. With no network nothing confirms. ConfirmationPoller wires it. - The reconciler's late-confirmation check of failed rows (B-78) now puts each claimed (txid, height) to the same proof instead of counting histories. - refresh_user splits reinstatement like a credit: find_reinstate_candidates (read-only; same outpoint, recorded amount, confirmed) -> corroborate_utxo_reinstatement (same quorum as a credit) -> reinstate_reappeared_utxos writes only what was corroborated, rechecking the mark under the lock. Nothing is committed in the read phase any more. - Broadcast fan-out, the third part of the finding, shipped with B-76. Tests changed because the spec changed: every poll_once call now passes a network that proves (FakeNetwork) — the confirmation tests are otherwise unchanged; the failed-row "quorum does not confirm" test now asserts the claim goes to the proof. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
a4b160e240 |
Abandon a transaction only on proof, and keep watching failed rows (B-78, TX-15, TX-17, TX-23, TX-26, ELC-28, DAT-33, DAT-36, DAT-69, DAT-72, AUD-11, AUD-12, AUD-31, AUD-39, API-32)
reconcile_once abandoned a row when the *active* server's history lacked its txid:
after 120s for `building` (which a restart onto a different server makes likely),
after 6h for `pending`, and all at once after a forward clock jump. It re-checked the
status but not the txid, so it raced the bumper, and a `failed` row was never looked at
again.
- The reconciler asks every server (ElectrumListener.get_histories, active + others) for
the history of the row's address and looks for every txid of the row. Present on any
server -> promoted (adopting the version found). Absent is only "no server of the
quorum knows it" and needs the active server plus the quorum minimum of the others to
have answered; even then nothing is released without proof:
(a) an input spent in a block by a transaction that is none of the row's versions,
agreed by a strict majority of a quorum of the other servers (confirmed_spender), or
(b) a rebroadcast of the latest raw tx to every server (rebroadcast_everywhere)
refused by all of them, explicitly, for missing/spent inputs.
Accepted or "already known" by any server means alive; a fee refusal is not proof
(`building` -> `pending` for the bumper, `pending` untouched); too few answers or
anything else leaves the row alone. failure_reason names the proof.
- The write re-reads status *and* current_txid under begin_immediate (TX-26).
- pending_transactions.failed_at (set by _abandon and _release_failed_withdrawal) starts
a 14-day watch: a failed row that a majority of the other servers (at least the quorum
minimum) list in a block becomes `confirmed`, writes pending_tx_resurrected once and
pauses the lottery with paused_reason `resurrected_tx`; linked rows are not corrected.
The confirmation is by quorum of histories for now; merkle proofs come with B-83.
- round_config.paused_reason (migration b7d2e41c9a05, with the failed_at one): `operator`
from POST /admin/pause (which accepts an optional `note` and never overwrites an
automatic reason), cleared by resume, which logs the reason it lifted; exposed on
/admin/config and GET /rounds/current. rounds/pause.py:pause_automatically is the
system's way in.
- PendingTransactionReconciler takes the listener as `network`; with none it can promote
but never abandon.
Tests changed because the spec changed: the three reconciler tests that expect an
abandonment (and the one in test_tx_versions) now supply a network that proves the
transaction is gone.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
||
|
davideandClaude Sonnet 5.5
|
f936d6660c |
Track every version of a bumped transaction and write the RBF replacement before broadcasting it (B-77, TX-12, TX-13, TX-19, TX-21, DAT-13, DAT-14, DAT-34, DAT-44, DAT-45, DAT-70, DAT-71)
The poller and the reconciler matched current_txid only. If an earlier version of a bumped transaction confirmed it was never seen, and after 6h the row was abandoned: a confirmed bet's participant deleted, a sent withdrawal marked failed, a paid payout rebuilt. bump_fee also broadcast in phase 2 and persisted in phase 3 unconditionally, so a crash or a `database is locked` in between left a transaction on the network that the DB did not know, and an abandon or a confirmation in the meantime was overwritten. - New table pending_transaction_txids (migration a3f5c2d9b7e1; txid unique, index on the owning row; existing rows backfilled with current_txid + raw and the txid their last bump replaced, raw_hex empty). Written with the row by place_bet, request_withdrawal and _trigger_payout; removed with a bet row that a verified refusal deletes (DAT-38). - bump_fee in four phases: read, build and sign, commit the replacement as a candidate version, broadcast (verdict of B-76, fanned out when accepted; an explicit refusal raises RbfError and leaves the row), then switch the row over with `UPDATE ... WHERE status='pending' AND current_txid=<replaced>`. Zero rows matched is a logged conflict: nothing is overwritten and the candidate stays recorded. An uncertain broadcast adopts the replacement. A retry rebuilds the same candidate without tripping the unique txid. - poll_once and reconcile_once look for every txid of a row. When the version found is not the current one, broadcast.adopt_version makes it current (raw tx included) and retargets spent_txid, bet_txid, withdrawals.txid and payout_txid as a bump does, before the confirmation handler runs. _release_inputs frees inputs claimed by any version of the row. - RbfBumper takes the listener as `network`. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
2b6ddb019b |
Never roll back an ambiguous broadcast as if it had been refused (B-76, TX-04, TX-05, TX-20, TX-21, BET-16, BET-17, BET-23, WDR-11, WDR-12, WDR-16, API-13)
place_bet and request_withdrawal released everything on any exception from the broadcast, including the 15s timeout and a connection dropped after the write. The tx may have propagated: a bet's sats landed in the pool with no participant, a withdrawal left while its row said failed and its coins were spendable again. - client: ElectrumRpcError (subclass of ElectrumError) is raised only for a reply carrying a JSON-RPC `error`; timeouts, drops and write errors stay plain ElectrumError. - tx/outcome.py:broadcast_with_verdict sorts a broadcast into accepted / refused / uncertain. A refusal is verified first: ElectrumListener.transaction_known asks blockchain.transaction.get of the active server and every other one (10s each) — any server returning the tx means accepted; the active server plus MIN_QUORUM_RESPONSES others all denying it means refused (ALLOW_NO_QUORUM keeps the reduced rule); too few answers is uncertain. "already known" wording is only a shortcut that skips the check. With no network to ask, a refusal is uncertain. - place_bet / request_withdrawal / _trigger_payout use it: only a verified refusal releases; uncertain leaves the rows building for the reconciler, publishes SSE and POST /bets and POST /withdrawals answer 202 with status building; "already known" promotes a payout (payout_sent) like any accepted broadcast. - An accepted transaction is fanned out to every other server (ElectrumListener.fan_out_transaction, best effort, off the caller's path). The RBF replacement in bump_fee is left for B-77, which rewrites that function. Tests changed because the spec changed: the existing "refused broadcast" tests raised a bare RuntimeError (now an uncertain outcome) and are made explicit refusals (rpc_error) with a network that denies the tx; direct route calls pass a Response; FakePayoutListener gains the network surface. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
fd6a91e93d |
Give every round its own pool address (B-80, RND-35, DAT-60, DAT-11, REG-14, SEC-08, BET-12, RND-22, WDR-14, SEC-43, API-23, API-36)
All bets of all rounds paid one address (m/84'/746'/0'/1/0), so its history grew forever, other rounds' money and stray coins mixed with a round's own, and the payout picked from everything it held. - hd.py: derive_pool_key/derive_pool_address(round_id) on branch 2 (m/84'/746'/0'/2/<round_id>); no argument is the reserve address 1/0. derive_round_pool_key checks the key against the address stored on the round, so a payout can never be signed for money its key does not control. - rounds.pool_address (migration 477bddae1a84, existing rounds backfilled with the reserve address since that is where their bets went; add_column, not a batch alter, so the expression indexes stay): written in the transaction that inserts the round, once its id exists. - place_bet pays round.pool_address; the payout reads, signs and takes change at the round's own address; own_address_for and bump_fee's _signing_context follow the round (confirmation poller and reconciler pass round_id). - Withdrawals to any round pool address, the reserve address or fee_address are refused (withdrawal_to_platform_address, 400, all seven languages). - GET /rounds/current and GET /admin/rounds expose pool_address. - RND-22/DAT-11 stay "da implementare" for B-82: the payout still spends every confirmed coin at the round's address, not only the participants' outpoints (participants carry no bet_vout yet). - tests/unit/conftest.py: every test now gets a throwaway master key. Opening a round derives an address, and tests that merely opened a round were silently reading the real key from the local .env. - Existing tests changed because the spec changed: payout fixtures give their round a pool_address; the reconciler's address-cache test uses two payouts of one round (rows of different rounds no longer share an address); own_address_for now takes the round id. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5.5
|
7ba8e22bb4 |
Real SQLite transactions: BEGIN IMMEDIATE and synchronous=FULL (B-88, DAT-74, DAT-52, ARC-07, TX-28, BET-11, DAT-16)
pysqlite's legacy mode emitted BEGIN only at the first DML, so every read before it ran outside any transaction and a lock wait surfaced as "database is locked" at an arbitrary later statement; place_bet also mistook that error for a write-snapshot conflict, which SQLite cannot produce. - _register_sqlite_pragmas: synchronous=FULL, isolation_level=None and a `begin` event that emits BEGIN / BEGIN IMMEDIATE. - begin_immediate(session): takes the write lock before the first read; called first thing in place_bet, request_withdrawal, the scheduler transitions, _trigger_payout, the confirmation poller, the reconciler, bump_fee's persist phase, deposit crediting / external-spend marking / reinstating, and the derivation-index allocation at registration. - A lock not granted before a broadcast answers service_busy (503, all seven languages), nothing built or sent; after a broadcast nothing is undone, the rows stay `building` for the reconciler. - Spec, storico, BUGS.md and CLAUDE.md updated; 37 new tests on a real file DB. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
666cb1a0c9 |
Retire in-code comments that outlived what they described (B-69)
- app/tx/reconcile.py called the payout retry "a future payout-retry routine — still an open gap". It shipped as B-26: clearing payout_txid leaves the round in exactly the state _retry_payout_if_due picks up, so an abandoned payout rebuilds itself and the log line next to it is an alert, not the recovery path. Reading it the old way, an operator would go hand-fix a round the scheduler was already retrying. - app/db/base.py sized the SQLite busy timeout against "five concurrent background tasks" and then listed only the non-listener ones; the lifespan starts six. - The third item (app/auth/routes.py citing B-31 where it meant B-33) was already correct in the tree; the test pins it so it stays that way. tests/unit/test_code_comments.py derives the task count from the lifespan's own create_task calls rather than restating it, so the comment fails the next time a task is added or removed instead of quietly going stale again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
davide
|
4124dc08e6 |
Check confirmation/existence via scripthash history, not verbose replies (B-41)
poll_once and reconcile.py's existence check both called
blockchain.transaction.get(txid, verbose=True). Several Electrum server
implementations and versions reject the verbose flag outright
("verbose transactions are currently unsupported"), which would have
meant no confirmations and no reconciliation ever running against such
a server, read as a plain transport error. reconcile.py additionally
decided whether to abandon a transaction - releasing its funds - by
substring-matching the error text ("missing", "not found", ...), which
only works against ElectrumX's specific wording.
Both now ask blockchain.scripthash.get_history for the address that
owns every input of the transaction (a user's own address for a
bet/withdrawal, the pool address for a payout) and look for the txid in
the result: present with height > 0 means confirmed, present with
height <= 0 means still in the mempool, absent means the server
doesn't know it. get_history is a plain, universally-supported Electrum
method, and "not in the list" replaces the old substring-matching
entirely - no more guessing at error wording to decide whether to
release funds. History is cached per scripthash within one pass, since
every "payout" row shares the same pool address.
New app/tx/pending_address.py factors out own_address_for (the
address derivation was previously duplicated informally inside
tx/broadcast.py's signing context) so confirmation.py and reconcile.py
share one definition instead of two that could compute different
addresses for the same row.
tests/unit/test_confirmation.py and test_reconcile.py needed real User
rows and a master-key bootstrap they didn't have before, since address
derivation is now exercised for real rather than assumed. Suite grows
from 217 to 222 tests. BUGS.md moves B-41 to Previously fixed - no
Medium-severity finding remains open.
|
||
|
davide
|
08c566d547 |
Restructure bump_fee into three phases, drop float fee math (B-40)
bump_fee issued one get_transaction per input (up to 15s each) and then a broadcast, all with the caller's DB session held open - exactly the pattern already fixed elsewhere for the same reason (B-18's _trigger_payout, B-31's refresh_user). Also, _prevout_amount computed a prevout's satoshi value via round(value_coins * 100_000_000) on a float the server reported, in a codebase that is otherwise strictly integer-satoshi. bump_fee now takes a session_factory and a pending_id instead of a live session and row, with three phases: read what's needed (the signing key, current fee rate, raw tx) and close the session before any network call; do the chain reads, signing and broadcast with no session open; reopen a session only to persist the outcome. _prevout_amount now asks for the raw (non-verbose) transaction and reads embit's parsed TransactionOutput.value directly - already an exact integer, no float conversion involved at all. A pending_transaction that's no longer "pending" by the time bump_fee actually runs (it confirmed in the meantime, a normal race) is now a quiet no-op returning None, rather than being folded into RbfBumper's error-logging path. Suite grows from 214 to 217 tests. BUGS.md moves B-40 to Previously fixed, and trims its own now-stale claim that bump_fee still depended on verbose=True (B-41) - it no longer does. |
||
|
davide
|
17c557b8a3 |
Meet BIP125's relay minimum on every RBF bump, and cap the fee rate (B-32)
bump_fee computed fee_delta as new_fee - old_fee, falling back to a flat 1-satoshi bump whenever that came out zero or negative - which happened whenever old_fee (the actual fee paid, from real prevout amounts) already exceeded the naive target, e.g. because dust change had been folded into the original fee (psbt_builder.py's DUST_LIMIT_SATS handling). A 1-satoshi total increase is nowhere near BIP125 rule 4's minimum (the replacement must pay at least the incremental relay fee rate times its own vsize more than what it replaces), so the node rejected it every time - and since bump_fee raised before touching `pending`, the next tick retried with identical parameters every 30 seconds, forever. Separately, the fee rate climbed by 1 sat/vB every bump with no ceiling. fee_delta is now max(target_fee - old_fee, vsize * the incremental relay rate) - always at least the relay-mandated minimum regardless of what the naive arithmetic produces. pending.fee_rate_sat_vb is set to the actual resulting rate rather than the naive target, so a later bump's arithmetic starts from what's really being paid instead of drifting from it. Once a transaction reaches MAX_FEE_RATE_SAT_VB (a new constant, 10,000 sat/vB, shared with RoundConfig.fee_rate_sat_vb's existing admin-facing bound so the two can't drift apart - the same reason MIN_PASSWORD_LENGTH is shared elsewhere) bump_fee refuses to bump further; the reconciler abandons it if it never confirms (B-27) instead of this retrying forever. Suite grows from 185 to 187 tests. BUGS.md moves B-32 to Previously fixed. |
||
|
davideandClaude Sonnet 5
|
933760e948 |
Decouple the RBF abandon clock from the bump clock (B-27)
bump_fee (tx/broadcast.py) used to overwrite PendingTransaction. broadcast_at on every fee bump, but reconcile.py's abandon-after-N- hours grace period is measured from that same column. A transaction successfully bumped every rbf_timeout_seconds (900s by default) but never mined reset that clock before it could ever reach the 6-hour abandon window, so it was never abandoned: its UTXOs never returned to the user, and if it was a bet the round stayed in "closing" indefinitely. PendingTransaction gains a last_broadcast_at column (migration 861e76aaf34c, backfilled from broadcast_at for existing rows before the NOT NULL constraint is applied). broadcast_at is now never rewritten after creation, so reconcile.py's _is_due keeps measuring from the first broadcast unchanged. bump_fee updates last_broadcast_at instead, and should_bump now reads last_broadcast_at rather than broadcast_at — correct, since whether another bump is due should reset after every bump, unlike the reconciler's abandon check, which must not. BUGS.md moves B-27 to "Previously fixed" with the fix description; the suite grows from 148 to 151 tests, including a direct proof that a tx bumped a minute ago but first broadcast 7 hours ago still gets abandoned. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
davideandClaude Opus 5
|
d528c5b475 |
Let the system recover from a broadcast that never confirms
The code treated a broadcast as final: money moved on-chain and the DB was updated on the assumption it would either confirm or be fee-bumped until it did. Neither is guaranteed, and every way that assumption broke was permanent (BUGS.md B-02, B-03, B-04, B-07, B-08, B-20, B-21). Persist before broadcasting. place_bet and request_withdrawal now write their rows in a "building" state and commit, then broadcast, then promote to broadcast/pending in a second commit. Before, a failure or crash between the broadcast and the commit left the coins irreversibly spent with no trace: no participant (so no entry in the draw), no pending row (so no RBF and no confirmation tracking), and the UTXOs not even marked spent, so the next bet would try to double-spend them. A refused broadcast now releases the reserved UTXOs, restores the balance, removes the participant (or marks the withdrawal failed), audit-logs it, and answers a translatable broadcast_failed — as 502, since the network refused it, not the caller, where it used to be an opaque 500. Reconcile what's in flight against the chain. New PendingTransactionReconciler (app/tx/reconcile.py, every 120s and once at startup) asks whether each non-terminal tx exists: present -> promote, gone -> mark failed with a reason, release the inputs, roll the domain row back, audit-log it. Grace periods differ by state (120s for "building", 6h for "pending", so the RBF bumper gets its attempts first). It is deliberately biased to inaction: only a server that positively doesn't know the tx counts as absent, and a transport failure never abandons anything, because releasing a UTXO whose tx is actually alive would invite a double-spend. Verified against the live server, which answers "No such mempool or blockchain transaction" for an unknown txid. Stop keying on a value that changes. An RBF bump changes the txid, and _on_bet_confirmed looked the participant up by bet_txid — so a bumped bet confirmed under a txid no participant carried, the row stayed "broadcast" forever, and the scheduler waited on it forever: the round could never close and the lottery stopped. Handlers now resolve by immutable ids (round_id/user_id, withdrawal_id), and bump_fee retargets every stored txid — bet_txid, Withdrawal.txid, Round.payout_txid and UtxoEvent.spent_txid — plus records the previous one in replaced_by_txid, which was never written at all. One bad row no longer blocks the rest. The confirmation poller's per-tx lookup is guarded: a txid the server can't resolve used to abort the whole pass, so nothing confirmed again until an operator intervened. It also selects plain columns instead of hydrating entities that outlive their session. Tests: 6 reconciler cases including "a broken connection must not release coins"; the bet-ordering test probes committed state from an independent session during the broadcast, and caught a real mistake in the first draft of this change (the _pending_transaction helper still hardcoded status="pending", so rows were born already-broadcast and would have got the 6-hour grace instead of 120s). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5
|
f229f91632 |
Add server-push (SSE) notifications for round/bet/balance state changes
Frontend dashboards previously found out about state changes only on their next poll tick (up to 15s, or 3s during a draw) — this adds a push channel so updates land as soon as they happen instead. - app/rounds/events.py: a small in-process pub/sub (RoundEventBroadcaster). The message carries no payload — it's just a "something changed, go refetch" signal, so it needs no auth and no knowledge of who's allowed to see what; personalization stays entirely in the existing REST endpoints. - GET /rounds/stream: an SSE endpoint exposing that channel, with keep-alive comments so it survives idle periods behind a reverse proxy, and a defensive MAX_SUBSCRIBERS cap (well above the ~100 concurrent users expected) — past it, the endpoint returns 503 instead of opening a stream, and callers just keep working off polling. - publish() calls added at every point that actually changes what a dashboard would want to know: new round opened, round status transitions (closing/drawing/paying_out/closed), a bet or withdrawal broadcast, any pending tx confirming (bet/withdrawal/payout), a deposit credited, and a new block tip arriving (the exact moment the "drawing" phase is waiting on). - Caddyfile: excludes /rounds/stream from gzip encoding, since compression would buffer output and defeat the point of a live stream. Single-process only by design for now (no cross-worker fan-out) and the notification is a generic broadcast rather than a per-user channel — both are deliberate scope decisions for the current ~100-user, single-container deployment, not oversights. Polling is left fully in place as a fallback; this is purely additive. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5
|
30bde96b6e |
Move all business/round parameters into DB config, out of env entirely
RoundConfig gains round_duration_seconds, round_cooldown_seconds, min_amount_sats, fee_rate_sat_vb and rbf_timeout_seconds (plus a hardcoded default for the pre-existing bet_amount_sats) as column defaults on the model itself — get_round_config no longer seeds from Settings at all. Every call site that read these from settings (scheduler, bets, withdrawals, rounds service/route, RBF bumper) now reads the DB-backed RoundConfig instead. app/config.py now holds only true env-driven infra/secrets (database URL, Electrum connection, master key, JWT, admin token) — no business parameter has an env var anymore, matching an explicit decision to drop the "seed from settings" indirection entirely rather than keep a env fallback nobody should rely on. Migration backfills the existing round_config row via server_default (matching the old settings defaults) then drops the default, so future rows go through the ORM/model defaults instead of a stale constant. Tests updated: should_bump's timeout_seconds is now required (no settings fallback); test_scheduler.py seeds a RoundConfig row directly instead of monkeypatching settings; test_withdrawals.py and test_rounds_service.py use local constants mirroring the model defaults instead of reading them off settings. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
davideandClaude Sonnet 5
|
fc2aadbc7e |
Add transaction broadcast, confirmation polling and RBF fee-bump
Shared per-user locking to serialize bet/withdrawal PSBT builds (tx/locks.py), a confirmation poller for pending outgoing transactions, and the timeout->fee-bump->rebroadcast loop used by bets, payouts and withdrawals alike (tx/broadcast.py). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |