diff --git a/BUGS.md b/BUGS.md index 663f542..b2c6d23 100644 --- a/BUGS.md +++ b/BUGS.md @@ -137,21 +137,6 @@ the intermediate headers (`blockchain.block.headers`) and validate the chain; on linkage mismatch walk back up to 10 headers to find the fork point and follow the reorg instead of ending the session (reorg handling: B-85). -### B-77 — Only the latest txid is tracked, and `bump_fee` broadcasts before it persists - -The poller and the reconciler match `current_txid` only (`replaced_by_txid` keeps -one step and nothing reads it). If an earlier version of a bumped tx confirms, it is -never seen; after 6h the row is abandoned — a confirmed bet's participant deleted, a -sent withdrawal marked failed, a paid payout rebuilt. `bump_fee` broadcasts in phase -2 and persists in phase 3, unconditionally, so a crash or a `database is locked` in -between, or an abandon that ran meanwhile, desynchronises DB and chain. - -Fix: a `pending_transaction_txids` table holding every txid (and raw tx) a row ever -had; a confirmation of *any* of them confirms the row and retargets every reference -to it. The replacement is written (as a candidate) before it is broadcast; phase 3 -updates with `WHERE status='pending' AND current_txid=` and treats zero rows as -a conflict to reconcile. - ### B-78 — Abandonment rests on one server's silence `reconcile_once` abandons when the active server's `get_history` lacks the txid — diff --git a/CLAUDE.md b/CLAUDE.md index a5bd7f5..b0b36dd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ The user communicates in Italian in chat — reply to them in Italian. Everythin ## Project status -All 10 stages of the original build order are code-complete and unit-tested — 463 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. +All 10 stages of the original build order are code-complete and unit-tested — 480 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. Verified on mainnet with real money: registration + address derivation, deposit crediting (1-conf), a real 10 PLM bet (broadcast → confirmed → change credited back), and one full round cycle (close → draw on a real block hash → 70/30 payout with sat math checked against the broadcast tx → confirmation → close → next round auto-opened). **Withdrawal and the RBF bump path have never been exercised against a live broadcast** — unit-tested only. @@ -49,7 +49,7 @@ PYTHONPATH=. python scripts/decrypt_master_key.py # ops recovery: decrypt+pr PYTHONPATH=. python scripts/encrypt_master_key.py # ops bootstrap: import an externally-generated xprv (--overwrite to replace) PYTHONPATH=. python scripts/electrum_smoke_test.py # manual check: connect, handshake, subscribe to headers, print the tip -python -m pytest # all 463 tests +python -m pytest # all 480 tests python -m pytest tests/unit/test_hd.py # one file python -m pytest tests/unit/test_hd.py::test_derivation_is_deterministic # one test ``` @@ -132,7 +132,7 @@ Source of truth: the `PalladiumWallet` repo — [ChainProfiles.cs](../PalladiumW | `app/deposits/` | crediting / external-spend detection / reinstatement (`service.py`), periodic sweep (`reconcile.py`) | | `app/bets/`, `app/withdrawals/` | build+broadcast services and their confirmation handlers | | `app/rounds/` | `scheduler.py` (close/draw/payout), `service.py` (open/active-round rules), `draw.py` (header math + winner pick), `config.py`, `events.py` (SSE pub/sub) | -| `app/tx/` | `broadcast.py` (RBF bumper), `confirmation.py` (poller + handler registry), `reconcile.py`, `locks.py` (per-user locks) | +| `app/tx/` | `broadcast.py` (RBF bumper), `outcome.py` (broadcast verdict, B-76), `versions.py` (every txid of a row, B-77), `confirmation.py` (poller + handler registry), `reconcile.py`, `locks.py` (per-user locks) | | `app/static/` | the two SPAs (`index.html`/`app.js`/`style.css`, `admin.html`/`admin.js`/`admin.css`) + `i18n.js` | ## Background tasks @@ -217,7 +217,7 @@ Everything that spends money is written **before** it is broadcast and resolved **SQLite transactions are real ones (B-88).** `_register_sqlite_pragmas` sets `synchronous=FULL` and switches pysqlite's implicit `BEGIN` off (`isolation_level = None`); an engine `begin` event emits `BEGIN` itself, so every read is inside a transaction. Every transaction that moves money or fixes its state opens with `await begin_immediate(session)` (`app/db/base.py`) — `BEGIN IMMEDIATE`, i.e. the write lock *before* the first read (it commits the caller's read-only transaction first and refuses a session with unflushed changes). A lock not granted within `busy_timeout` raises `OperationalError`: **before** a broadcast that means "refuse" (`service_busy`, 503, nothing built or sent); **after** one nothing is undone — the row stays `building` and the reconciler resolves it. New money-moving write code must call `begin_immediate` first, and tests of that must use a file DB whose engine went through `_register_sqlite_pragmas` (an in-memory engine without it never emits `BEGIN IMMEDIATE`). -`UtxoEvent.spent_txid` must always equal the tx's *current* txid, so `bump_fee` retargets it along with `RoundParticipant.bet_txid`, `Withdrawal.txid` and `Round.payout_txid` on every bump. `broadcast_at` is the *first* broadcast and is never rewritten (the reconciler's abandon clock measures from it); `last_broadcast_at` is what a bump updates and `should_bump` reads. Confirmation handlers key off immutable ids (`round_id`/`user_id`, `withdrawal_id`), never the txid, which changes under them. +`UtxoEvent.spent_txid` must always equal the tx's *current* txid, so `bump_fee` retargets it along with `RoundParticipant.bet_txid`, `Withdrawal.txid` and `Round.payout_txid` on every bump. **Every version a row ever had is tracked (B-77):** `pending_transaction_txids` (`app/tx/versions.py`) holds each txid with its raw tx, written with the row and, for an RBF replacement, *before* it is broadcast; the poller and the reconciler look for all of them, and when the network holds a version other than the current one `broadcast.py:adopt_version` makes it current and retargets everything. `bump_fee` ends with a conditional `UPDATE … WHERE status='pending' AND current_txid=`; zero rows is a logged conflict, never an overwrite. `broadcast_at` is the *first* broadcast and is never rewritten (the reconciler's abandon clock measures from it); `last_broadcast_at` is what a bump updates and `should_bump` reads. Confirmation handlers key off immutable ids (`round_id`/`user_id`, `withdrawal_id`), never the txid, which changes under them. **Payouts retry, and are guarded against paying twice.** Every tick re-examines a `paying_out` round: `_retry_payout_if_due` throttles to one attempt per 60s using the latest `payout_failed` audit entry as its clock (a build failure leaves no DB row to throttle on), and every early return in `_trigger_payout` writes one, so `/admin` shows *why* a round is stuck. Before building, `_trigger_payout` refuses if a `building`/`pending` payout already exists for the round, and `_reserved_payout_outpoints` excludes pool UTXOs claimed by any unresolved payout — without both, a retry would pay the winner twice. diff --git a/README.md b/README.md index cbfc6d9..67b8413 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ python -m pytest # all tests python -m pytest tests/unit/test_hd.py # one file ``` -463 unit tests cover HD derivation, PSBT building, the Electrum client, bets, +480 unit tests cover HD derivation, PSBT building, the Electrum client, bets, deposits, withdrawals, the round/draw engine, RBF fee-bumping, admin config, the pending-inclusive balance calculation, and the SSE push channel. No automated integration tests against a live Electrum connection — mainnet diff --git a/app/bets/service.py b/app/bets/service.py index 9132e1f..a6c2188 100644 --- a/app/bets/service.py +++ b/app/bets/service.py @@ -14,6 +14,7 @@ from app.electrum.client import ElectrumClient from app.rounds.config import get_round_config from app.rounds.events import broadcaster from app.rounds.service import open_new_round_if_needed, round_accepts_bets +from app.tx.versions import delete_versions, record_version from app.tx.outcome import BroadcastOutcome, TransactionNetwork, broadcast_with_verdict from app.wallet.balance import recompute_balance from app.wallet.hd import derive_user_key @@ -148,6 +149,7 @@ async def place_bet( session.add(participant) pending = _pending_transaction(round_.id, user.id, built, config.fee_rate_sat_vb) session.add(pending) + await record_version(session, pending, built.txid, built.raw_hex) # B-77: the first version, with the row # B-53: the deadline check at the top of this function happened before the UTXO # scan and the signing above, so re-check it here against the clock as it is now — @@ -258,6 +260,7 @@ async def _release_failed_bet( if row is not None: row.spent_txid = None await session.delete(participant) + await delete_versions(session, pending.id) # B-77 (DAT-38): the versions go with the row await session.delete(pending) await recompute_balance(session, user_id) await write_audit_log( diff --git a/app/db/models.py b/app/db/models.py index fa31a2b..96ca293 100644 --- a/app/db/models.py +++ b/app/db/models.py @@ -200,6 +200,27 @@ class PendingTransaction(Base): updated_at: Mapped[datetime] = mapped_column(default=utcnow, onupdate=utcnow) +class PendingTransactionTxid(Base): + """Every version a PendingTransaction has ever had (B-77, DAT-70, DAT-71): the first + one, each RBF replacement and each candidate written before a bump is broadcast. + + The network may confirm *any* of them — the original while a bump was in flight, a + replacement the process crashed before recording — so the poller and the reconciler + look for every txid of a row, not only `current_txid`. Rows are never modified or + deleted (except with the `bet` row `_release_failed_bet` removes); `txid` is unique + over the whole table, so a txid belongs to exactly one platform transaction. + `raw_hex` is empty only for a `replaced_by_txid` reconstructed by the migration, + whose raw transaction is no longer available.""" + + __tablename__ = "pending_transaction_txids" + + id: Mapped[int] = mapped_column(primary_key=True) + pending_transaction_id: Mapped[int] = mapped_column(ForeignKey("pending_transactions.id"), index=True) + txid: Mapped[str] = mapped_column(String(64), unique=True) + raw_hex: Mapped[str] = mapped_column(Text) + created_at: Mapped[datetime] = mapped_column(default=utcnow) + + class Withdrawal(Base): __tablename__ = "withdrawals" diff --git a/app/main.py b/app/main.py index 021297d..fd612d1 100644 --- a/app/main.py +++ b/app/main.py @@ -63,7 +63,7 @@ async def lifespan(app: FastAPI): scheduler = RoundScheduler(AsyncSessionLocal, listener) poller = ConfirmationPoller(AsyncSessionLocal, lambda: listener.client) - bumper = RbfBumper(AsyncSessionLocal, lambda: listener.client) + bumper = RbfBumper(AsyncSessionLocal, lambda: listener.client, network=listener) # Resolves in-flight transactions against the chain — the piece that lets the # system recover on its own from a broadcast that never confirmed (B-04/B-08). reconciler = PendingTransactionReconciler(AsyncSessionLocal, lambda: listener.client) diff --git a/app/rounds/scheduler.py b/app/rounds/scheduler.py index eb51589..5a1074f 100644 --- a/app/rounds/scheduler.py +++ b/app/rounds/scheduler.py @@ -18,6 +18,7 @@ from app.rounds.draw import draw_winner, header_hex_to_block_hash from app.rounds.events import broadcaster from app.rounds.service import open_new_round_if_needed, round_deadline, winner_share from app.tx.outcome import BroadcastOutcome, broadcast_with_verdict +from app.tx.versions import record_version from app.wallet.hd import derive_round_pool_key from app.wallet.plm_network import PLM_MAINNET from app.wallet.psbt_builder import InsufficientFundsError, Utxo, build_payout_transaction @@ -493,6 +494,7 @@ class RoundScheduler: status="building", ) session.add(pending) + await record_version(session, pending, built.txid, built.raw_hex) # B-77: the first version await session.commit() pending_id = pending.id diff --git a/app/tx/broadcast.py b/app/tx/broadcast.py index 294e49c..50ef066 100644 --- a/app/tx/broadcast.py +++ b/app/tx/broadcast.py @@ -6,13 +6,23 @@ from embit import script from embit.psbt import PSBT from embit.transaction import Transaction, TransactionInput, TransactionOutput from embit.finalizer import finalize_psbt -from sqlalchemy import select +from sqlalchemy import select, update from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker from app.db.base import begin_immediate -from app.db.models import PendingTransaction, Round, RoundParticipant, User, UtxoEvent, Withdrawal +from app.db.models import ( + PendingTransaction, + PendingTransactionTxid, + Round, + RoundParticipant, + User, + UtxoEvent, + Withdrawal, +) from app.electrum.client import ElectrumClient from app.rounds.config import get_round_config +from app.tx.outcome import BroadcastOutcome, TransactionNetwork, broadcast_with_verdict +from app.tx.versions import record_version from app.wallet.hd import derive_round_pool_key, derive_user_key from app.wallet.plm_network import PLM_MAINNET from app.wallet.psbt_builder import MAX_FEE_RATE_SAT_VB, RBF_SEQUENCE, estimate_vsize @@ -93,28 +103,42 @@ def _find_change_output(tx: Transaction, change_address: str) -> int | None: async def bump_fee( - session_factory: async_sessionmaker, client: ElectrumClient, pending_id: int + session_factory: async_sessionmaker, + client: ElectrumClient, + pending_id: int, + network: TransactionNetwork | None = None, ) -> str | None: """Rebuild pending_transaction `pending_id`'s transaction with a higher fee (same inputs, same recipient outputs, the extra fee taken from the change output) and rebroadcast. Returns the new txid, or None if there was nothing to do (the row is gone or already left "pending" — a normal race with - confirmation, not an error). + confirmation, not an error, or a conflict recorded below). - Three phases, so no DB session is held across the network calls this needs + Four phases, so no DB session is held across the network calls this needs (one get_transaction per input, then a broadcast) — the same shape used elsewhere for exactly this reason (B-18, rounds/scheduler.py:_trigger_payout; B-31, electrum/listener.py:refresh_user) and now here too (B-40): read what's needed and close the session, do the chain work, then reopen to persist. + B-77 (TX-12, TX-13, DAT-34): the replacement is **written before it is broadcast**. + It used to be sent in the chain phase and recorded afterwards, unconditionally, so + a crash or a `database is locked` in between left a transaction on the network + that the database did not know, and an abandon or a confirmation that happened in + the meantime was silently overwritten. Now the signed replacement is committed as + a *candidate* version (pending_transaction_txids) first; a candidate that is never + broadcast is harmless, and one that is — and then confirms — is recognised like any + other version. After the broadcast the row is switched over with a conditional + UPDATE (status still pending, current_txid still the one replaced); zero rows + matched is a conflict: the row is left alone, the candidate stays recorded, and + the reconciler resolves it. + Only handles the common case: exactly one change output paying back to the tx's own sender address, large enough to absorb the increase. If there's no such output (e.g. an exact-amount bet with no change), this raises RbfError — bumping such a tx would require selecting additional inputs, which isn't implemented for the MVP; it needs manual operator intervention. Also raises RbfError, rather than bumping, once the row is already at MAX_FEE_RATE_SAT_VB - (B-32) — the reconciler abandons it if it never confirms (B-27), instead of - this retrying an ever-higher fee forever. + (B-32), and when the network explicitly refuses the replacement (TX-24). """ # --- Phase 1: read what's needed, close the session before any network call --- async with session_factory() as session: @@ -131,9 +155,10 @@ async def bump_fee( kind = pending.kind current_fee_rate = pending.fee_rate_sat_vb raw_tx_hex = pending.raw_tx_hex + old_txid = pending.current_txid signing_key, own_script, own_address = await _signing_context(session, kind, pending.user_id, pending.round_id) - # --- Phase 2: chain reads, signing, and the broadcast — no DB session open ---- + # --- Phase 2: chain reads and signing — no DB session open, nothing sent yet --- old_tx = Transaction.parse(bytes.fromhex(raw_tx_hex)) input_amounts = [await _prevout_amount(client, vin) for vin in old_tx.vin] total_in = sum(input_amounts) @@ -178,27 +203,69 @@ async def bump_fee( raw_hex = final_tx.serialize().hex() new_txid = final_tx.txid().hex() - await client.broadcast(raw_hex) - # --- Phase 3: persist the outcome ---------------------------------------------- + # --- Phase 3: record the replacement as a candidate, *then* broadcast ---------- async with session_factory() as session: await begin_immediate(session) # B-88 pending = await session.get(PendingTransaction, pending_id) - old_txid = pending.current_txid - pending.replaced_by_txid = old_txid # points backwards: what current_txid replaced - pending.current_txid = new_txid - pending.raw_tx_hex = raw_hex + if pending is None or pending.status != "pending" or pending.current_txid != old_txid: + logger.info("pending_transaction %s changed while its bump was being built; skipping", pending_id) + return None + await record_version(session, pending, new_txid, raw_hex) # B-77 (TX-12, DAT-71) + await session.commit() + + verdict = await broadcast_with_verdict(client, network, raw_hex, new_txid) # B-76: also fans out + if verdict.outcome is BroadcastOutcome.REFUSED: + # The network explicitly refused the replacement and no server knows it: the row + # is unchanged and keeps the version that is on the network (TX-24). The + # candidate stays recorded — it was never relayed, so nobody will confirm it. + raise RbfError(f"pending_transaction {pending_id}: the network refused the replacement: {verdict.reason}") + # ACCEPTED, or UNCERTAIN: the replacement may well be on the network, so the row + # follows it (TX-12); the old version stays recorded and is still recognised should + # it be the one that confirms. + + # --- Phase 4: switch the row over, conditionally -------------------------------- + async with session_factory() as session: + await begin_immediate(session) # B-88 # The *actual* resulting rate, not target_fee_rate: when the BIP125-minimum # floor above raised fee_delta past the naive target, the tx now pays more # than target_fee_rate implied. Recording the true rate keeps the next bump's # arithmetic honest instead of drifting from what's really being paid. - pending.fee_rate_sat_vb = (old_fee + fee_delta) // vsize - pending.attempt_count += 1 # last_broadcast_at, not broadcast_at (B-27): broadcast_at must stay the *first* # broadcast, since reconcile.py's abandon-after-N-hours grace period is measured # from it — overwriting it here used to reset that clock on every bump, so a # repeatedly-bumped-but-never-mined tx was never abandoned. - pending.last_broadcast_at = datetime.now(timezone.utc) + result = await session.execute( + update(PendingTransaction) + .where( + PendingTransaction.id == pending_id, + PendingTransaction.status == "pending", + PendingTransaction.current_txid == old_txid, + ) + .values( + replaced_by_txid=old_txid, # points backwards: what current_txid replaced + current_txid=new_txid, + raw_tx_hex=raw_hex, + fee_rate_sat_vb=(old_fee + fee_delta) // vsize, + attempt_count=PendingTransaction.attempt_count + 1, + last_broadcast_at=datetime.now(timezone.utc), + ) + .execution_options(synchronize_session=False) + ) + if result.rowcount != 1: + # Confirmed, abandoned or already re-bumped while we were broadcasting. The + # row is not overwritten; the candidate is still recorded, and confirmation + # and reconciliation look for it like any other version (TX-13). + await session.rollback() + logger.warning( + "bump of pending_transaction %s: the row changed during the broadcast (replaced %s by %s); " + "left to the reconciler", + pending_id, + old_txid, + new_txid, + ) + return None + pending = await session.get(PendingTransaction, pending_id) await _retarget_txid_references(session, pending, old_txid, new_txid) await session.commit() @@ -206,6 +273,29 @@ async def bump_fee( return new_txid +async def adopt_version(session: AsyncSession, pending: PendingTransaction, txid: str) -> None: + """The network confirmed (or holds) a version of this row other than the current one + — an earlier version of a bumped transaction, or a candidate whose switch-over never + happened (B-77, TX-19, DAT-14). `current_txid` and `raw_tx_hex` become that + version's, and the txid is propagated everywhere it is recorded, exactly as after a + bump; without it the inputs of a replaced transaction would no longer be + recognised as the row's (DAT-13).""" + if txid == pending.current_txid: + return + version = await session.scalar( + select(PendingTransactionTxid).where( + PendingTransactionTxid.txid == txid, PendingTransactionTxid.pending_transaction_id == pending.id + ) + ) + old_txid = pending.current_txid + pending.replaced_by_txid = old_txid + pending.current_txid = txid + if version is not None and version.raw_hex: + pending.raw_tx_hex = version.raw_hex # empty only for a migration-reconstructed txid + await _retarget_txid_references(session, pending, old_txid, txid) + logger.warning("pending_transaction %s: version %s found on the network, replacing %s", pending.id, txid, old_txid) + + async def _retarget_txid_references( session: AsyncSession, pending: PendingTransaction, old_txid: str, new_txid: str ) -> None: @@ -247,9 +337,10 @@ async def _retarget_txid_references( class RbfBumper: - def __init__(self, session_factory: async_sessionmaker, get_client): + def __init__(self, session_factory: async_sessionmaker, get_client, network: TransactionNetwork | None = None): self._session_factory = session_factory self._get_client = get_client + self._network = network # B-76/B-77: verifies and fans out a replacement async def run(self) -> None: while True: @@ -274,7 +365,7 @@ class RbfBumper: for pending_id in due_ids: try: - await bump_fee(self._session_factory, client, pending_id) + await bump_fee(self._session_factory, client, pending_id, network=self._network) except RbfError: logger.exception("could not bump pending_transaction %s", pending_id) except Exception: diff --git a/app/tx/confirmation.py b/app/tx/confirmation.py index 99adb43..8df3dca 100644 --- a/app/tx/confirmation.py +++ b/app/tx/confirmation.py @@ -10,7 +10,9 @@ from app.db.models import PendingTransaction from app.electrum.client import ElectrumClient from app.electrum.scripthash import address_to_scripthash from app.rounds.events import broadcaster +from app.tx.broadcast import adopt_version from app.tx.pending_address import own_address_for +from app.tx.versions import txids_by_row logger = logging.getLogger(__name__) @@ -47,6 +49,9 @@ async def poll_once(session_factory: async_sessionmaker, client: ElectrumClient) # by scripthash below — the payout rows of one round share its pool # address (B-80), so this also avoids asking the server the same history twice. scripthash_by_id: dict[int, str] = {} + # B-77 (TX-06, TX-19): every txid a row ever had, not only the current one — an + # earlier version of a bumped transaction may be the one that confirms. + txids_by_id = await txids_by_row(session, {pending_id: txid for pending_id, txid, *_ in candidates}) for pending_id, _txid, kind, user_id, round_id in candidates: try: address = await own_address_for(session, kind, user_id, round_id) @@ -74,11 +79,15 @@ async def poll_once(session_factory: async_sessionmaker, client: ElectrumClient) logger.warning("could not fetch history for pending_transaction %s (txid %s)", pending_id, txid, exc_info=True) continue - entry = next((e for e in history_cache[scripthash] if e.get("tx_hash") == txid), None) + known_txids = txids_by_id[pending_id] # height > 0 means confirmed at that height; 0 or absent means still in # the mempool (or the server doesn't know this txid at all yet) — either # way, not confirmed, so keep waiting. - if entry is None or entry.get("height", 0) <= 0: + entry = next( + (e for e in history_cache[scripthash] if e.get("tx_hash") in known_txids and e.get("height", 0) > 0), + None, + ) + if entry is None: continue async with session_factory() as session: @@ -86,6 +95,10 @@ async def poll_once(session_factory: async_sessionmaker, client: ElectrumClient) row = await session.get(PendingTransaction, pending_id) if row is None or row.status != "pending": continue + if entry["tx_hash"] != row.current_txid: + # The confirmed version is not the current one (B-77, DAT-14): realign + # the row and everything that recorded its txid *before* the handler runs. + await adopt_version(session, row, entry["tx_hash"]) row.status = "confirmed" handler = _handlers.get(kind) if handler is not None: diff --git a/app/tx/reconcile.py b/app/tx/reconcile.py index 22e0019..39db0aa 100644 --- a/app/tx/reconcile.py +++ b/app/tx/reconcile.py @@ -39,7 +39,9 @@ from app.db.models import PendingTransaction, Round, RoundParticipant, UtxoEvent from app.electrum.client import ElectrumClient from app.electrum.scripthash import address_to_scripthash from app.rounds.events import broadcaster +from app.tx.broadcast import adopt_version from app.tx.pending_address import own_address_for +from app.tx.versions import txids_by_row, txids_of from app.wallet.balance import recompute_balance logger = logging.getLogger(__name__) @@ -98,6 +100,8 @@ async def reconcile_once(session_factory: async_sessionmaker, client: ElectrumCl ) ).all() candidates = [] + # B-77 (TX-15, TX-19): a row's every txid, not only the current one. + txids_by_id = await txids_by_row(session, {row.id: row.current_txid for row in rows}) for row in rows: if not _is_due(row, now): continue @@ -107,15 +111,16 @@ async def reconcile_once(session_factory: async_sessionmaker, client: ElectrumCl except Exception: logger.exception("could not derive the address for pending_transaction %s", row.id) continue - candidates.append((row.id, row.status, row.current_txid, scripthash)) + candidates.append((row.id, row.status, row.current_txid, scripthash, txids_by_id[row.id])) resolved = 0 history_cache: dict[str, list[dict]] = {} - for row_id, status, txid, scripthash in candidates: + for row_id, status, txid, scripthash, known_txids in candidates: try: if scripthash not in history_cache: history_cache[scripthash] = await client.get_history(scripthash) - exists = any(entry.get("tx_hash") == txid for entry in history_cache[scripthash]) + found = [entry["tx_hash"] for entry in history_cache[scripthash] if entry.get("tx_hash") in known_txids] + exists = bool(found) except Exception: # Transport/server problem — say nothing about this tx and try again on # the next pass rather than abandoning a tx that may be perfectly alive. @@ -128,6 +133,11 @@ async def reconcile_once(session_factory: async_sessionmaker, client: ElectrumCl if row is None or row.status != status: continue # something else moved it while we were asking if exists: + if txid not in found: + # Another version of the row is the one on the network (B-77, TX-19): + # the row follows it, so the inputs and the domain row point at it. + await adopt_version(session, row, found[0]) + resolved += 1 if row.status == "building": await _promote(session, row) resolved += 1 @@ -183,7 +193,7 @@ async def _abandon(session: AsyncSession, row: PendingTransaction, reason: str) row.status = "failed" row.failure_reason = reason[:128] - released = await _release_inputs(session, row) + released = await _release_inputs(session, row, await txids_of(session, row)) if row.kind == "bet": participant = await session.scalar( @@ -241,7 +251,7 @@ async def _abandon(session: AsyncSession, row: PendingTransaction, reason: str) ) -async def _release_inputs(session: AsyncSession, row: PendingTransaction) -> int: +async def _release_inputs(session: AsyncSession, row: PendingTransaction, own_txids: set[str] | None = None) -> int: """Clear spent_txid on every UTXO this transaction consumed, so the balance counts them again. The inputs come from the stored raw tx, which is kept current across RBF bumps, so this works for a bumped tx too.""" @@ -257,8 +267,9 @@ async def _release_inputs(session: AsyncSession, row: PendingTransaction) -> int select(UtxoEvent).where(UtxoEvent.txid == vin.txid.hex(), UtxoEvent.vout == vin.vout) ) # Only release what this tx actually reserved: if another tx has since spent - # the same UTXO, its claim is the live one and must not be cleared. - if utxo is not None and utxo.spent_txid == row.current_txid: + # the same UTXO, its claim is the live one and must not be cleared. "This tx" + # is any version of the row (B-77, DAT-13), not only its current one. + if utxo is not None and utxo.spent_txid in (own_txids or {row.current_txid}): utxo.spent_txid = None released += 1 return released diff --git a/app/tx/versions.py b/app/tx/versions.py new file mode 100644 index 0000000..e884e65 --- /dev/null +++ b/app/tx/versions.py @@ -0,0 +1,53 @@ +"""The versions of a pending transaction (B-77, DAT-70, DAT-71, TX-19). + +A bumped transaction has several txids and any of them may be the one that confirms. +Everything that looks a transaction up on the chain asks for all of them.""" + +from sqlalchemy import delete, select +from sqlalchemy.ext.asyncio import AsyncSession + +from app.db.models import PendingTransaction, PendingTransactionTxid + + +async def record_version(session: AsyncSession, pending: PendingTransaction, txid: str, raw_hex: str) -> None: + """Write `txid` as a version of `pending` in the caller's transaction. Idempotent for + the same pair (a bump retried after an uncertain broadcast rebuilds the same + transaction); a txid already held by another row is an error — it is unique across + the platform.""" + if pending.id is None: + await session.flush() + existing = await session.scalar(select(PendingTransactionTxid).where(PendingTransactionTxid.txid == txid)) + if existing is not None: + if existing.pending_transaction_id != pending.id: + raise ValueError(f"txid {txid} already belongs to pending_transaction {existing.pending_transaction_id}") + return + session.add(PendingTransactionTxid(pending_transaction_id=pending.id, txid=txid, raw_hex=raw_hex)) + + +async def txids_of(session: AsyncSession, pending: PendingTransaction) -> set[str]: + """Every txid of the row: all its recorded versions plus `current_txid` (a row + written before the table existed, or by a test, may have no version row for it).""" + rows = await session.scalars( + select(PendingTransactionTxid.txid).where(PendingTransactionTxid.pending_transaction_id == pending.id) + ) + return {pending.current_txid, *rows.all()} + + +async def txids_by_row(session: AsyncSession, current: dict[int, str]) -> dict[int, set[str]]: + """`txids_of` for many rows at once; `current` maps row id -> current_txid.""" + result = {row_id: {txid} for row_id, txid in current.items()} + if not result: + return result + rows = await session.execute( + select(PendingTransactionTxid.pending_transaction_id, PendingTransactionTxid.txid).where( + PendingTransactionTxid.pending_transaction_id.in_(list(result)) + ) + ) + for row_id, txid in rows.all(): + result[row_id].add(txid) + return result + + +async def delete_versions(session: AsyncSession, pending_id: int) -> None: + """The versions go with a `bet` row that a verified refusal deletes (DAT-38).""" + await session.execute(delete(PendingTransactionTxid).where(PendingTransactionTxid.pending_transaction_id == pending_id)) diff --git a/app/withdrawals/service.py b/app/withdrawals/service.py index 88f400b..fe8e836 100644 --- a/app/withdrawals/service.py +++ b/app/withdrawals/service.py @@ -12,6 +12,7 @@ from app.db.models import PendingTransaction, Round, User, UtxoEvent, Withdrawal from app.electrum.client import ElectrumClient from app.rounds.config import get_round_config from app.rounds.events import broadcaster +from app.tx.versions import record_version from app.tx.outcome import BroadcastOutcome, TransactionNetwork, broadcast_with_verdict from app.wallet.address import is_valid_plm_address from app.wallet.balance import compute_pending_balance, recompute_balance @@ -166,6 +167,7 @@ async def request_withdrawal( status="building", ) session.add(pending) + await record_version(session, pending, built.txid, built.raw_hex) # B-77: the first version, with the row await session.commit() # B-76 (TX-04, TX-20): only a verified refusal releases the coins. A timeout, a diff --git a/docs/adr/ADR-31-tutto-cio-spende-denaro-scrive-prima.md b/docs/adr/ADR-31-tutto-cio-spende-denaro-scrive-prima.md index 428a309..e0e6cfb 100644 --- a/docs/adr/ADR-31-tutto-cio-spende-denaro-scrive-prima.md +++ b/docs/adr/ADR-31-tutto-cio-spende-denaro-scrive-prima.md @@ -1,6 +1,6 @@ # ADR-31 — Tutto ciò che spende denaro si scrive prima di inviarlo -- **Stato:** Accettata — implementazione da completare (B-77) +- **Stato:** Accettata - **Area:** Denaro in movimento - **Fonte di verità per:** la scelta di progetto descritta qui sotto; i requisiti (`PREFISSO-NN`) che la attuano stanno nei documenti di dominio, indicati nelle *Conseguenze*. @@ -20,5 +20,4 @@ riconciliatore e risponde **202** ([ADR-51](ADR-51-dubbio-liberano-mai-fondi.md) *Codice:* `app/bets/service.py:place_bet` · `app/withdrawals/service.py:request_withdrawal` · `app/rounds/scheduler.py:RoundScheduler._trigger_payout` · `app/tx/broadcast.py:bump_fee` · `app/tx/reconcile.py:reconcile_once` · *Test:* `tests/unit/test_bets.py::test_bet_is_persisted_before_it_is_broadcast`, `tests/unit/test_broadcast_outcome.py`, `tests/unit/test_scheduler.py::test_trigger_payout_persists_before_broadcasting`, `tests/unit/test_reconcile.py` -*Stato:* da implementare (B-77), per le sostituzioni RBF · *Oggi:* -`bump_fee` invia la sostituzione prima di salvarla. + diff --git a/docs/adr/ADR-51-dubbio-liberano-mai-fondi.md b/docs/adr/ADR-51-dubbio-liberano-mai-fondi.md index 9ba0ca2..d89288f 100644 --- a/docs/adr/ADR-51-dubbio-liberano-mai-fondi.md +++ b/docs/adr/ADR-51-dubbio-liberano-mai-fondi.md @@ -1,6 +1,6 @@ # ADR-51 — Nel dubbio non si liberano mai i fondi -- **Stato:** Accettata — implementazione da completare (B-77, B-78) +- **Stato:** Accettata — implementazione da completare (B-78) - **Area:** Denaro in movimento - **Fonte di verità per:** la scelta di progetto descritta qui sotto; i requisiti (`PREFISSO-NN`) che la attuano stanno nei documenti di dominio, indicati nelle *Conseguenze*. @@ -37,7 +37,7 @@ pausa la lotteria ([ADR-28](ADR-28-pausa-manutenzione-ferma-round-successivo.md) *Codice:* `app/tx/reconcile.py:reconcile_once` · `app/tx/reconcile.py:_is_due` · `app/bets/service.py:place_bet` · `app/withdrawals/service.py:request_withdrawal` · `app/tx/confirmation.py:poll_once` · `app/tx/broadcast.py:bump_fee` · *Test:* `tests/unit/test_broadcast_outcome.py` -*Stato:* da implementare (B-77, B-78) · *Oggi:* l'abbandono avviene -quando la cronologia del solo server attivo non contiene il txid corrente, -dopo 120s per `building` e 6 ore per `pending`; si segue solo l'ultimo txid; -le righe `failed` non sono più controllate. +*Stato:* da implementare (B-78) · *Oggi:* l'abbandono avviene +quando la cronologia del solo server attivo non contiene alcun txid della +riga, dopo 120s per `building` e 6 ore per `pending`; le righe `failed` non +sono più controllate. diff --git a/docs/architettura.md b/docs/architettura.md index 211687f..f8e5dcd 100644 --- a/docs/architettura.md +++ b/docs/architettura.md @@ -564,8 +564,8 @@ implementare» non esistono ancora nel codice (vedi [ARC-60] e [ARC-22]). ([LIM-01](limiti-noti.md#robustezza-del-ciclo-di-vita)). - **[ARC-28]** **`ConfirmationPoller`** (10s), **`RbfBumper`** (30s) e **`PendingTransactionReconciler`** (al primo giro subito, poi ogni 120s) - lavorano sulla tabella `pending_transactions` (e, dal piano, sulla tabella - di tutti i txid di ogni riga, B-77); il riconciliatore parte subito perché + lavorano sulla tabella `pending_transactions` (e sulla tabella + di tutti i txid di ogni riga, `pending_transaction_txids`); il riconciliatore parte subito perché un crash durante un invio lascia proprio una riga `building` da risolvere al riavvio. Regole in [transazioni-e-riconciliazione.md](transazioni-e-riconciliazione.md). diff --git a/docs/connessione-electrum.md b/docs/connessione-electrum.md index 5884f25..ec2254f 100644 --- a/docs/connessione-electrum.md +++ b/docs/connessione-electrum.md @@ -349,8 +349,7 @@ L'**invio** di una transazione non è una corroborazione ma usa gli stessi server: dopo che il server attivo l'ha accettata, la transazione grezza viene inviata anche a ogni server di riserva, al meglio. La regola è in [transazioni-e-riconciliazione.md](transazioni-e-riconciliazione.md) (B-76, -B-83); oggi l'invio va anche a tutti gli altri server, salvo la sostituzione -RBF di `bump_fee` (B-77). +B-83). ## Riconnessione diff --git a/docs/modello-dati.md b/docs/modello-dati.md index 7e1198f..b32ef87 100644 --- a/docs/modello-dati.md +++ b/docs/modello-dati.md @@ -42,8 +42,7 @@ Cosa **non** c'è qui, e dove trovarlo: ## Diagramma delle relazioni -Dieci tabelle applicative nello schema atteso (nove oggi: manca -`pending_transaction_txids`, B-77). `round_config` non ha relazioni: è una +Dieci tabelle applicative. `round_config` non ha relazioni: è una riga singola di parametri. Gli indirizzi della piattaforma che non appartengono a un utente sono derivati dalla master key, e i loro UTXO si leggono dalla rete, non da una tabella: @@ -67,7 +66,7 @@ erDiagram rounds |o--o{ pending_transactions : "bet / payout" withdrawals |o--o{ pending_transactions : "withdrawal" users |o--o{ pending_transactions : "bet / withdrawal" - pending_transactions ||--|{ pending_transaction_txids : "ogni versione (da implementare)" + pending_transactions ||--|{ pending_transaction_txids : "ogni versione" users |o--o{ bug_reports : "segnala" users |o--o{ audit_log : "riguarda" rounds |o--o{ audit_log : "riguarda" @@ -233,7 +232,7 @@ erDiagram ([DAT-52]). Nessuna chiave esterna ha `ON DELETE`. Le sole cancellazioni fisiche nel codice sono quelle di una scommessa fallita (la riga `round_participants` e la sua `pending_transactions`, con le sue versioni in - `pending_transaction_txids` nello schema atteso, + `pending_transaction_txids`, [DAT-24], [DAT-38], [DAT-70]); nessun'altra tabella perde righe, quindi i riferimenti restano validi per costruzione, non per vincolo. *Codice:* `app/db/base.py:_register_sqlite_pragmas` · @@ -402,9 +401,7 @@ del database sono tutti uguali, perché arrivano tutti dallo stesso percorso stesso output risulta impegnato da un'altra transazione, quella rivendicazione è quella valida e non va toccata. *Codice:* `app/tx/reconcile.py:_release_inputs` · - *Test:* `tests/unit/test_reconcile.py` - *Stato:* da implementare (B-77), per le versioni precedenti · *Oggi:* il - confronto è solo con `current_txid`. + *Test:* `tests/unit/test_reconcile.py`, `tests/unit/test_tx_versions.py` - **[DAT-14]** Per ogni output impegnato, `spent_txid` è sempre il txid **corrente** della transazione che lo spende, cioè il `current_txid` della sua @@ -419,9 +416,7 @@ del database sono tutti uguali, perché arrivano tutti dallo stesso percorso *Codice:* `app/tx/broadcast.py:bump_fee` · `app/tx/broadcast.py:_retarget_txid_references` · `app/tx/confirmation.py:poll_once` · - *Test:* `tests/unit/test_broadcast.py` - *Stato:* da implementare (B-77) · *Oggi:* solo il bump riallinea; la - conferma di una versione precedente non viene riconosciuta. + *Test:* `tests/unit/test_broadcast.py`, `tests/unit/test_tx_versions.py` ### `rounds` @@ -904,7 +899,7 @@ riconciliatore (vedi | `broadcast_at` | DateTime | no | `utcnow` | Prima trasmissione, vedi [DAT-35] | ORM, alla creazione | | `last_broadcast_at` | DateTime | no | `utcnow` | Ultima trasmissione, vedi [DAT-35] | ORM; `bump_fee` | | `status` | String(16) | no | `"pending"` (mai usato: si crea sempre `building`) | Vedi [DAT-33] | vedi diagramma | -| `replaced_by_txid` | String(64) | sì | `NULL` | Il txid **sostituito** dall'ultimo bump (nonostante il nome, punta all'indietro). Nello schema atteso la cronologia completa è in `pending_transaction_txids` ([DAT-70]) e questa colonna resta solo informativa | `bump_fee` | +| `replaced_by_txid` | String(64) | sì | `NULL` | Il txid **sostituito** dall'ultimo bump (nonostante il nome, punta all'indietro). La cronologia completa è in `pending_transaction_txids` ([DAT-70]) e questa colonna resta solo informativa | `bump_fee` | | `attempt_count` | Integer | no | `1` | 1 + numero di bump riusciti | `bump_fee` | | `failure_reason` | String(128) | sì | `NULL` | Perché la transazione è fallita; solo con `status = failed` | `_abandon`; `_release_failed_withdrawal` | | `failed_at` | DateTime | sì | `NULL` | Istante del passaggio a `failed`; da lì si contano i 14 giorni di sorveglianza ([DAT-72]) — *da implementare (B-78)* | `_abandon`; `_release_failed_withdrawal` | @@ -995,12 +990,7 @@ Le funzioni citate sono `app/bets/service.py:place_bet`, `fee_rate_sat_vb` registra la commissione effettivamente pagata dalla nuova versione, non quella obiettivo. - *Codice:* `app/tx/broadcast.py:bump_fee` · *Test:* `tests/unit/test_broadcast.py` - *Stato:* da implementare (B-77) · *Oggi:* la nuova versione è trasmessa - nella fase 2 prima di esistere nel database, e la fase 3 riscrive la riga - **senza ricontrollarne** stato e `current_txid`: un abbandono o una - conferma avvenuti nel frattempo vengono sovrascritti. `replaced_by_txid` - conserva solo il txid immediatamente precedente. + *Codice:* `app/tx/broadcast.py:bump_fee` · *Test:* `tests/unit/test_broadcast.py`, `tests/unit/test_tx_versions.py` - **[DAT-35]** `broadcast_at` è la **prima** trasmissione e non viene mai riscritta: è il riferimento da cui il riconciliatore misura i propri tempi @@ -1062,7 +1052,7 @@ Le funzioni citate sono `app/bets/service.py:place_bet`, ### `pending_transaction_txids` -*Tabella da implementare (B-77).* Ogni versione che una riga +Ogni versione che una riga `pending_transactions` abbia mai avuto: la prima, ogni sostituzione RBF trasmessa e ogni candidata scritta prima di un bump. Serve perché una qualunque di queste versioni può essere quella che la rete conferma. @@ -1086,12 +1076,9 @@ qualunque di queste versioni può essere quella che la rete conferma. `current_txid`: la conferma di uno qualunque conferma la riga ([DAT-14]). La riga `bet` cancellata da `_release_failed_bet` ([DAT-38]) porta con sé le proprie versioni, nella stessa transazione. - *Codice:* `app/db/models.py` · `app/tx/broadcast.py:bump_fee` · + *Codice:* `app/db/models.py:PendingTransactionTxid` · `app/tx/versions.py` · `app/tx/broadcast.py:bump_fee` · `app/tx/confirmation.py:poll_once` · `app/tx/reconcile.py:reconcile_once` · - *Test:* — - *Stato:* da implementare (B-77) · *Oggi:* la tabella non esiste; poller e - riconciliatore confrontano solo `current_txid`, e `replaced_by_txid` - ricorda un solo passo indietro. + *Test:* `tests/unit/test_tx_versions.py`, `tests/unit/test_migrations.py` - **[DAT-71]** Una versione **candidata** è una riga di questa tabella il cui txid non è (ancora) `current_txid`: `bump_fee` l'ha scritta ma la fase 3 non @@ -1102,9 +1089,7 @@ qualunque di queste versioni può essere quella che la rete conferma. Il riconciliatore, quando ritrasmette, usa il `raw_tx_hex` corrente, non quello di una candidata. *Codice:* `app/tx/broadcast.py:bump_fee` · `app/tx/reconcile.py:reconcile_once` · - *Test:* — - *Stato:* da implementare (B-77) · *Oggi:* non esistono candidate: la - sostituzione esiste nel database solo dopo il suo broadcast riuscito. + *Test:* `tests/unit/test_tx_versions.py` ### `withdrawals` @@ -1240,8 +1225,8 @@ Tabella di servizio di Alembic: una riga con la revisione applicata | `ix_rounds_single_active` | `rounds` | `UNIQUE ((1)) WHERE status IN ('open','closing','drawing','paying_out')` | al massimo un round attivo, [DAT-17] (B-09) | | unico (`round_id`, `user_id`) | `round_participants` | `UNIQUE` | una scommessa per utente per round, [DAT-23] | | unico (`round_id`, `draw_position`) | `round_participants` | `UNIQUE` (i `NULL` non collidono) | una posizione di estrazione per partecipante, [DAT-66] — *da implementare (B-81)* | -| unico `txid` | `pending_transaction_txids` | `UNIQUE` | un txid appartiene a una sola transazione, [DAT-70] — *da implementare (B-77)* | -| `ix_pending_transaction_txids_pending_transaction_id` | `pending_transaction_txids` | indice | versioni di una transazione — *da implementare (B-77)* | +| unico `txid` | `pending_transaction_txids` | `UNIQUE` | un txid appartiene a una sola transazione, [DAT-70] | +| `ix_pending_transaction_txids_pending_transaction_id` | `pending_transaction_txids` | indice | versioni di una transazione | | `ix_round_participants_round_id` | `round_participants` | indice | partecipanti di un round | | `ix_round_participants_user_id` | `round_participants` | indice | round di un utente (`last-round-result`) | | `ix_withdrawals_user_id` | `withdrawals` | indice | prelievi di un utente | @@ -1267,12 +1252,7 @@ letti con una scansione completa. `app/withdrawals/service.py:request_withdrawal` · `app/rounds/scheduler.py:RoundScheduler._trigger_payout` · `app/tx/broadcast.py:bump_fee` · - *Test:* `tests/unit/test_bets.py`, `tests/unit/test_scheduler.py` - *Stato:* da implementare (B-77), per le sostituzioni RBF · *Oggi:* - l'invariante vale per la prima versione; una sostituzione RBF viene - trasmessa nella fase 2 di `bump_fee` e scritta nel database solo nella - fase 3, quindi un arresto o un `database is locked` fra le due lascia sulla - rete una transazione che il database non conosce. + *Test:* `tests/unit/test_bets.py`, `tests/unit/test_scheduler.py`, `tests/unit/test_tx_versions.py` - **[DAT-45]** **Ogni versione è riconoscibile, una sola è corrente.** Per ogni transazione non terminale, `pending_transactions.current_txid` coincide con @@ -1287,12 +1267,7 @@ letti con una scansione completa. txid. *Codice:* `app/tx/broadcast.py:_retarget_txid_references` · `app/tx/broadcast.py:bump_fee` · `app/tx/confirmation.py:poll_once` · - *Test:* `tests/unit/test_broadcast.py` - *Stato:* da implementare (B-77) · *Oggi:* l'aggiornamento dei riferimenti - avviene in una transazione del database separata dal broadcast e non - condizionata, e i txid precedenti sono dimenticati (salvo l'ultimo, in - `replaced_by_txid`, che nessuno legge): se conferma una versione - precedente, poller e riconciliatore non la riconoscono. + *Test:* `tests/unit/test_broadcast.py`, `tests/unit/test_tx_versions.py` - **[DAT-46]** **Il montepremi di ogni round si conserva, separato dagli altri.** Per un round pagato, con i soli partecipanti rimasti `confirmed` diff --git a/docs/operativita.md b/docs/operativita.md index b2ddcb0..b3f1dd6 100644 --- a/docs/operativita.md +++ b/docs/operativita.md @@ -615,8 +615,8 @@ con `docker compose logs app` ([architettura.md](architettura.md) ARC-09). indirizzo secondo il **solo server attivo**: mai partita (riga `building` da oltre 120s) o scomparsa dal mempool (riga `pending` da oltre 6 ore dal primo invio). Non è una prova: un server diverso da quello che ha ricevuto l'invio - può non conoscerla, e una versione precedente di una transazione accelerata - con RBF non viene cercata (B-77). + può non conoscerla (si cercano tutte le versioni di una transazione + accelerata con RBF, ma solo sul server attivo). **Effetto e cosa fare.** Il sistema ha già annullato tutto: UTXO liberati e saldo ricalcolato; per una scommessa il partecipante è rimosso (l'utente può riscommettere se il round è ancora aperto); per un prelievo la richiesta è diff --git a/docs/storico-bug.md b/docs/storico-bug.md index d05c50c..b5a9a4c 100644 --- a/docs/storico-bug.md +++ b/docs/storico-bug.md @@ -225,7 +225,7 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)). | B-74 | Header accettati senza controllo della difficoltà minima; un salto di più blocchi non viene collegato; una riorganizzazione di un blocco chiude la sessione | Critica | **aperto** | — | ELC | | B-75 | Una risposta Electrum oltre 64 KiB chiude la connessione, e la cronologia dell'indirizzo del montepremi la supera dopo ~800 transazioni | Critica | *da registrare* | `test_electrum_client.py`, `test_deposits.py`, `test_psbt_builder.py` | ELC, TX | | B-76 | Un invio ambiguo (timeout, connessione caduta) viene annullato come se fosse rifiutato | Critica | *da registrare* | `test_broadcast_outcome.py`, `test_electrum_client.py` | TX, BET, WDR | -| B-77 | Si ricorda solo l'ultimo txid di una transazione accelerata, e la sostituzione viene inviata prima di essere salvata | Critica | **aperto** | — | TX | +| B-77 | Si ricorda solo l'ultimo txid di una transazione accelerata, e la sostituzione viene inviata prima di essere salvata | Critica | *da registrare* | `test_tx_versions.py`, `test_migrations.py`, `test_broadcast.py` | TX | | B-78 | Una transazione viene abbandonata sulla base del silenzio di un solo server | Critica | **aperto** | — | TX, ELC | | B-79 | Il pagamento di norma non ha resto, quindi non si può accelerare e può bloccare la lotteria | Critica | **aperto** | — | RND, TX | | B-80 | Un solo indirizzo del montepremi per tutti i round | Critica | *da registrare* | `test_round_pool_address.py`, `test_pending_address.py`, `test_scheduler.py` | RND, BET, WDR | @@ -252,9 +252,9 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)). - **Corretti:** B-01 … B-70, tranne B-16, che è chiuso nella sostanza (il crash è eliminato e `/guida` serve un segnaposto) ma lascia la guida vera - come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76. + come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76, B-77. - **Aperti:** B-71, B-72 (gravità bassa, operativi o di documentazione) e - B-73 … B-98 dalla revisione del 2026-09-28 tranne i corretti (7 critici, + B-73 … B-98 dalla revisione del 2026-09-28 tranne i corretti (6 critici, 7 alti, 6 medi), da implementare nell'ordine di [piano-correzioni.md](piano-correzioni.md). - **Prossimo numero libero:** `B-99`. diff --git a/docs/transazioni-e-riconciliazione.md b/docs/transazioni-e-riconciliazione.md index 6907c7a..833a20f 100644 --- a/docs/transazioni-e-riconciliazione.md +++ b/docs/transazioni-e-riconciliazione.md @@ -69,11 +69,8 @@ stateDiagram-v2 la versione trovata in catena non è quella corrente, `current_txid` e `raw_tx_hex` diventano quelli della versione trovata e il txid viene propagato ovunque sia registrato, come dopo un RBF ([TX-12]). - *Codice:* `app/tx/confirmation.py:poll_once`, `app/tx/reconcile.py:reconcile_once`, `app/tx/broadcast.py:_retarget_txid_references` · - *Test:* — - *Stato:* da implementare (B-77) · *Oggi:* si conserva solo il txid corrente - (più il precedente in `replaced_by_txid`, che nessuno legge); se conferma - una versione precedente a un RBF non viene riconosciuta. + *Codice:* `app/tx/versions.py`, `app/tx/confirmation.py:poll_once`, `app/tx/reconcile.py:reconcile_once`, `app/tx/broadcast.py:adopt_version`, `app/tx/broadcast.py:_retarget_txid_references` · + *Test:* `tests/unit/test_tx_versions.py`, `tests/unit/test_migrations.py` - **[TX-28]** Ogni transazione del database che sposta denaro — la scrittura di `building` ([TX-01]), la promozione dopo l'invio, l'annullamento dopo un rifiuto, le due scritture dell'RBF ([TX-12]), la promozione e l'abbandono @@ -147,9 +144,7 @@ monete: questi invii viene solo registrato nei log, non cambia lo stato della riga e non ritarda la risposta all'utente. *Codice:* `app/electrum/listener.py:ElectrumListener.fan_out_transaction`, `app/tx/outcome.py:broadcast_with_verdict`, `app/bets/service.py:place_bet`, `app/withdrawals/service.py:request_withdrawal`, `app/rounds/scheduler.py:RoundScheduler._trigger_payout`, `app/tx/broadcast.py:bump_fee` · - *Test:* `tests/unit/test_broadcast_outcome.py` - *Stato:* da implementare (B-77), per la sostituzione RBF · *Oggi:* la - sostituzione di `bump_fee` viene inviata solo al server attivo. + *Test:* `tests/unit/test_broadcast_outcome.py`, `tests/unit/test_tx_versions.py` - **[TX-05]** Se l'invio di un pagamento non è accettato — rifiuto esplicito o esito incerto — la riga resta `building` e si scrive `payout_failed` (`broadcast rejected`): è il riconciliatore a decidere, interrogando la @@ -172,10 +167,9 @@ monete: per indirizzo a ogni giro; un indirizzo che non si riesce a leggere non blocca gli altri. *Codice:* `app/tx/confirmation.py:poll_once`, `app/tx/pending_address.py:own_address_for` · - *Test:* `tests/unit/test_confirmation.py`, `tests/unit/test_round_pool_address.py` - *Stato:* da implementare (B-77) · *Oggi:* cerca solo `current_txid`, - nella cronologia dell'indirizzo del montepremi del round per i pagamenti; un'altezza maggiore di 0 riportata dal solo server - attivo basta a confermare ([TX-22]). + *Test:* `tests/unit/test_confirmation.py`, `tests/unit/test_round_pool_address.py`, `tests/unit/test_tx_versions.py` + *Stato:* da implementare (B-83) · *Oggi:* un'altezza maggiore di 0 + riportata dal solo server attivo basta a confermare ([TX-22]). - **[TX-22]** **Conferma verificata.** Un candidato ([TX-06]) è confermato solo quando il quorum dei server ([connessione-electrum.md](connessione-electrum.md) (B-84)) restituisce per @@ -277,11 +271,7 @@ sostituibilità fin dal primo invio. round (`payout_txid`). Se la sostituzione è rifiutata esplicitamente la riga non cambia ([TX-24]). *Codice:* `app/tx/broadcast.py:bump_fee`, `app/tx/broadcast.py:_retarget_txid_references` · - *Test:* `tests/unit/test_broadcast.py` - *Stato:* da implementare (B-77) · *Oggi:* la sostituzione viene inviata - senza essere registrata, e la riga viene aggiornata solo dopo l'invio, senza - condizioni: un crash o un `database is locked` in mezzo lascia in rete una - transazione che il database non conosce. + *Test:* `tests/unit/test_broadcast.py`, `tests/unit/test_tx_versions.py` - **[TX-13]** L'RBF non tocca una riga che è cambiata sotto di lui. Prima di costruire la sostituzione, se la riga non esiste più o non è più `pending` (ad esempio è appena stata confermata), non si fa nulla. Dopo @@ -291,11 +281,7 @@ sostituibilità fin dal primo invio. candidata resta registrata — la conferma e la riconciliazione la cercano comunque ([TX-19]) — e il conflitto viene scritto nei log. *Codice:* `app/tx/broadcast.py:bump_fee` · - *Test:* `tests/unit/test_broadcast.py` - *Stato:* da implementare (B-77) · *Oggi:* il controllo avviene solo prima - della costruzione; dopo l'invio la riga viene aggiornata senza ricontrollare - né lo stato né `current_txid`, anche se nel frattempo è stata confermata o - abbandonata. + *Test:* `tests/unit/test_broadcast.py`, `tests/unit/test_tx_versions.py` - **[TX-24]** Un'accelerazione impossibile ([TX-11], o la riga già a `MAX_FEE_RATE_SAT_VB`) o una sostituzione rifiutata esplicitamente dalla rete scrivono un evento `rbf_bump_failed` ([audit-log.md](audit-log.md) @@ -342,9 +328,9 @@ riserva di [TX-11] serve proprio a dargli un'uscita da ridurre. ([TX-17]). Un **errore di rete** non porta a nulla: la riga resta com'è fino al giro successivo. *Codice:* `app/tx/reconcile.py:reconcile_once`, `app/electrum/listener.py:ElectrumListener._corroborate_majority` · - *Test:* `tests/unit/test_reconcile.py` + *Test:* `tests/unit/test_reconcile.py`, `tests/unit/test_tx_versions.py` *Stato:* da implementare (B-78) · *Oggi:* si interroga solo il server - attivo e solo per `current_txid`, e un txid assente dalla sua cronologia è + attivo, e un txid assente dalla sua cronologia è trattato come "sconosciuto alla rete": dopo un riavvio su un server diverso una riga `building` viene abbandonata dopo 120s anche se la transazione è nella mempool degli altri nodi (che la tengono fino a 336 ore con le diff --git a/migrations/versions/a3f5c2d9b7e1_add_pending_transaction_txids.py b/migrations/versions/a3f5c2d9b7e1_add_pending_transaction_txids.py new file mode 100644 index 0000000..0e4db42 --- /dev/null +++ b/migrations/versions/a3f5c2d9b7e1_add_pending_transaction_txids.py @@ -0,0 +1,68 @@ +"""add pending_transaction_txids + +Revision ID: a3f5c2d9b7e1 +Revises: 477bddae1a84 +Create Date: 2026-10-14 15:02:41.118204 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'a3f5c2d9b7e1' +down_revision: Union[str, Sequence[str], None] = '477bddae1a84' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema. + + B-77 (DAT-70): every version a pending transaction has ever had. A new table, so + nothing here rebuilds an existing one and the expression indexes (B-97) are not at + risk. Rows that already exist are backfilled with what they still remember: their + `current_txid` with its raw transaction and, when it differs, the `replaced_by_txid` + of their last bump with an empty `raw_hex` (that raw transaction was overwritten). + """ + op.create_table( + "pending_transaction_txids", + sa.Column("id", sa.Integer(), nullable=False), + sa.Column("pending_transaction_id", sa.Integer(), nullable=False), + sa.Column("txid", sa.String(length=64), nullable=False), + sa.Column("raw_hex", sa.Text(), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=False), + sa.ForeignKeyConstraint(["pending_transaction_id"], ["pending_transactions.id"]), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint("txid"), + ) + op.create_index( + op.f("ix_pending_transaction_txids_pending_transaction_id"), + "pending_transaction_txids", + ["pending_transaction_id"], + unique=False, + ) + + op.execute( + """ + INSERT INTO pending_transaction_txids (pending_transaction_id, txid, raw_hex, created_at) + SELECT id, current_txid, raw_tx_hex, broadcast_at FROM pending_transactions + """ + ) + op.execute( + """ + INSERT INTO pending_transaction_txids (pending_transaction_id, txid, raw_hex, created_at) + SELECT id, replaced_by_txid, '', broadcast_at FROM pending_transactions + WHERE replaced_by_txid IS NOT NULL + AND replaced_by_txid != current_txid + AND replaced_by_txid NOT IN (SELECT txid FROM pending_transaction_txids) + """ + ) + + +def downgrade() -> None: + """Downgrade schema.""" + op.drop_index(op.f("ix_pending_transaction_txids_pending_transaction_id"), table_name="pending_transaction_txids") + op.drop_table("pending_transaction_txids") diff --git a/tests/unit/test_migrations.py b/tests/unit/test_migrations.py index 2ed1154..9387017 100644 --- a/tests/unit/test_migrations.py +++ b/tests/unit/test_migrations.py @@ -74,3 +74,53 @@ def test_the_migrated_database_still_enforces_both_invariants(migrated_db): conn.execute(f"INSERT INTO users {columns} VALUES ('Alice', 'x', 0, 'plm1qa', 0, 0, '2026-01-01 00:00:00')") with pytest.raises(sqlite3.IntegrityError): conn.execute(f"INSERT INTO users {columns} VALUES ('aLICE', 'x', 1, 'plm1qb', 0, 0, '2026-01-01 00:00:00')") + + +def _alembic(db_path: Path, *args: str) -> None: + env = {**os.environ, "DATABASE_URL": f"sqlite+aiosqlite:///{db_path}"} + result = subprocess.run( + [sys.executable, "-m", "alembic", *args], cwd=_ROOT, env=env, capture_output=True, text=True, timeout=120 + ) + assert result.returncode == 0, f"alembic {' '.join(args)} failed:\n{result.stderr[-3000:]}" + + +def test_pending_transaction_txids_table_matches_the_model(migrated_db): + """B-77 (DAT-70): the migrated table has the model's columns, the unique txid and the + index on the owning row.""" + from app.db.models import PendingTransactionTxid + + with sqlite3.connect(migrated_db) as conn: + columns = {row[1]: row for row in conn.execute("PRAGMA table_info(pending_transaction_txids)")} + sql = conn.execute("SELECT sql FROM sqlite_master WHERE name = 'pending_transaction_txids'").fetchone()[0] + indexes = {row[1] for row in conn.execute("PRAGMA index_list(pending_transaction_txids)")} + assert set(columns) == {c.name for c in PendingTransactionTxid.__table__.columns} + assert "unique" in sql.lower() + assert "ix_pending_transaction_txids_pending_transaction_id" in indexes + + +def test_existing_pending_transactions_are_backfilled_with_the_txids_they_remember(tmp_path): + """B-77 (DAT-70): upgrading a database that already has rows records each one's + current txid with its raw transaction, and the txid a bump replaced with an empty + raw_hex — and the expression indexes survive the step.""" + db_path = tmp_path / "old.db" + _alembic(db_path, "upgrade", "477bddae1a84") + with sqlite3.connect(db_path) as conn: + columns = ( + "(kind, current_txid, fee_rate_sat_vb, raw_tx_hex, broadcast_at, last_broadcast_at, status, " + "replaced_by_txid, attempt_count, updated_at)" + ) + stamp = "'2026-01-01 00:00:00'" + conn.execute( + f"INSERT INTO pending_transactions {columns} VALUES " + f"('bet', 'cc', 2, 'rawcc', {stamp}, {stamp}, 'pending', 'bb', 2, {stamp})" + ) + conn.execute( + f"INSERT INTO pending_transactions {columns} VALUES " + f"('withdrawal', 'dd', 1, 'rawdd', {stamp}, {stamp}, 'building', NULL, 1, {stamp})" + ) + _alembic(db_path, "upgrade", "head") + with sqlite3.connect(db_path) as conn: + rows = sorted(conn.execute("SELECT pending_transaction_id, txid, raw_hex FROM pending_transaction_txids")) + assert rows == [(1, "bb", ""), (1, "cc", "rawcc"), (2, "dd", "rawdd")] + assert _index_sql(db_path, "ix_rounds_single_active") is not None + assert _index_sql(db_path, "ix_users_username_lower") is not None diff --git a/tests/unit/test_tx_versions.py b/tests/unit/test_tx_versions.py new file mode 100644 index 0000000..1730f12 --- /dev/null +++ b/tests/unit/test_tx_versions.py @@ -0,0 +1,448 @@ +"""B-77 (TX-19, TX-12, TX-13, TX-21, TX-06, TX-15, DAT-13, DAT-14, DAT-34, DAT-38, DAT-70, DAT-71): +every version of a bumped transaction is tracked, and an RBF replacement is written +before it is broadcast. + +Before the fix only `current_txid` was recognised: if an earlier version of a bumped +transaction confirmed, nothing noticed, and six hours later the reconciler abandoned the +row — a confirmed bet's participant deleted, a sent withdrawal marked failed, a paid +payout rebuilt. And `bump_fee` broadcast first and persisted afterwards, unconditionally. +""" + +from datetime import datetime, timedelta, timezone + +import pytest +from embit import script +from sqlalchemy import select +from sqlalchemy.exc import OperationalError + +import app.bets.confirmation # noqa: F401 (registers the "bet" handler) +import app.rounds.confirmation # noqa: F401 (registers the "payout" handler) +import app.withdrawals.confirmation # noqa: F401 (registers the "withdrawal" handler) +from app.bets.service import place_bet +from app.db.models import ( + AuditLog, + PendingTransaction, + PendingTransactionTxid, + Round, + RoundParticipant, + User, + UtxoEvent, + Withdrawal, +) +from app.tx import broadcast as broadcast_module +from app.tx.broadcast import RbfError, bump_fee +from app.tx.confirmation import poll_once +from app.tx.reconcile import reconcile_once +from app.wallet.hd import derive_pool_address, derive_user_address, derive_user_key +from app.wallet.plm_network import PLM_MAINNET +from app.wallet.psbt_builder import Utxo, build_signed_transaction +from app.withdrawals.service import request_withdrawal +from tests.unit.fake_network import FakeNetwork, rpc_error +from tests.unit.test_bets import _make_funded_user +from tests.unit.test_bets import session_factory as bets_session_factory # noqa: F401 (fixture) +from tests.unit.test_broadcast import FakeClient as ChainClient, session_factory # noqa: F401 (fixture) + +_UTXO_TXID = "11" * 32 +_UTXO_AMOUNT = 150_000_000 + + +async def _seed_pending_bet(session_factory, *, age_seconds: int = 1000) -> dict: + """A user, a round, a participant and a `pending` bet row, with its first version + recorded the way place_bet records it.""" + signer = derive_user_key(0) + address = derive_user_address(0) + built = build_signed_transaction( + signing_key=signer, + from_script=script.p2wpkh(signer.to_public()), + utxos=[Utxo(_UTXO_TXID, 0, _UTXO_AMOUNT)], + to_address=derive_pool_address(1), + amount_sats=10_000_000, + change_address=address, + fee_rate_sat_vb=1, + ) + broadcast_at = datetime.now(timezone.utc) - timedelta(seconds=age_seconds) + async with session_factory() as session: + user = User(username="alice", password_hash="x", derivation_index=0, address=address) + session.add(user) + await session.flush() + session.add(Round(id=1, status="open", pool_address=derive_pool_address(1))) + session.add( + UtxoEvent(user_id=user.id, txid=_UTXO_TXID, vout=0, amount_sats=_UTXO_AMOUNT, confirmed_height=5, spent_txid=built.txid) + ) + session.add( + RoundParticipant( + round_id=1, user_id=user.id, bet_amount_sats=10_000_000, bet_txid=built.txid, status="broadcast" + ) + ) + pending = PendingTransaction( + kind="bet", + round_id=1, + user_id=user.id, + current_txid=built.txid, + fee_rate_sat_vb=1, + raw_tx_hex=built.raw_hex, + status="pending", + broadcast_at=broadcast_at, + last_broadcast_at=broadcast_at, + ) + session.add(pending) + await session.flush() + session.add(PendingTransactionTxid(pending_transaction_id=pending.id, txid=built.txid, raw_hex=built.raw_hex)) + await session.commit() + return {"pending_id": pending.id, "user_id": user.id, "first_txid": built.txid, "first_raw": built.raw_hex} + + +class _Chain(ChainClient): + """prevout lookups for bump_fee, plus a scriptable broadcast and history.""" + + def __init__(self, on_broadcast=None, fail_with=None): + super().__init__({_UTXO_TXID: _UTXO_AMOUNT}) + self.on_broadcast = on_broadcast + self.fail_with = fail_with + self.history: list[dict] = [] + + async def broadcast(self, raw_tx_hex: str) -> str: + if self.on_broadcast is not None: + await self.on_broadcast(raw_tx_hex) + if self.fail_with is not None: + raise self.fail_with + return await super().broadcast(raw_tx_hex) + + async def get_history(self, scripthash: str) -> list[dict]: + return self.history + + +async def _versions(session_factory, pending_id): + async with session_factory() as session: + rows = ( + await session.scalars( + select(PendingTransactionTxid) + .where(PendingTransactionTxid.pending_transaction_id == pending_id) + .order_by(PendingTransactionTxid.id) + ) + ).all() + return [(r.txid, r.raw_hex) for r in rows] + + +# --- the first version is written with the row (DAT-70) -------------------------------- + + +async def test_a_bet_records_its_first_version_and_a_refusal_deletes_it_with_the_row(bets_session_factory): + user_id = await _make_funded_user(bets_session_factory, 4, 3_000_000_000) + + class _Ok: + async def broadcast(self, raw): + return "x" + + async with bets_session_factory() as session: + user = await session.get(User, user_id) + participant = await place_bet(session, _Ok(), user, network=FakeNetwork()) + async with bets_session_factory() as session: + pending = (await session.scalars(select(PendingTransaction))).one() + version = (await session.scalars(select(PendingTransactionTxid))).one() + assert (version.pending_transaction_id, version.txid) == (pending.id, participant.bet_txid) + assert version.raw_hex == pending.raw_tx_hex + + other = await _make_funded_user(bets_session_factory, 5, 3_000_000_000) + + class _Refuse: + async def broadcast(self, raw): + raise rpc_error("min relay fee not met") + + from app.bets.service import BetError + + async with bets_session_factory() as session: + user = await session.get(User, other) + with pytest.raises(BetError): + await place_bet(session, _Refuse(), user, network=FakeNetwork(known=False)) + async with bets_session_factory() as session: + # DAT-38: the refused bet's row and its versions are gone; the first bet's stay. + assert len((await session.scalars(select(PendingTransactionTxid))).all()) == 1 + + +async def test_a_withdrawal_records_its_first_version(bets_session_factory): + user_id = await _make_funded_user(bets_session_factory, 7, 3_000_000_000) + + class _Ok: + async def broadcast(self, raw): + return "x" + + async with bets_session_factory() as session: + user = await session.get(User, user_id) + withdrawal = await request_withdrawal( + session, _Ok(), user, "plm1qqph9qup2mp7w7g5nlsdhdc9m2pp44ampzw0ctx", 1_000_000_000, network=FakeNetwork() + ) + async with bets_session_factory() as session: + pending = (await session.scalars(select(PendingTransaction))).one() + version = (await session.scalars(select(PendingTransactionTxid))).one() + assert (version.txid, version.raw_hex) == (withdrawal.txid, pending.raw_tx_hex) + + +async def test_a_payout_records_its_first_version(payout_session_factory): + from app.rounds.scheduler import RoundScheduler + from tests.unit.test_scheduler import FakePayoutClient, FakePayoutListener, _POOL_AMOUNT_SATS, _seed_paying_out_round + + await _seed_paying_out_round(payout_session_factory) + entries = [{"tx_hash": "44" * 32, "tx_pos": 0, "height": 10, "value": _POOL_AMOUNT_SATS + 100_000}] + await RoundScheduler(payout_session_factory, FakePayoutListener(FakePayoutClient(entries)))._trigger_payout(1) + async with payout_session_factory() as session: + pending = (await session.scalars(select(PendingTransaction))).one() + version = (await session.scalars(select(PendingTransactionTxid))).one() + assert (version.txid, version.raw_hex) == (pending.current_txid, pending.raw_tx_hex) + + +from tests.unit.test_scheduler import payout_session_factory # noqa: E402,F401 (fixture) + + +# --- bump_fee: write first, then broadcast (TX-12, DAT-34) ------------------------------ + + +async def test_the_replacement_is_recorded_as_a_candidate_before_it_is_broadcast(session_factory): + seeded = await _seed_pending_bet(session_factory) + seen: dict = {} + + async def observe(raw_tx_hex: str) -> None: + # An independent session, mid-broadcast: what has been committed so far. + async with session_factory() as probe: + seen["versions"] = [ + v.txid for v in (await probe.scalars(select(PendingTransactionTxid))).all() + ] + seen["current"] = (await probe.get(PendingTransaction, seeded["pending_id"])).current_txid + + client = _Chain(on_broadcast=observe) + new_txid = await bump_fee(session_factory, client, seeded["pending_id"]) + + assert new_txid is not None and new_txid != seeded["first_txid"] + assert seen["current"] == seeded["first_txid"] # the row had not been switched yet... + assert new_txid in seen["versions"] # ...but the replacement was already on record + assert [txid for txid, _ in await _versions(session_factory, seeded["pending_id"])] == [seeded["first_txid"], new_txid] + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + assert row.current_txid == new_txid and row.replaced_by_txid == seeded["first_txid"] + assert row.attempt_count == 2 and row.fee_rate_sat_vb == 2 + participant = (await session.scalars(select(RoundParticipant))).one() + utxo = (await session.scalars(select(UtxoEvent))).one() + assert participant.bet_txid == utxo.spent_txid == new_txid + + +async def test_every_bump_adds_a_version_and_all_of_them_stay(session_factory): + seeded = await _seed_pending_bet(session_factory) + client = _Chain() + second = await bump_fee(session_factory, client, seeded["pending_id"]) + third = await bump_fee(session_factory, client, seeded["pending_id"]) + assert [txid for txid, _ in await _versions(session_factory, seeded["pending_id"])] == [ + seeded["first_txid"], + second, + third, + ] + # DAT-70: the raw transaction of every version is kept. + assert all(raw for _, raw in await _versions(session_factory, seeded["pending_id"])) + + +async def test_a_crash_between_the_broadcast_and_the_switch_over_leaves_the_replacement_on_record( + session_factory, monkeypatch +): + """The old phase 3 ran after the broadcast and was unconditional, so a crash or a + `database is locked` there left a transaction on the network the DB did not know. Now + the candidate is already committed, and the confirmation poller recognises it.""" + seeded = await _seed_pending_bet(session_factory) + real = broadcast_module.begin_immediate + calls = {"n": 0} + + async def flaky(session): + # begin_immediate calls in bump_fee: the candidate write (1), then the switch-over + # (2). Failing the second is the lock timing out right after the broadcast. + calls["n"] += 1 + if calls["n"] == 2: + raise OperationalError("BEGIN IMMEDIATE", {}, Exception("database is locked")) + await real(session) + + monkeypatch.setattr(broadcast_module, "begin_immediate", flaky) + client = _Chain() + with pytest.raises(OperationalError): + await bump_fee(session_factory, client, seeded["pending_id"]) + monkeypatch.setattr(broadcast_module, "begin_immediate", real) + + assert len(client.broadcasted) == 1 # the replacement is on the network + versions = await _versions(session_factory, seeded["pending_id"]) + assert len(versions) == 2 # ...and in the database + candidate = versions[1][0] + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + assert row.current_txid == seeded["first_txid"] # not switched over + + # The network confirms the candidate: the poller recognises it and realigns the row. + client.history = [{"tx_hash": candidate, "height": 101}] + assert await poll_once(session_factory, client) == 1 + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + assert row.status == "confirmed" and row.current_txid == candidate + assert (await session.scalars(select(UtxoEvent))).one().spent_txid == candidate + participant = (await session.scalars(select(RoundParticipant))).one() + assert participant.bet_txid == candidate and participant.status == "confirmed" + + +async def test_a_bump_retried_after_a_lost_switch_over_reuses_the_candidate(session_factory, monkeypatch): + """The same replacement is rebuilt (deterministic signing): recording it again must + not trip the unique txid.""" + seeded = await _seed_pending_bet(session_factory) + real = broadcast_module.begin_immediate + calls = {"n": 0} + + async def flaky(session): + calls["n"] += 1 + if calls["n"] == 2: + raise OperationalError("BEGIN IMMEDIATE", {}, Exception("database is locked")) + await real(session) + + monkeypatch.setattr(broadcast_module, "begin_immediate", flaky) + with pytest.raises(OperationalError): + await bump_fee(session_factory, _Chain(), seeded["pending_id"]) + monkeypatch.setattr(broadcast_module, "begin_immediate", real) + + new_txid = await bump_fee(session_factory, _Chain(), seeded["pending_id"]) + assert new_txid is not None + assert len(await _versions(session_factory, seeded["pending_id"])) == 2 + + +# --- the conditional update (TX-13) ------------------------------------------------------ + + +async def test_a_row_that_changed_during_the_broadcast_is_not_overwritten(session_factory): + """An abandon or a confirmation that lands between the broadcast and the switch-over + used to be silently overwritten. Now zero matched rows is a conflict: the row stays as + it is, the candidate stays recorded.""" + seeded = await _seed_pending_bet(session_factory) + + async def confirmed_meanwhile(raw_tx_hex: str) -> None: + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + row.status = "confirmed" + await session.commit() + + result = await bump_fee(session_factory, _Chain(on_broadcast=confirmed_meanwhile), seeded["pending_id"]) + + assert result is None + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + assert row.status == "confirmed" + assert row.current_txid == seeded["first_txid"] and row.attempt_count == 1 + assert (await session.scalars(select(UtxoEvent))).one().spent_txid == seeded["first_txid"] + assert len(await _versions(session_factory, seeded["pending_id"])) == 2 # the candidate stays + + +async def test_a_row_rebumped_during_the_broadcast_is_not_overwritten(session_factory): + seeded = await _seed_pending_bet(session_factory) + + async def rebumped_meanwhile(raw_tx_hex: str) -> None: + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + row.current_txid = "ee" * 32 + await session.commit() + + assert await bump_fee(session_factory, _Chain(on_broadcast=rebumped_meanwhile), seeded["pending_id"]) is None + async with session_factory() as session: + assert (await session.get(PendingTransaction, seeded["pending_id"])).current_txid == "ee" * 32 + + +async def test_an_explicitly_refused_replacement_leaves_the_row_alone(session_factory): + """TX-24: a verified refusal raises RbfError and the row keeps the version that is + on the network.""" + seeded = await _seed_pending_bet(session_factory) + client = _Chain(fail_with=rpc_error("insufficient fee, rejecting replacement")) + with pytest.raises(RbfError, match="refused the replacement"): + await bump_fee(session_factory, client, seeded["pending_id"], network=FakeNetwork(known=False)) + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + assert row.current_txid == seeded["first_txid"] and row.attempt_count == 1 + + +async def test_an_uncertain_replacement_is_adopted_and_an_accepted_one_is_fanned_out(session_factory): + from app.electrum.client import ElectrumError + + seeded = await _seed_pending_bet(session_factory) + network = FakeNetwork(known=False) + new_txid = await bump_fee( + session_factory, _Chain(fail_with=ElectrumError("connection closed")), seeded["pending_id"], network=network + ) + assert new_txid is not None and network.fanned_out == [] # uncertain: not fanned out, but adopted + async with session_factory() as session: + assert (await session.get(PendingTransaction, seeded["pending_id"])).current_txid == new_txid + + ok_network = FakeNetwork() + client = _Chain() + third = await bump_fee(session_factory, client, seeded["pending_id"], network=ok_network) + assert third is not None and ok_network.fanned_out == client.broadcasted # TX-21 + + +# --- an earlier version confirming (TX-19, DAT-14) ----------------------------------------- + + +async def test_the_confirmation_of_an_earlier_version_confirms_the_row_and_retargets_everything(session_factory): + """The case that used to be missed: v1 is bumped to v2 and v3, then *v1* confirms. + Only v3 was looked for, so the row stayed pending, and six hours later was abandoned + — the participant of a confirmed bet deleted.""" + seeded = await _seed_pending_bet(session_factory) + client = _Chain() + await bump_fee(session_factory, client, seeded["pending_id"]) + third = await bump_fee(session_factory, client, seeded["pending_id"]) + + client.history = [{"tx_hash": seeded["first_txid"], "height": 77}] + assert await poll_once(session_factory, client) == 1 + + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + assert row.status == "confirmed" + assert row.current_txid == seeded["first_txid"] != third + assert row.raw_tx_hex == seeded["first_raw"] + assert (await session.scalars(select(UtxoEvent))).one().spent_txid == seeded["first_txid"] + participant = (await session.scalars(select(RoundParticipant))).one() + assert participant.bet_txid == seeded["first_txid"] and participant.status == "confirmed" + + +async def test_an_unrelated_txid_in_the_history_does_not_confirm_the_row(session_factory): + seeded = await _seed_pending_bet(session_factory) + client = _Chain() + client.history = [{"tx_hash": "ab" * 32, "height": 5}] + assert await poll_once(session_factory, client) == 0 + async with session_factory() as session: + assert (await session.get(PendingTransaction, seeded["pending_id"])).status == "pending" + + +# --- the reconciler (TX-15, DAT-13) ---------------------------------------------------------- + + +async def test_the_reconciler_does_not_abandon_a_row_whose_earlier_version_is_on_the_network(session_factory): + """Before the fix a row older than 6h whose *current* txid the server did not list + was abandoned — even though an earlier version of it was in the mempool or mined.""" + seeded = await _seed_pending_bet(session_factory, age_seconds=7 * 3600) + client = _Chain() + await bump_fee(session_factory, client, seeded["pending_id"]) + client.history = [{"tx_hash": seeded["first_txid"], "height": 0}] # only the old version is known + + await reconcile_once(session_factory, client) + + async with session_factory() as session: + row = await session.get(PendingTransaction, seeded["pending_id"]) + assert row.status == "pending" + assert row.current_txid == seeded["first_txid"] # the row follows the version on the network + assert (await session.scalars(select(RoundParticipant))).one().bet_txid == seeded["first_txid"] + assert "pending_tx_abandoned" not in [e.event_type for e in (await session.scalars(select(AuditLog))).all()] + + +async def test_abandoning_a_row_releases_inputs_claimed_by_any_of_its_versions(session_factory): + """DAT-13: an input still marked with an earlier version's txid is this row's claim + too, so abandoning the row frees it.""" + seeded = await _seed_pending_bet(session_factory, age_seconds=7 * 3600) + client = _Chain() + await bump_fee(session_factory, client, seeded["pending_id"]) + async with session_factory() as session: # simulate a claim left on the earlier version + (await session.scalars(select(UtxoEvent))).one().spent_txid = seeded["first_txid"] + await session.commit() + client.history = [] # nobody knows any version + + await reconcile_once(session_factory, client) + + async with session_factory() as session: + assert (await session.get(PendingTransaction, seeded["pending_id"])).status == "failed" + assert (await session.scalars(select(UtxoEvent))).one().spent_txid is None