Let the system recover from a broadcast that never confirms

The code treated a broadcast as final: money moved on-chain and the DB was updated on the assumption it would either confirm or be fee-bumped until it did. Neither is guaranteed, and every way that assumption broke was permanent (BUGS.md B-02, B-03, B-04, B-07, B-08, B-20, B-21). Persist before broadcasting. place_bet and request_withdrawal now write their rows in a "building" state and commit, then broadcast, then promote to broadcast/pending in a second commit. Before, a failure or crash between the broadcast and the commit left the coins irreversibly spent with no trace: no participant (so no entry in the draw), no pending row (so no RBF and no confirmation tracking), and the UTXOs not even marked spent, so the next bet would try to double-spend them. A refused broadcast now releases the reserved UTXOs, restores the balance, removes the participant (or marks the withdrawal failed), audit-logs it, and answers a translatable broadcast_failed — as 502, since the network refused it, not the caller, where it used to be an opaque 500. Reconcile what's in flight against the chain. New PendingTransactionReconciler (app/tx/reconcile.py, every 120s and once at startup) asks whether each non-terminal tx exists: present -> promote, gone -> mark failed with a reason, release the inputs, roll the domain row back, audit-log it. Grace periods differ by state (120s for "building", 6h for "pending", so the RBF bumper gets its attempts first). It is deliberately biased to inaction: only a server that positively doesn't know the tx counts as absent, and a transport failure never abandons anything, because releasing a UTXO whose tx is actually alive would invite a double-spend. Verified against the live server, which answers "No such mempool or blockchain transaction" for an unknown txid. Stop keying on a value that changes. An RBF bump changes the txid, and _on_bet_confirmed looked the participant up by bet_txid — so a bumped bet confirmed under a txid no participant carried, the row stayed "broadcast" forever, and the scheduler waited on it forever: the round could never close and the lottery stopped. Handlers now resolve by immutable ids (round_id/user_id, withdrawal_id), and bump_fee retargets every stored txid — bet_txid, Withdrawal.txid, Round.payout_txid and UtxoEvent.spent_txid — plus records the previous one in replaced_by_txid, which was never written at all. One bad row no longer blocks the rest. The confirmation poller's per-tx lookup is guarded: a txid the server can't resolve used to abort the whole pass, so nothing confirmed again until an operator intervened. It also selects plain columns instead of hydrating entities that outlive their session. Tests: 6 reconciler cases including "a broken connection must not release coins"; the bet-ordering test probes committed state from an independent session during the broadcast, and caught a real mistake in the first draft of this change (the _pending_transaction helper still hardcoded status="pending", so rows were born already-broadcast and would have got the 6-hour grace instead of 120s). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-27 00:31:24 +02:00
parent cc88763a9d
commit d528c5b475
16 changed files with 988 additions and 36 deletions
@@ -8,10 +8,24 @@ from app.tx.confirmation import register_handler


 async def _on_bet_confirmed(session: AsyncSession, pending: PendingTransaction) -> None:
+    """Resolved by (round_id, user_id) — the pair is unique per participant and,
+    unlike the txid, cannot change under us. Keying this on bet_txid meant an
+    RBF-bumped bet confirmed under a txid no participant carried, so the row stayed
+    "broadcast" forever and the round could never close (B-02). The txid is kept in
+    step by tx/broadcast.py too, but correctness here no longer depends on it."""
    participant = await session.scalar(
-        select(RoundParticipant).where(RoundParticipant.bet_txid == pending.current_txid)
+        select(RoundParticipant).where(
+            RoundParticipant.round_id == pending.round_id,
+            RoundParticipant.user_id == pending.user_id,
+        )
    )
-    if participant is not None and participant.status == "broadcast":
+    if participant is None:
+        # Fall back to the txid for rows written before this changed, and for any
+        # pending row missing its round/user link.
+        participant = await session.scalar(
+            select(RoundParticipant).where(RoundParticipant.bet_txid == pending.current_txid)
+        )
+    if participant is not None and participant.status in ("building", "broadcast"):
        participant.status = "confirmed"
        participant.confirmed_at = datetime.now(timezone.utc)

@@ -64,12 +64,17 @@ async def place_bet(session: AsyncSession, client: ElectrumClient, user: User) -
    except InsufficientFundsError as exc:
        raise BetError(exc.code, str(exc)) from exc

-    await client.broadcast(built.raw_hex)
-
+    # --- Phase 1: record the intent, *then* broadcast (B-08) --------------------
+    # Broadcasting first meant a failure (or a crash) between the broadcast and the
+    # commit left the coins irreversibly spent on-chain with no trace in the DB: no
+    # participant, so no entry in the draw; no pending row, so no RBF and no
+    # confirmation tracking; and the UTXOs not even marked spent, so the next bet
+    # would try to double-spend them. Writing "building" rows first means the worst
+    # case is a row the reconciler (app/tx/reconcile.py) can resolve either way by
+    # asking the chain whether the tx exists.
    spent_by_key = {(u.txid, u.vout): u for u in unspent}
    for spent in built.spent_utxos:
-        row = spent_by_key[(spent.txid, spent.vout)]
-        row.spent_txid = built.txid
+        spent_by_key[(spent.txid, spent.vout)].spent_txid = built.txid
    await recompute_balance(session, user.id)

    broadcast_at = datetime.now(timezone.utc)
@@ -79,10 +84,25 @@ async def place_bet(session: AsyncSession, client: ElectrumClient, user: User) -
        bet_amount_sats=built.recipient_sats,
        bet_txid=built.txid,
        broadcast_at=broadcast_at,
-        status="broadcast",
+        status="building",
    )
    session.add(participant)
-    session.add(_pending_transaction(round_.id, user.id, built, config.fee_rate_sat_vb))
+    pending = _pending_transaction(round_.id, user.id, built, config.fee_rate_sat_vb)
+    session.add(pending)
+    await session.commit()
+
+    # --- Phase 2: broadcast, then promote both rows to their live state ---------
+    try:
+        await client.broadcast(built.raw_hex)
+    except Exception as exc:
+        # The node refused it (fee too low, dust, mempool conflict, or simply an
+        # unreachable server) — nothing is on-chain, so undo phase 1 completely and
+        # give the user a translatable failure instead of a bare 500 (B-07).
+        await _release_failed_bet(session, participant, pending, built, user.id, str(exc))
+        raise BetError("broadcast_failed", f"the network refused the transaction: {exc}") from exc
+
+    participant.status = "broadcast"
+    pending.status = "pending"
    await write_audit_log(
        session,
        "bet_placed",
@@ -97,6 +117,35 @@ async def place_bet(session: AsyncSession, client: ElectrumClient, user: User) -
    return participant


+async def _release_failed_bet(
+    session: AsyncSession,
+    participant: RoundParticipant,
+    pending: PendingTransaction,
+    built: BuiltTransaction,
+    user_id: int,
+    reason: str,
+) -> None:
+    """Undo phase 1 after a failed broadcast: free the UTXOs the build reserved, drop
+    the two rows, and restore the balance. Same shape as what the reconciler does for
+    a tx that turns out never to have made it onto the chain."""
+    for spent in built.spent_utxos:
+        row = await session.scalar(
+            select(UtxoEvent).where(UtxoEvent.txid == spent.txid, UtxoEvent.vout == spent.vout)
+        )
+        if row is not None:
+            row.spent_txid = None
+    await session.delete(participant)
+    await session.delete(pending)
+    await recompute_balance(session, user_id)
+    await write_audit_log(
+        session,
+        "bet_broadcast_failed",
+        {"txid": built.txid, "reason": reason[:200]},
+        user_id=user_id,
+    )
+    await session.commit()
+
+
 def _pending_transaction(
    round_id: int, user_id: int, built: BuiltTransaction, fee_rate_sat_vb: int
 ) -> PendingTransaction:
@@ -107,5 +156,9 @@ def _pending_transaction(
        current_txid=built.txid,
        fee_rate_sat_vb=fee_rate_sat_vb,
        raw_tx_hex=built.raw_hex,
-        status="pending",
+        # "building" until the broadcast succeeds — see place_bet's two phases. It
+        # matters which one this starts as: the reconciler gives a "building" row a
+        # short grace period (we may have died mid-broadcast) and a "pending" one a
+        # long one (a node accepted it once, so it deserves the RBF attempts first).
+        status="building",
    )