Let the system recover from a broadcast that never confirms

The code treated a broadcast as final: money moved on-chain and the DB was
updated on the assumption it would either confirm or be fee-bumped until it
did. Neither is guaranteed, and every way that assumption broke was permanent
(BUGS.md B-02, B-03, B-04, B-07, B-08, B-20, B-21).

Persist before broadcasting. place_bet and request_withdrawal now write their
rows in a "building" state and commit, then broadcast, then promote to
broadcast/pending in a second commit. Before, a failure or crash between the
broadcast and the commit left the coins irreversibly spent with no trace: no
participant (so no entry in the draw), no pending row (so no RBF and no
confirmation tracking), and the UTXOs not even marked spent, so the next bet
would try to double-spend them. A refused broadcast now releases the reserved
UTXOs, restores the balance, removes the participant (or marks the withdrawal
failed), audit-logs it, and answers a translatable broadcast_failed — as 502,
since the network refused it, not the caller, where it used to be an opaque 500.

Reconcile what's in flight against the chain. New PendingTransactionReconciler
(app/tx/reconcile.py, every 120s and once at startup) asks whether each
non-terminal tx exists: present -> promote, gone -> mark failed with a reason,
release the inputs, roll the domain row back, audit-log it. Grace periods differ
by state (120s for "building", 6h for "pending", so the RBF bumper gets its
attempts first). It is deliberately biased to inaction: only a server that
positively doesn't know the tx counts as absent, and a transport failure never
abandons anything, because releasing a UTXO whose tx is actually alive would
invite a double-spend. Verified against the live server, which answers "No such
mempool or blockchain transaction" for an unknown txid.

Stop keying on a value that changes. An RBF bump changes the txid, and
_on_bet_confirmed looked the participant up by bet_txid — so a bumped bet
confirmed under a txid no participant carried, the row stayed "broadcast"
forever, and the scheduler waited on it forever: the round could never close and
the lottery stopped. Handlers now resolve by immutable ids (round_id/user_id,
withdrawal_id), and bump_fee retargets every stored txid — bet_txid,
Withdrawal.txid, Round.payout_txid and UtxoEvent.spent_txid — plus records the
previous one in replaced_by_txid, which was never written at all.

One bad row no longer blocks the rest. The confirmation poller's per-tx lookup
is guarded: a txid the server can't resolve used to abort the whole pass, so
nothing confirmed again until an operator intervened. It also selects plain
columns instead of hydrating entities that outlive their session.

Tests: 6 reconciler cases including "a broken connection must not release coins";
the bet-ordering test probes committed state from an independent session during
the broadcast, and caught a real mistake in the first draft of this change (the
_pending_transaction helper still hardcoded status="pending", so rows were born
already-broadcast and would have got the 6-hour grace instead of 120s).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
davideandClaude Opus 5 committed 2026-07-27 00:31:24 +02:00
1 parent cc88763a9d
commit d528c5b475
16 files changed
+988 -36

No files matched your search

+45 -2
View File
@@ -9,7 +9,7 @@ from embit.finalizer import finalize_psbt
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from app.db.models import PendingTransaction, User
from app.db.models import PendingTransaction, Round, RoundParticipant, User, UtxoEvent, Withdrawal
from app.electrum.client import ElectrumClient
from app.rounds.config import get_round_config
from app.wallet.hd import derive_pool_key, derive_user_key
@@ -113,17 +113,60 @@ async def bump_fee(session: AsyncSession, client: ElectrumClient, pending: Pendi
new_txid = final_tx.txid().hex()
await client.broadcast(raw_hex)
old_txid = pending.current_txid
pending.replaced_by_txid = old_txid # points backwards: what current_txid replaced
pending.current_txid = new_txid
pending.raw_tx_hex = raw_hex
pending.fee_rate_sat_vb = new_fee_rate
pending.attempt_count += 1
pending.broadcast_at = datetime.now(timezone.utc)
await _retarget_txid_references(session, pending, old_txid, new_txid)
await session.commit()
logger.info("bumped %s pending_transaction %s: %s -> %s", pending.kind, pending.id, pending.current_txid, new_txid)
logger.info("bumped %s pending_transaction %s: %s -> %s", pending.kind, pending.id, old_txid, new_txid)
return new_txid
async def _retarget_txid_references(
session: AsyncSession, pending: PendingTransaction, old_txid: str, new_txid: str
) -> None:
"""A bump changes the txid, and everything that recorded the old one has to
follow — otherwise the bumped tx confirms and nothing recognizes it (B-02).
The worst case was the bet path: _on_bet_confirmed used to look the participant
up by bet_txid, so after a bump it found nothing, the participant stayed
"broadcast" forever, and the scheduler waited on it forever — the round could
never close and the lottery stopped. The handlers now key off immutable ids
(round_id/user_id, withdrawal_id) as well, so this update is about keeping the
stored txids *true* — for the admin UI, for the audit trail, and for
reconcile.py, which matches UtxoEvent.spent_txid against current_txid.
"""
if pending.kind == "bet":
participant = await session.scalar(
select(RoundParticipant).where(
RoundParticipant.round_id == pending.round_id,
RoundParticipant.user_id == pending.user_id,
)
)
if participant is not None:
participant.bet_txid = new_txid
elif pending.kind == "withdrawal" and pending.withdrawal_id is not None:
withdrawal = await session.get(Withdrawal, pending.withdrawal_id)
if withdrawal is not None:
withdrawal.txid = new_txid
elif pending.kind == "payout" and pending.round_id is not None:
round_ = await session.get(Round, pending.round_id)
if round_ is not None and round_.payout_txid == old_txid:
round_.payout_txid = new_txid
# The UTXOs this tx spends are still the same UTXOs — only the id of the tx
# spending them changed. Keeping this in step is what lets reconcile.py tell
# "reserved by this pending tx" from "spent by something else".
spent = (await session.scalars(select(UtxoEvent).where(UtxoEvent.spent_txid == old_txid))).all()
for utxo in spent:
utxo.spent_txid = new_txid
class RbfBumper:
def __init__(self, session_factory: async_sessionmaker, get_client):
self._session_factory = session_factory