davide/plm-lottery
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Retry a stuck payout automatically, and log every failure (B-26)

Browse Source 
_trigger_payout used to run exactly once, from _close_and_draw. Any
failure after that point — no Electrum client, insufficient pool
UTXOs, a missing fee_address, a rejected broadcast — wedged the round
in paying_out forever, and every one of those early returns except the
generic exception handler logged nothing at all: /admin showed a
stalled round with no explanation. A process restart while paying_out
hit the same dead end.

_tick now handles status == "paying_out": it calls the new
_retry_payout_if_due, which re-invokes _trigger_payout unless the most
recent payout_failed audit entry for the round is younger than
_PAYOUT_RETRY_INTERVAL_SECONDS (60s) — throttled so a persistently
broken payout (e.g. no fee_address set yet) doesn't retry, and re-log
a failure, on every 5-second tick. Every early return in
_trigger_payout now calls _log_payout_failure with a reason string, so
that throttle always has something to check against and /admin always
shows why a round is stuck.

This is safe to fire on a restart too, because B-25 already made
_trigger_payout idempotent (it no-ops if a non-terminal payout
PendingTransaction already exists) and persists before broadcasting —
so a round found paying_out at startup, whatever state its payout was
actually in, gets retried the same way. That closes the paying_out
half of the "scheduler doesn't resume mid-flight rounds after a
restart" gap in CLAUDE.md; the drawing/block-wait half is untouched
(see BUGS.md B-36).

BUGS.md moves B-26 to "Previously fixed" with the fix description; the
suite grows from 143 to 148 tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
davide
2026-07-27 09:29:14 +02:00
co-authored by Claude Sonnet 5
parent f13f6850b7
commit 50a43ae3ca
3 changed files with 223 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BUGS.md
+58 -36
View File
@@ -1,21 +1,23 @@
# Known bugs
A second full-codebase audit on 2026-07-27 found **25 further issues** (4 critical, 6 high,
7 medium, 8 low), listed below as B-25 … B-49. B-25 is fixed as of 2026-07-27; the other 24
are open. The 139-test suite was green at the time of the audit, so none of these were caught
by existing coverage — every fix lands with a regression test (B-25's four tests brought the
suite to 143).
7 medium, 8 low), listed below as B-25 … B-49. B-25 and B-26 are fixed as of 2026-07-27; the
other 23 are open. The 139-test suite was green at the time of the audit, so none of these
were caught by existing coverage — every fix lands with a regression test (B-25 and B-26
together brought the suite from 139 to 148).
The recurring pattern across B-26, B-27, B-29 and B-36 is worth stating once: the code is
rigorous about the failure modes that have actually been hit, and silent about the ones that
have not. Outgoing transactions reconcile; deposits do not. Broadcast failures are
audit-logged; *pre*-broadcast failures (no client, insufficient pool funds) are not.
The recurring pattern across B-27, B-29 and B-36 is worth stating once: the code is rigorous
about the failure modes that have actually been hit, and silent about the ones that have not.
Outgoing transactions reconcile; deposits do not.
**Highest remaining priority: make `paying_out` fully recoverable, not just idempotent.**
B-25 made a payout retry *safe* (persisted before broadcast, guarded against double-spend);
B-26 is what would make a retry actually *happen* automatically. Together with the
already-known "scheduler doesn't resume" gap, that's every way the lottery currently stops
and cannot restart on its own.
**`paying_out` is now fully recoverable, not just idempotent.** B-25 made a payout retry
*safe* (persisted before broadcast, guarded against double-spend); B-26 made it *automatic*
(the scheduler retries a stuck `paying_out` round on its own, throttled, and every failure —
including ones that used to fail silently — is now audit-logged with a reason). Together
these close every way a payout specifically could wedge the lottery forever. What's still open
in the same family is narrower: the "drawing" phase (waiting on a block) has no equivalent
resume-after-restart or stall visibility — see B-36 and the "scheduler doesn't resume" entry
in CLAUDE.md's Known gaps, which this doesn't touch.
For limitations that are accepted by design rather than bugs (single-shared-token admin auth,
single-process assumptions, no user-facing history, etc.), see "Known gaps / TODO" in
@@ -25,27 +27,6 @@ single-process assumptions, no user-facing history, etc.), see "Known gaps / TOD
## Critical
### B-26 — A transient failure at payout time wedges the lottery permanently
`rounds/scheduler.py:166-169`: if `listener.client is None` when `_trigger_payout` starts, it
returns. `_trigger_payout` is called exactly once, from `_close_and_draw`, and `_tick`
ignores any round not in `open`/`closing` (`:58`). The round stays in `paying_out`, no new
round can open, and — unlike the `except Exception` branch — nothing is written to
`audit_log`, so `/admin` shows a stalled state with no explanation.
The payout runs immediately after a ~2-minute wait on a block, so an Electrum drop in that
window is entirely plausible. Same shape at `:215-217`: `InsufficientFundsError` returns
without calling `_log_payout_failure`.
CLAUDE.md lists "payout retry" as an accepted gap, but treats it as an operational
inconvenience; in practice it is a single point of failure that stops the whole platform.
**Proposed fix.** (a) Call `_log_payout_failure` on *every* early return, with a reason in the
payload, so the operator sees it. (b) Make `_tick` handle `paying_out`: if the round has no
non-terminal payout `PendingTransaction`, re-run `_trigger_payout`. That turns every early
return into a retry rather than a dead end, and — combined with B-25's idempotency guard —
also covers the process-restart case.
### B-27 — Every RBF bump resets the reconciler's abandon clock, so it never fires
`tx/broadcast.py:122` sets `pending.broadcast_at = now` on each bump, but
@@ -392,13 +373,54 @@ not just this round's — so a stale payout from an earlier round that the recon
abandoned yet can't be double-spent by a fresh attempt. `should_bump`/reconciler retry timing
around a fee-bumped payout is unaffected by this fix (see B-27, still open).
This makes a payout retry *safe*; it does not yet make one *automatic* — that is B-26, still
open. Regression tests: `tests/unit/test_scheduler.py`
This made a payout retry *safe*; B-26 (below) is what makes one *automatic*. Regression
tests: `tests/unit/test_scheduler.py`
(`test_trigger_payout_persists_before_broadcasting`,
`test_trigger_payout_broadcast_failure_leaves_a_recoverable_row`,
`test_trigger_payout_skips_when_already_in_flight`,
`test_reserved_payout_outpoints_excludes_utxos_claimed_by_a_stale_payout`).
### B-26 — A transient failure at payout time wedges the lottery permanently
`rounds/scheduler.py`: if `listener.client is None` when `_trigger_payout` starts, it returned
without recording anything. `_trigger_payout` was called exactly once, from `_close_and_draw`,
and `_tick` ignored any round not in `open`/`closing`. The round stayed in `paying_out`, no new
round could open, and — unlike the generic `except Exception` branch — nothing was written to
`audit_log`, so `/admin` showed a stalled state with no explanation.
The payout runs immediately after a ~2-minute wait on a block, so an Electrum drop in that
window is entirely plausible. Same shape applied to `InsufficientFundsError`, a missing
`fee_address` and a missing winner user — none of them logged anything either.
CLAUDE.md listed "payout retry" as an accepted gap, but treated it as an operational
inconvenience; in practice it was a single point of failure that stopped the whole platform,
including across a process restart while a round was `paying_out`.
**Fixed:** two changes, matching the proposed fix exactly. (a) Every early return in
`_trigger_payout` — not connected, no `fee_address`, winner not found, insufficient pool
UTXOs, a build error, a rejected broadcast — now calls `_log_payout_failure` with a `reason`
string in the payload, so `/admin`'s audit log always shows *why* a round is stuck, not just
that it is. (b) `_tick` now handles `status == "paying_out"` by calling the new
`_retry_payout_if_due`, which re-invokes `_trigger_payout` unless the most recent
`payout_failed` audit entry for this round is younger than `_PAYOUT_RETRY_INTERVAL_SECONDS`
(60s) — throttled so a persistently-broken payout (e.g. an operator hasn't set `fee_address`
yet) doesn't retry, and re-log a failure, on every 5-second tick.
Because B-25 already made `_trigger_payout` idempotent (it no-ops if a non-terminal payout
`PendingTransaction` already exists for the round) and persists before broadcasting, this
retry is safe to fire on a process restart too: a round found `paying_out` at startup — whose
payout may have already broadcast, may never have been attempted, or may have been abandoned
by the reconciler — is retried the same way, closing the `paying_out` half of the "scheduler
doesn't resume mid-flight rounds after a restart" gap in CLAUDE.md (the "drawing"/block-wait
half is unrelated and still open, see B-36).
Regression tests: `tests/unit/test_scheduler.py`
(`test_trigger_payout_logs_a_failure_when_not_connected`,
`test_trigger_payout_logs_a_failure_when_fee_address_missing`,
`test_tick_retries_a_stuck_paying_out_round_with_no_recent_failure`,
`test_tick_throttles_retry_after_a_recent_payout_failure`,
`test_tick_retries_once_the_throttle_window_has_elapsed`).
A full-codebase audit on 2026-07-26 (commit `d4e0974`) found 24 bugs across every Python
module under `app/`, both static frontends, and the Docker/Caddy deployment — 5 critical,
7 high, 7 medium, 5 low. All 24 were fixed and verified against the current code on