diff --git a/BUGS.md b/BUGS.md index 5a0b5b1..f88467c 100644 --- a/BUGS.md +++ b/BUGS.md @@ -179,17 +179,6 @@ winner's own bet is gone the payout is held (`payout_blocked`) and the lottery p ## High -### B-83 — Confirmations and reinstatement trust the active server alone - -`poll_once` marks a tx confirmed on one server's `height > 0`, and -`reinstate_reappeared_utxos` restores a UTXO on its word, so a hostile active server -can report a never-relayed payout as confirmed (winner unpaid) or a phantom bet. -Broadcasts go to the active server only. - -Fix: a confirmation is accepted once the quorum returns a valid merkle proof -(`blockchain.transaction.get_merkle`) against a header we hold; reinstatement needs -the same quorum as a credit (B-59); broadcasts fan out (B-76). - ### B-85 — Reorgs are neither followed nor surfaced Nothing detects a reorg; a 1-block reorg is treated as a hostile server (B-74). diff --git a/CLAUDE.md b/CLAUDE.md index 9839ffe..75b7e69 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ The user communicates in Italian in chat — reply to them in Italian. Everythin ## Project status -All 10 stages of the original build order are code-complete and unit-tested — 516 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. +All 10 stages of the original build order are code-complete and unit-tested — 556 tests, all under `tests/unit/` (`tests/integration/` is an empty package). Beyond them: Docker + Caddy deployment, admin dashboard (`/admin`), static test UI (`/`), pending-inclusive balance display, an SSE push channel layered over the original polling, self-service password change + admin password reset, and the reconciliation/corroboration machinery below. Verified on mainnet with real money: registration + address derivation, deposit crediting (1-conf), a real 10 PLM bet (broadcast → confirmed → change credited back), and one full round cycle (close → draw on a real block hash → 70/30 payout with sat math checked against the broadcast tx → confirmation → close → next round auto-opened). **Withdrawal and the RBF bump path have never been exercised against a live broadcast** — unit-tested only. @@ -49,7 +49,7 @@ PYTHONPATH=. python scripts/decrypt_master_key.py # ops recovery: decrypt+pr PYTHONPATH=. python scripts/encrypt_master_key.py # ops bootstrap: import an externally-generated xprv (--overwrite to replace) PYTHONPATH=. python scripts/electrum_smoke_test.py # manual check: connect, handshake, subscribe to headers, print the tip -python -m pytest # all 516 tests +python -m pytest # all 556 tests python -m pytest tests/unit/test_hd.py # one file python -m pytest tests/unit/test_hd.py::test_derivation_is_deterministic # one test ``` @@ -158,7 +158,7 @@ One connection serves everything — deposit credits, broadcasts, confirmations, - **Bounded requests** (`_REQUEST_TIMEOUT_SECONDS` = 15s); a timeout tears the connection down. Unbounded waits used to hang `POST /bets` *while holding the per-user lock*, and could stall the confirmation poller permanently. - **The drop is observable**: `client.wait_closed()` resolves when the read loop dies, and `_run_once` races it against the notification consumers and a 60s `server.ping`. Without it the listener sat on queues nobody would ever fill while `listener.client` still looked alive. - **Headers are validated, not trusted** (`_apply_header`): the tip never regresses, a header must meet the difficulty target it claims, and a single-block advance must chain from the current tip's hash. Failure raises `HeaderValidationError`, which ends the session like a dropped connection and rotates away — that header is the draw's only entropy, so a fabricated one picks the winner. Two more headers are refused *without* ending the session, since neither implies a hostile server (B-64): one at a height we already hold a header for (a reorg at the tip, or one server disagreeing — the hash committed to for a height is never swapped under us, and `corroborate_header` is what catches us holding an orphan), and one carrying no `hex` at all (nothing to validate or draw from, and applying the height alone would break the `tip_height`/`tip_header_hex` pairing). `_run_once` separately refuses to publish the client while *no* tip is known, so the ignore-don't-kill choice can't reopen B-63. -- **A quorum corroborates every money-moving decision** (`_corroborate_majority`, 10s per server, asking only the *other* endpoints — never the active one, which is what a MITM controls): `corroborate_header` before a block seeds the draw (B-28), `corroborate_utxo_spent` before a UTXO missing from one `listunspent` is written off as externally spent (B-29), and `corroborate_utxo_credit` before a new outpoint credits a balance — same outpoint, same amount, confirmed (B-59). Balances move in both directions, so both directions need the same quorum — with one exception today: `deposits/service.py:reinstate_reappeared_utxos` restores a reappeared UTXO on the active server's word alone. It needs **at least 2 answers** and a strict majority of those that answered (B-84): fewer → returns **False**, since one voice decides nothing independently and an unreachable network proves nothing. Only `ALLOW_NO_QUORUM=true` keeps the old reduced rule (no other server → True, otherwise a majority of whoever answers). A failed credit corroboration only *delays*: `find_new_credit_candidates` re-offers the outpoint on the next refresh or `DepositReconciler` sweep. +- **A quorum corroborates every money-moving decision** (`_corroborate_majority`, 10s per server, asking only the *other* endpoints — never the active one, which is what a MITM controls): `corroborate_header` before a block seeds the draw (B-28), `corroborate_utxo_spent` before a UTXO missing from one `listunspent` is written off as externally spent (B-29), and `corroborate_utxo_credit` before a new outpoint credits a balance — same outpoint, same amount, confirmed (B-59). Balances move in both directions, so both directions need the same quorum — and since B-83 that includes `corroborate_utxo_reinstatement` before a UTXO written off as externally spent is put back (same outpoint, amount, confirmed; `deposits/service.py:find_reinstate_candidates` only reads, `reinstate_reappeared_utxos` writes what the quorum agreed on), and `confirmation_proven` before a tx is marked confirmed: `tx/confirmation.py:poll_once` takes the height the active server reports only as a *claim*, and the row confirms when a strict majority of at least 2 of the *other* servers return a `get_merkle` proof that recomputes to the merkle root of the block's header (`electrum/merkle.py`; the header is the tip we hold, or for an older block the one the quorum agrees on and that meets its own target). With no `network` nothing confirms. The one decision still on the active server's word is the pool-coin listing for a payout (B-82). It needs **at least 2 answers** and a strict majority of those that answered (B-84): fewer → returns **False**, since one voice decides nothing independently and an unreachable network proves nothing. Only `ALLOW_NO_QUORUM=true` keeps the old reduced rule (no other server → True, otherwise a majority of whoever answers). A failed credit corroboration only *delays*: `find_new_credit_candidates` re-offers the outpoint on the next refresh or `DepositReconciler` sweep. On reconnect `_subscribe_all_users` runs as its own task with bounded concurrency (`_RESUBSCRIBE_CONCURRENCY` = 20) rather than inline and serially — otherwise a large user base froze `tip_height`, and with it an in-flight draw, for the whole sweep (B-31); one user's failure is logged and skipped. `address_for_new_user` (called right after registration) is best-effort by design: on failure that address stays unsubscribed until the next reconnect or `DepositReconciler` sweep. diff --git a/README.md b/README.md index 1c998fa..ba84794 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ python -m pytest # all tests python -m pytest tests/unit/test_hd.py # one file ``` -516 unit tests cover HD derivation, PSBT building, the Electrum client, bets, +556 unit tests cover HD derivation, PSBT building, the Electrum client, bets, deposits, withdrawals, the round/draw engine, RBF fee-bumping, admin config, the pending-inclusive balance calculation, and the SSE push channel. No automated integration tests against a live Electrum connection — mainnet diff --git a/app/deposits/service.py b/app/deposits/service.py index fd90e57..e34d6e5 100644 --- a/app/deposits/service.py +++ b/app/deposits/service.py @@ -103,12 +103,38 @@ async def credit_confirmed_utxos(session: AsyncSession, user_id: int, entries: l _EXTERNAL_SPEND_SENTINEL = "external-spend" +async def find_reinstate_candidates(session: AsyncSession, user_id: int, entries: list[dict]) -> list[dict]: + """Outpoints flagged as spent outside the platform that this refresh's `listunspent` + lists again, confirmed and for the amount we recorded — the *candidates* for + reinstatement (B-83, DEP-02, DEP-08). Read-only: like a new credit (B-59) and a + candidate external spend (B-29), a reinstatement moves a balance, so the caller + (electrum/listener.py:refresh_user) must have the other servers corroborate each + candidate before reinstate_reappeared_utxos writes it. An entry whose amount differs + from our record is not a candidate: it is not the coin we wrote off.""" + flagged = ( + await session.scalars( + select(UtxoEvent).where(UtxoEvent.user_id == user_id, UtxoEvent.spent_txid == _EXTERNAL_SPEND_SENTINEL) + ) + ).all() + amounts = {(row.txid, row.vout): row.amount_sats for row in flagged} + return [ + entry + for entry in entries + if amounts.get((entry["tx_hash"], entry["tx_pos"])) == entry.get("value") and (entry.get("height") or 0) > 0 + ] + + async def reinstate_reappeared_utxos(session: AsyncSession, user_id: int, entries: list[dict]) -> int: """The reverse of a mark applied by find_utxos_missing_from/ mark_utxos_spent_externally (B-29): if an outpoint we'd previously flagged as spent outside the platform reappears as unspent in a later listunspent, undo the mark instead of leaving it permanent no matter what the chain says - afterwards. Cheap and purely DB-side — always safe to run on every refresh. + afterwards. + + Persistence only, for entries the caller has **already corroborated** against the + other servers (B-83, ELC-25) — it used to run on the active server's word alone, so a + hostile one could put a spent coin back into a balance. Re-checks, under the write + lock, that each row still carries the mark before clearing it (DEP-08). """ current_keys = {(entry["tx_hash"], entry["tx_pos"]) for entry in entries} diff --git a/app/electrum/listener.py b/app/electrum/listener.py index 74b76fd..fecb3f6 100644 --- a/app/electrum/listener.py +++ b/app/electrum/listener.py @@ -11,12 +11,14 @@ from app.db.models import User from app.deposits.service import ( credit_confirmed_utxos, find_new_credit_candidates, + find_reinstate_candidates, find_utxos_missing_from, mark_utxos_spent_externally, reinstate_reappeared_utxos, ) from app.config import MIN_QUORUM_RESPONSES from app.electrum.client import ElectrumClient, ElectrumEndpoint, ElectrumRpcError +from app.electrum.merkle import proof_matches_header from app.electrum.quorum import ServerAnswers from app.electrum.scripthash import address_to_scripthash from app.rounds.draw import ( @@ -646,6 +648,26 @@ class ElectrumListener: only credited once the quorum agrees. """ + return await self._corroborate_unspent_confirmed( + scripthash, txid, vout, value, f"credit of {value} sats at {txid}:{vout}" + ) + + async def corroborate_utxo_reinstatement(self, scripthash: str, txid: str, vout: int, value: int) -> bool: + """B-83 (ELC-25, DEP-08): the same quorum as a credit, for an outpoint that was + written off as spent outside the platform and has reappeared among the unspent. + Balances move in both directions and both need the same agreement: reinstating + on the active server's word alone let a hostile one put a spent coin back into a + balance.""" + return await self._corroborate_unspent_confirmed( + scripthash, txid, vout, value, f"reinstatement of {value} sats at {txid}:{vout}" + ) + + async def _corroborate_unspent_confirmed( + self, scripthash: str, txid: str, vout: int, value: int, description: str + ) -> bool: + """Do the other servers list this outpoint as unspent, for the same amount, and + confirmed (height > 0)? Shared by credits (B-59) and reinstatements (B-83).""" + async def _ask(endpoint: ElectrumEndpoint) -> bool | None: client = self._client_factory(endpoint) try: @@ -665,9 +687,56 @@ class ElectrumListener: finally: await client.close() - return await self._corroborate_majority( - _ask, lambda agrees: agrees, f"credit of {value} sats at {txid}:{vout}" - ) + return await self._corroborate_majority(_ask, lambda agrees: agrees, description) + + # --- Confirmations proven by merkle branch (B-83, TX-22, ELC-26) --------------------- + + async def _header_at(self, height: int) -> str | None: + """The header of block `height` that a merkle proof is checked against: the tip + we hold (validated as it arrived), or for an older block the one a strict + majority of at least the quorum minimum of the other servers agree on, which must + also satisfy its own difficulty target. None when neither is available — in + particular for a height above our tip, which we have no way to check.""" + if height <= 0 or height > self.tip_height: + return None + if height == self.tip_height and self.tip_header_hex: + return self.tip_header_hex + + async def _ask(client: ElectrumClient) -> str | None: + header = await client.get_block_header(height) + return header if isinstance(header, str) and header_meets_its_own_target(header) else None + + answers = [a for a in await self._ask_other_servers(_ask) if a is not None] + if len(answers) < max(self._minimum_other_answers(), 1): + return None + header, votes = Counter(answers).most_common(1)[0] + return header if votes * 2 > len(answers) else None + + async def confirmation_proven(self, txid: str, height: int) -> bool: + """Is `txid` in the block at `height`? Accepted only when the quorum returns a + valid merkle proof — `blockchain.transaction.get_merkle` from the *other* servers + (never the active one, which is what a hostile peer controls) that recomputes to + the merkle root of that block's header, which we hold or the quorum agrees on. A + strict majority of at least the quorum minimum must prove it; otherwise the answer + is False and the caller simply asks again at its next pass (a delay, never a + loss).""" + header = await self._header_at(height) + if header is None: + logger.warning("cannot check the merkle proof of %s at height %s: no trusted header for it", txid, height) + return False + + async def _ask(endpoint: ElectrumEndpoint) -> bool | None: + client = self._client_factory(endpoint) + try: + await asyncio.wait_for(client.connect(), timeout=_CORROBORATION_TIMEOUT_SECONDS) + proof = await asyncio.wait_for(client.get_merkle(txid, height), timeout=_CORROBORATION_TIMEOUT_SECONDS) + return proof_matches_header(txid, height, proof, header) + except Exception: + return None + finally: + await client.close() + + return await self._corroborate_majority(_ask, lambda proven: proven, f"merkle proof of {txid} at height {height}") async def _consume_headers(self, queue: asyncio.Queue) -> None: while True: @@ -697,7 +766,8 @@ class ElectrumListener: async with self._session_factory() as session: credit_candidates = await find_new_credit_candidates(session, user_id, entries) - reinstated = await reinstate_reappeared_utxos(session, user_id, entries) + # B-83 (DEP-02): candidates only — nothing is written before the quorum agrees. + reinstate_candidates = await find_reinstate_candidates(session, user_id, entries) candidates = [ (row.id, row.txid, row.vout) for row in await find_utxos_missing_from(session, user_id, entries) @@ -711,6 +781,27 @@ class ElectrumListener: ) ] + # B-83 (ELC-25, DEP-08): a coin written off as spent outside the platform comes back + # into the balance only on the same quorum as a credit. + corroborated_reinstatements = [ + entry + for entry in reinstate_candidates + if await self.corroborate_utxo_reinstatement( + scripthash, entry["tx_hash"], entry["tx_pos"], entry["value"] + ) + ] + reinstated = 0 + if corroborated_reinstatements: + async with self._session_factory() as session: + reinstated = await reinstate_reappeared_utxos(session, user_id, corroborated_reinstatements) + if len(corroborated_reinstatements) < len(reinstate_candidates): + logger.warning( + "%s reappeared UTXO(s) for user_id=%s not corroborated by the other servers — " + "left out of the balance, will retry on the next refresh", + len(reinstate_candidates) - len(corroborated_reinstatements), + user_id, + ) + credited = 0 if corroborated_credits: async with self._session_factory() as session: diff --git a/app/electrum/merkle.py b/app/electrum/merkle.py new file mode 100644 index 0000000..6cf943b --- /dev/null +++ b/app/electrum/merkle.py @@ -0,0 +1,57 @@ +"""Merkle proofs (B-83, TX-22, ELC-26). + +`blockchain.transaction.get_merkle` answers with the branch that links a transaction to +its block's merkle root. Checking it against a header **we hold** is what turns "a server +says this tx is at height h" into something a hostile server cannot fake without mining. +""" + +import hashlib + +_HEADER_LENGTH_BYTES = 80 +_MERKLE_ROOT_OFFSET = 36 +_MERKLE_ROOT_LENGTH = 32 +_MAX_BRANCH_LENGTH = 40 # 2^40 transactions in a block is already absurd + + +def _sha256d(data: bytes) -> bytes: + return hashlib.sha256(hashlib.sha256(data).digest()).digest() + + +def header_merkle_root(header_hex: str) -> bytes: + """The merkle root committed to by a raw 80-byte header, in internal byte order.""" + header = bytes.fromhex(header_hex) + if len(header) != _HEADER_LENGTH_BYTES: + raise ValueError(f"a block header is {_HEADER_LENGTH_BYTES} bytes, got {len(header)}") + return header[_MERKLE_ROOT_OFFSET : _MERKLE_ROOT_OFFSET + _MERKLE_ROOT_LENGTH] + + +def merkle_root_from_branch(txid: str, branch: list[str], position: int) -> bytes: + """The merkle root (internal byte order) a transaction `txid` at index `position` of + its block reaches through `branch` — the sibling hashes, bottom-up, in the display + order Electrum uses for txids.""" + if position < 0 or len(branch) > _MAX_BRANCH_LENGTH or position >> len(branch): + raise ValueError("merkle position does not fit the branch") + current = bytes.fromhex(txid)[::-1] + index = position + for sibling_hex in branch: + sibling = bytes.fromhex(sibling_hex)[::-1] + if len(sibling) != 32: + raise ValueError("merkle branch element is not 32 bytes") + current = _sha256d(sibling + current) if index & 1 else _sha256d(current + sibling) + index >>= 1 + return current + + +def proof_matches_header(txid: str, height: int, proof: object, header_hex: str) -> bool: + """Does `proof` — a `get_merkle` reply — put `txid` in the block at `height` whose + header is `header_hex`? Any malformed reply is simply "no".""" + try: + if not isinstance(proof, dict) or int(proof["block_height"]) != height: + return False + branch = proof["merkle"] + if not isinstance(branch, list): + return False + root = merkle_root_from_branch(txid, [str(item) for item in branch], int(proof["pos"])) + return root == header_merkle_root(header_hex) + except (KeyError, TypeError, ValueError): + return False diff --git a/app/main.py b/app/main.py index 8fbca0a..682c201 100644 --- a/app/main.py +++ b/app/main.py @@ -62,7 +62,7 @@ async def lifespan(app: FastAPI): app.state.user_locks = UserLocks() scheduler = RoundScheduler(AsyncSessionLocal, listener) - poller = ConfirmationPoller(AsyncSessionLocal, lambda: listener.client) + poller = ConfirmationPoller(AsyncSessionLocal, lambda: listener.client, network=listener) bumper = RbfBumper(AsyncSessionLocal, lambda: listener.client, network=listener) # Resolves in-flight transactions against the chain — the piece that lets the # system recover on its own from a broadcast that never confirmed (B-04/B-08). diff --git a/app/tx/confirmation.py b/app/tx/confirmation.py index 8df3dca..2f62974 100644 --- a/app/tx/confirmation.py +++ b/app/tx/confirmation.py @@ -1,6 +1,7 @@ import asyncio import logging from collections.abc import Awaitable, Callable +from typing import Protocol from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker @@ -29,7 +30,19 @@ def register_handler(kind: str, handler: ConfirmationHandler) -> None: _handlers[kind] = handler -async def poll_once(session_factory: async_sessionmaker, client: ElectrumClient) -> int: +class ConfirmationProver(Protocol): + """The part of ElectrumListener a confirmation needs (B-83): the quorum's verdict on + a merkle proof of `txid` at `height`.""" + + async def confirmation_proven(self, txid: str, height: int) -> bool: ... + + +async def poll_once(session_factory: async_sessionmaker, client: ElectrumClient, network: ConfirmationProver | None) -> int: + """One pass over the `pending` rows. The active server's history only nominates a + candidate: a row is confirmed when the quorum proves it with a merkle proof + (B-83, TX-22). With no `network` to ask nothing can be confirmed — the active + server's word alone is exactly what a hostile one would use to declare a payout that + was never broadcast confirmed.""" async with session_factory() as session: # Plain columns, not entities: nothing then outlives the session, so this # can't break if expire_on_commit is ever turned on (B-21). @@ -90,6 +103,18 @@ async def poll_once(session_factory: async_sessionmaker, client: ElectrumClient) if entry is None: continue + # B-83 (TX-22, ELC-26): the height is a claim of the active server. The quorum has + # to return a valid merkle proof of the txid against that block's header; without + # one the row stays pending and is looked at again on the next pass. + if network is None or not await network.confirmation_proven(entry["tx_hash"], entry["height"]): + logger.warning( + "pending_transaction %s: %s reported at height %s but the quorum did not prove it — still pending", + pending_id, + entry["tx_hash"], + entry["height"], + ) + continue + async with session_factory() as session: await begin_immediate(session) # B-88: read the row and write its outcome under one lock row = await session.get(PendingTransaction, pending_id) @@ -111,16 +136,22 @@ async def poll_once(session_factory: async_sessionmaker, client: ElectrumClient) class ConfirmationPoller: - def __init__(self, session_factory: async_sessionmaker, get_client: Callable[[], ElectrumClient | None]): + def __init__( + self, + session_factory: async_sessionmaker, + get_client: Callable[[], ElectrumClient | None], + network: ConfirmationProver | None = None, + ): self._session_factory = session_factory self._get_client = get_client + self._network = network # the ElectrumListener, which holds the headers and asks the quorum async def run(self) -> None: while True: client = self._get_client() if client is not None: try: - await poll_once(self._session_factory, client) + await poll_once(self._session_factory, client, self._network) except asyncio.CancelledError: raise except Exception: diff --git a/app/tx/reconcile.py b/app/tx/reconcile.py index 8954d20..4d645b5 100644 --- a/app/tx/reconcile.py +++ b/app/tx/reconcile.py @@ -81,6 +81,9 @@ _ABANDON_AFTER_SECONDS = 6 * 60 * 60 # for dead can still arrive in a block. FAILED_WATCH_DAYS = 14 +# How many distinct (txid, height) claims of one failed row are put to the quorum per pass. +_MAX_RESURRECTION_CLAIMS = 3 + # What a node says when the transaction's inputs do not exist or are spent already. _MISSING_INPUTS_MARKERS = ("missing-inputs", "missingorspent", "missing inputs", "inputs-missing") # What it says when only the fee is the problem: not proof of anything. @@ -236,23 +239,17 @@ def _entries_for(answers: ServerAnswers, known_txids: set[str]) -> list[tuple[st return found -def _confirmed_by_the_quorum(answers: ServerAnswers, known_txids: set[str]) -> tuple[str, int] | None: - """A txid of the row that a strict majority of the answering *other* servers (at - least the quorum minimum of them) list in a block, with the height they report. - The active server's word is not part of it.""" - others = answers.answered_others - if len(others) < max(answers.others_needed, 1): - return None +def _confirmed_candidates(answers: ServerAnswers, known_txids: set[str]) -> list[tuple[str, int]]: + """(txid, height) of every txid of the row that any answering server lists in a block, + the one most servers agree on first. They are only *claims*: each is proven by the + quorum's merkle proofs before it is believed (B-83, TX-22).""" votes: dict[tuple[str, int], int] = {} - for history in others: + for history in answers.all: for entry in history: if entry.get("tx_hash") in known_txids and (entry.get("height") or 0) > 0: key = (entry["tx_hash"], entry["height"]) votes[key] = votes.get(key, 0) + 1 - for key, count in sorted(votes.items(), key=lambda item: -item[1]): - if count * 2 > len(others): - return key - return None + return [key for key, _ in sorted(votes.items(), key=lambda item: -item[1])] async def _decide(network, candidate: _Candidate, answers: ServerAnswers): @@ -260,8 +257,12 @@ async def _decide(network, candidate: _Candidate, answers: ServerAnswers): ("abandon", reason), ("resurrected", (txid, height)) or None for "nothing to do or not enough to tell". Pure reads — nothing is written here.""" if candidate.status == "failed": - confirmed = _confirmed_by_the_quorum(answers, candidate.known_txids) - return ("resurrected", confirmed) if confirmed is not None else None + if network is None: + return None # nothing can be proven without the quorum + for txid, height in _confirmed_candidates(answers, candidate.known_txids)[:_MAX_RESURRECTION_CLAIMS]: + if await network.confirmation_proven(txid, height): + return ("resurrected", (txid, height)) + return None found = _entries_for(answers, candidate.known_txids) if found: diff --git a/docs/adr/ADR-07-ogni-decisione-sposta-denaro-base-risposta.md b/docs/adr/ADR-07-ogni-decisione-sposta-denaro-base-risposta.md index 4f4d6b8..0df3fb1 100644 --- a/docs/adr/ADR-07-ogni-decisione-sposta-denaro-base-risposta.md +++ b/docs/adr/ADR-07-ogni-decisione-sposta-denaro-base-risposta.md @@ -1,6 +1,6 @@ # ADR-07 — Ogni decisione che sposta denaro in base a una risposta di rete è corroborata da un quorum degli altri server configurati -- **Stato:** Accettata — implementazione da completare (B-83) +- **Stato:** Accettata - **Area:** Conferme e fiducia nella rete - **Fonte di verità per:** la scelta di progetto descritta qui sotto; i requisiti (`PREFISSO-NN`) che la attuano stanno nei documenti di dominio, indicati nelle *Conseguenze*. @@ -31,7 +31,5 @@ di 2 server, la corroborazione fallisce. scelti dall'operatore ([LIM-11]). Regole in [connessione-electrum.md](../connessione-electrum.md#5-il-quorum) (B-83, B-84). -*Codice:* `app/electrum/listener.py:ElectrumListener._corroborate_majority` · `app/electrum/listener.py:ElectrumListener.corroborate_header` · `app/electrum/listener.py:ElectrumListener.corroborate_utxo_spent` · `app/electrum/listener.py:ElectrumListener.corroborate_utxo_credit` · `app/main.py:lifespan` · -*Test:* `tests/unit/test_electrum_listener.py` -*Stato:* da implementare (B-83) · *Oggi:* conferme, ripristini e abbandoni -si decidono sul solo server attivo. +*Codice:* `app/electrum/listener.py:ElectrumListener._corroborate_majority` · `app/electrum/listener.py:ElectrumListener.corroborate_header` · `app/electrum/listener.py:ElectrumListener.corroborate_utxo_spent` · `app/electrum/listener.py:ElectrumListener.corroborate_utxo_credit` · `app/electrum/listener.py:ElectrumListener.corroborate_utxo_reinstatement` · `app/electrum/listener.py:ElectrumListener.confirmation_proven` · `app/main.py:lifespan` · +*Test:* `tests/unit/test_electrum_listener.py`, `tests/unit/test_confirmation_proof.py` diff --git a/docs/audit-log.md b/docs/audit-log.md index 370b740..0268b2f 100644 --- a/docs/audit-log.md +++ b/docs/audit-log.md @@ -332,9 +332,7 @@ flowchart LR `user_id` sì, `round_id` no. Stessa transazione dell'annullamento e del ricalcolo del saldo. *Codice:* `app/deposits/service.py:reinstate_reappeared_utxos` · - *Test:* `tests/unit/test_deposits.py` - *Stato:* da implementare (B-83), per la conferma del quorum · *Oggi:* basta - la parola del server attivo. + *Test:* `tests/unit/test_deposits.py`, `tests/unit/test_confirmation_proof.py` - **[AUD-19]** `bet_placed` — la rete ha accettato la transazione di una scommessa. Payload `{"txid", "amount_sats"}`: `txid` è quello del primo diff --git a/docs/connessione-electrum.md b/docs/connessione-electrum.md index 609a076..e668db6 100644 --- a/docs/connessione-electrum.md +++ b/docs/connessione-electrum.md @@ -274,13 +274,12 @@ Ogni decisione che sposta denaro in base a una risposta della rete viene confermata dagli **altri** server configurati, mai dal solo server attivo (che è proprio quello che un attaccante controllerebbe). -Oggi non è ancora così per tre decisioni: le conferme delle transazioni -(`app/tx/confirmation.py:poll_once`), il reintegro di una moneta ricomparsa -(`app/deposits/service.py:reinstate_reappeared_utxos`) e la lettura delle -monete del montepremi per il pagamento -(`app/rounds/scheduler.py:RoundScheduler._trigger_payout`) si basano sul solo -server attivo; sono le righe [ELC-26], [ELC-25] e [ELC-27]. L'abbandono di una -transazione ([ELC-28]) e l'invio ([TX-04], [TX-21]) usano già tutti i server. +Oggi non è ancora così per una sola decisione: la lettura delle monete del +montepremi per il pagamento +(`app/rounds/scheduler.py:RoundScheduler._trigger_payout`) si basa sul solo +server attivo, ed è la riga [ELC-27]. Le conferme delle transazioni +([ELC-26]), il reintegro di una moneta ricomparsa ([ELC-25]), l'abbandono +([ELC-28]) e l'invio ([TX-04], [TX-21]) usano già gli altri server. - **[ELC-14]** La corroborazione interroga in parallelo tutti i server configurati tranne quello attivo, ciascuno con una connessione nuova e un @@ -308,8 +307,10 @@ transazione ([ELC-28]) e l'invio ([TX-04], [TX-21]) usano già tutti i server. *Codice:* `app/electrum/listener.py:ElectrumListener.corroborate_header`, `app/electrum/listener.py:ElectrumListener.corroborate_utxo_spent`, `app/electrum/listener.py:ElectrumListener.corroborate_utxo_credit`, +`app/electrum/listener.py:ElectrumListener.corroborate_utxo_reinstatement`, +`app/electrum/listener.py:ElectrumListener.confirmation_proven`, `app/electrum/listener.py:ElectrumListener._corroborate_majority` · -*Test:* `tests/unit/test_electrum_listener.py` +*Test:* `tests/unit/test_electrum_listener.py`, `tests/unit/test_confirmation_proof.py` Stato delle righe della tabella: @@ -319,11 +320,6 @@ Stato delle righe della tabella: successivo invece di richiedere la stessa altezza. - [ELC-17] — *Stato:* da implementare (B-82) · *Oggi:* basta un'altezza maggiore di 0 (1 conferma). -- [ELC-25] — *Stato:* da implementare (B-83) · *Oggi:* il reintegro si basa - sulla sola risposta del server attivo. -- [ELC-26] — *Stato:* da implementare (B-83) · *Oggi:* una transazione è - confermata quando la cronologia dell'indirizzo sul server attivo la riporta - con altezza maggiore di 0. - [ELC-27] — *Stato:* da implementare (B-82) · *Oggi:* il pagamento sceglie fra le monete che il server attivo elenca per il montepremi. diff --git a/docs/limiti-noti.md b/docs/limiti-noti.md index 7a587ab..3ec3458 100644 --- a/docs/limiti-noti.md +++ b/docs/limiti-noti.md @@ -37,7 +37,7 @@ implementato il [piano delle correzioni](piano-correzioni.md). | [LIM-03] Attesa in `closing` senza limite di tempo | **Sì, in caso di congestione** | Resta senza limite di tempo per scelta ([ADR-51](adr/ADR-51-dubbio-liberano-mai-fondi.md)); con B-78 e B-87 finisce quando la rete decide | | [LIM-04] Prelievo e RBF mai provati in rete reale | **Sì, come verifica** | Da chiudere con la prova su rete reale ([operativita.md](operativita.md#prova-su-rete-reale)) prima di aprire al pubblico | | [LIM-10] Riorganizzazioni non gestite | **Sì** — una scommessa sparita rende il round impagabile | Chiuso da B-82 e B-85; resta [LIM-24] | -| [LIM-11], [LIM-12] Quorum facoltativo e parziale | **Sì** — un solo server decide conferme e abbandoni | Chiusi da B-83 e B-84 | +| [LIM-11], [LIM-12] Quorum facoltativo e parziale | **Sì** — un solo server decide conferme e abbandoni | Chiusi da B-83 e B-84 (resta la lettura delle monete del montepremi, B-82) | | [LIM-13] Estrazione manipolabile | **Sì** — per un giocatore che mina | Chiuso da B-73; resta [LIM-25] | | [LIM-14] Header senza difficoltà minima | **Sì** — un header falso congela ogni estrazione | Chiuso da B-74 per il blocco; resta il costo basso di un header a difficoltà minima | | [LIM-06] Token admin unico | No per il funzionamento, ma è il rischio di sicurezza più ampio | Invariato | @@ -276,18 +276,16 @@ implementato il [piano delle correzioni](piano-correzioni.md). *Codice:* `app/electrum/listener.py:ElectrumListener._corroborate_majority` - **[LIM-12]** **Non tutte le letture di rete sono corroborate.** Passano dal - solo server attivo: la conferma di scommesse, pagamenti e prelievi - (`poll_once`), l'elenco delle monete del montepremi usato per il pagamento, e il - ripristino di una moneta prima segnata come spesa all'esterno. - - *Impatto:* un server ostile potrebbe far considerare confermata una - scommessa che non lo è, lasciando il database in disaccordo con la - catena; il ripristino non corroborato può reintegrare nel saldo una - moneta davvero spesa. + solo server attivo: l'elenco delle monete del montepremi usato per il + pagamento. + - *Impatto:* un server ostile potrebbe far scegliere al pagamento monete che + non esistono (il pagamento verrebbe rifiutato dalla rete) o nasconderne + di vere (il pagamento fallirebbe e verrebbe ritentato). - *Mitigazione attuale:* l'abbandono richiede una prova ([ADR-51](adr/ADR-51-dubbio-liberano-mai-fondi.md)) e l'invio arriva a tutti i server; un invio basato su monete inesistenti è rifiutato dalla rete. - - *Stato:* da chiudere con B-83 (conferme e ripristini) e B-82 (monete del - round prima del pagamento). + - *Stato:* da chiudere con B-82 (monete del round prima del pagamento); + conferme e ripristini sono chiusi da B-83. - *Per chiuderlo:* una conferma vale quando il quorum restituisce una prova merkle valida (`blockchain.transaction.get_merkle`) contro un header che possediamo; il ripristino richiede lo stesso quorum di un accredito; diff --git a/docs/modello-dati.md b/docs/modello-dati.md index 2fe0dfc..f80950d 100644 --- a/docs/modello-dati.md +++ b/docs/modello-dati.md @@ -390,9 +390,8 @@ del database sono tutti uguali, perché arrivano tutti dallo stesso percorso `spent_txid IS NULL`. *Codice:* le funzioni in tabella · *Test:* `tests/unit/test_deposits.py`, `tests/unit/test_bets.py`, - `tests/unit/test_withdrawals.py`, `tests/unit/test_reconcile.py` - *Stato:* da implementare (B-83) · *Oggi:* il reintegro avviene sulla parola - del solo server attivo. + `tests/unit/test_withdrawals.py`, `tests/unit/test_reconcile.py`, + `tests/unit/test_confirmation_proof.py` - **[DAT-13]** Il riconciliatore libera un input solo se il suo `spent_txid` è uno dei txid della riga che sta abbandonando (`current_txid` o una sua diff --git a/docs/registrazione-e-depositi.md b/docs/registrazione-e-depositi.md index 9145767..b1c97c6 100644 --- a/docs/registrazione-e-depositi.md +++ b/docs/registrazione-e-depositi.md @@ -237,11 +237,7 @@ monete sotto la soglia non compaiono nel saldo mostrato. casi corroborati, ricontrollando lo stato di ogni moneta al salvataggio. Nessuna sessione del database resta aperta durante le chiamate di rete. *Codice:* `app/electrum/listener.py:ElectrumListener.refresh_user` · - *Test:* `tests/unit/test_electrum_listener.py` - *Stato:* da implementare (B-83) · *Oggi:* il reintegro delle monete - ricomparse viene **scritto e committato già nella fase di lettura**, prima - di qualunque corroborazione; solo accrediti e sparizioni sono corroborati - prima di essere scritti. + *Test:* `tests/unit/test_electrum_listener.py`, `tests/unit/test_confirmation_proof.py` - **[DEP-03]** **Soglia di conferme.** Una moneta è candidata all'accredito solo se è confermata (altezza maggiore di 0: le monete ancora in mempool sono ignorate), non è già registrata e ha raggiunto la sua soglia di @@ -313,9 +309,7 @@ monete sotto la soglia non compaiono nel saldo mostrato. si invia una notifica SSE. Se la verifica non riesce la moneta resta fuori dal saldo e viene riproposta all'aggiornamento successivo. *Codice:* `app/deposits/service.py:reinstate_reappeared_utxos`, `app/electrum/listener.py:ElectrumListener.refresh_user` · - *Test:* `tests/unit/test_deposits.py` - *Stato:* da implementare (B-83) · *Oggi:* il reintegro si basa sulla sola - risposta del server attivo, senza corroborazione. + *Test:* `tests/unit/test_deposits.py`, `tests/unit/test_confirmation_proof.py` - **[DEP-09]** **Due vie, un solo codice.** Un aggiornamento parte quando il server notifica un cambiamento su un indirizzo osservato, alla (re)iscrizione di un indirizzo, e al giro periodico: tutte e tre usano la diff --git a/docs/storico-bug.md b/docs/storico-bug.md index aa98125..95abded 100644 --- a/docs/storico-bug.md +++ b/docs/storico-bug.md @@ -231,7 +231,7 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)). | B-80 | Un solo indirizzo del montepremi per tutti i round | Critica | *da registrare* | `test_round_pool_address.py`, `test_pending_address.py`, `test_scheduler.py` | RND, BET, WDR | | B-81 | `drawing` resta bloccato dopo qualsiasi eccezione, non solo dopo un riavvio | Critica | **aperto** | — | RND | | B-82 | Un deposito annullato da una riorganizzazione dopo la scommessa rende il round impagabile; coinbase accreditate a 1 conferma | Critica | **aperto** | — | DEP, RND | -| B-83 | Conferme e reintegri si basano solo sul server attivo | Alta | **aperto** | — | ELC, TX, DEP | +| B-83 | Conferme e reintegri si basano solo sul server attivo | Alta | *da registrare* | `test_confirmation_proof.py`, `test_confirmation.py` | ELC, TX, DEP | | B-84 | Il quorum può ridursi a un solo server o a nessuno senza avvisi; certificati TLS non verificati | Alta | *da registrare* | `test_electrum_client.py`, `test_electrum_listener.py` | ELC, SEC | | B-85 | Le riorganizzazioni della catena non vengono né seguite né segnalate | Alta | **aperto** | — | ELC, RND | | B-86 | Un server connesso ma fermo non viene mai sostituito | Alta | **aperto** | — | ELC, OPS | @@ -252,9 +252,9 @@ cifrava di nuovo il file sbagliato (vedi [operativita.md](operativita.md)). - **Corretti:** B-01 … B-70, tranne B-16, che è chiuso nella sostanza (il crash è eliminato e `/guida` serve un segnaposto) ma lascia la guida vera - come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76, B-77, B-78. + come limite accettato; dalla revisione del 2026-09-28: B-88, B-75, B-97, B-80, B-84, B-76, B-77, B-78, B-83. - **Aperti:** B-71, B-72 (gravità bassa, operativi o di documentazione) e B-73 … B-98 dalla revisione del 2026-09-28 tranne i corretti (5 critici, - 7 alti, 6 medi), da implementare nell'ordine di + 6 alti, 6 medi), da implementare nell'ordine di [piano-correzioni.md](piano-correzioni.md). - **Prossimo numero libero:** `B-99`. diff --git a/docs/transazioni-e-riconciliazione.md b/docs/transazioni-e-riconciliazione.md index 9856c40..fe11034 100644 --- a/docs/transazioni-e-riconciliazione.md +++ b/docs/transazioni-e-riconciliazione.md @@ -167,9 +167,7 @@ monete: per indirizzo a ogni giro; un indirizzo che non si riesce a leggere non blocca gli altri. *Codice:* `app/tx/confirmation.py:poll_once`, `app/tx/pending_address.py:own_address_for` · - *Test:* `tests/unit/test_confirmation.py`, `tests/unit/test_round_pool_address.py`, `tests/unit/test_tx_versions.py` - *Stato:* da implementare (B-83) · *Oggi:* un'altezza maggiore di 0 - riportata dal solo server attivo basta a confermare ([TX-22]). + *Test:* `tests/unit/test_confirmation.py`, `tests/unit/test_round_pool_address.py`, `tests/unit/test_tx_versions.py`, `tests/unit/test_confirmation_proof.py` - **[TX-22]** **Conferma verificata.** Un candidato ([TX-06]) è confermato solo quando il quorum dei server ([connessione-electrum.md](connessione-electrum.md) (B-84)) restituisce per @@ -179,11 +177,8 @@ monete: il listener non lo tiene più, è quello su cui il quorum concorda. Senza quorum, o con una prova non valida, la riga resta `pending` e si riprova al giro successivo. - *Codice:* `app/tx/confirmation.py:poll_once`, `app/electrum/listener.py:ElectrumListener._corroborate_majority` · - *Test:* — - *Stato:* da implementare (B-83) · *Oggi:* basta la parola del server - attivo: un server ostile può dichiarare confermato un pagamento mai - trasmesso. + *Codice:* `app/tx/confirmation.py:poll_once`, `app/electrum/listener.py:ElectrumListener.confirmation_proven`, `app/electrum/merkle.py:proof_matches_header` · + *Test:* `tests/unit/test_confirmation_proof.py` - **[TX-07]** Alla conferma la riga passa a `confirmed` e, nella stessa transazione del database, viene chiamato il gestore del suo tipo; poi si invia una notifica SSE: @@ -381,10 +376,6 @@ riserva di [TX-11] serve proprio a dargli un'uscita da ridurre. riconciliatore non corregge da solo le righe collegate. *Codice:* `app/tx/reconcile.py:reconcile_once`, `app/tx/reconcile.py:_resurrect`, `app/rounds/pause.py:pause_automatically` · *Test:* `tests/unit/test_reconcile_proof.py`, `tests/unit/test_pause_reason.py` - *Stato:* da implementare (B-83), per la conferma con prova merkle · *Oggi:* - la conferma di una riga `failed` è quella dei server **altri dall'attivo**: - una maggioranza stretta di essi (almeno il minimo del quorum) elenca uno - dei suoi txid in un blocco, senza prova merkle ([TX-22]). - **[TX-27]** Il riconciliatore invia una notifica SSE solo per le righe di cui ha cambiato qualcosa (promozione, abbandono, riallineamento del txid), non per ogni riga esaminata. diff --git a/tests/unit/fake_network.py b/tests/unit/fake_network.py index 74b6846..1b582ba 100644 --- a/tests/unit/fake_network.py +++ b/tests/unit/fake_network.py @@ -13,10 +13,12 @@ class FakeNetwork: `transaction_known` answers (True: some server has it, False: every server denies it, None: too few answers to tell); fanned-out transactions are recorded.""" - def __init__(self, known: bool | None = False): + def __init__(self, known: bool | None = False, proven: bool = True): self.known = known + self.proven = proven # what `confirmation_proven` answers (B-83) self.asked: list[str] = [] self.fanned_out: list[str] = [] + self.proofs_asked: list[tuple[str, int]] = [] async def transaction_known(self, txid: str) -> bool | None: self.asked.append(txid) @@ -25,6 +27,10 @@ class FakeNetwork: def fan_out_transaction(self, raw_tx_hex: str) -> None: self.fanned_out.append(raw_tx_hex) + async def confirmation_proven(self, txid: str, height: int) -> bool: + self.proofs_asked.append((txid, height)) + return self.proven + MISSING_INPUTS = "bad-txns-inputs-missingorspent" @@ -39,8 +45,8 @@ class ScriptedNetwork(FakeNetwork): (active, others) pair of verdicts `rebroadcast_everywhere` returns: ("accepted", ""), ("refused", message) or None.""" - def __init__(self, *, active=(), others=((), ()), spender=None, rebroadcast=None, others_needed=2): - super().__init__(known=False) + def __init__(self, *, active=(), others=((), ()), spender=None, rebroadcast=None, others_needed=2, proven=True): + super().__init__(known=False, proven=proven) self.active = None if active is None else list(active) self.others = tuple(None if o is None else list(o) for o in others) self.spender = spender diff --git a/tests/unit/test_confirmation.py b/tests/unit/test_confirmation.py index 26912ae..a0c6b65 100644 --- a/tests/unit/test_confirmation.py +++ b/tests/unit/test_confirmation.py @@ -10,6 +10,7 @@ from app.db.base import Base from app.db.models import PendingTransaction, Round, RoundParticipant, User from app.electrum.scripthash import address_to_scripthash from app.tx.confirmation import poll_once +from tests.unit.fake_network import FakeNetwork from app.wallet.hd import derive_pool_address, derive_user_address @@ -82,7 +83,7 @@ async def test_bet_confirmation_marks_participant_confirmed(session_factory): await session.commit() client = FakeClient({"tx1": 1}) - confirmed = await poll_once(session_factory, client) + confirmed = await poll_once(session_factory, client, FakeNetwork()) assert confirmed == 1 async with session_factory() as session: @@ -109,7 +110,7 @@ async def test_unconfirmed_tx_is_left_pending(session_factory): await session.commit() client = FakeClient({"tx2": 0}) - confirmed = await poll_once(session_factory, client) + confirmed = await poll_once(session_factory, client, FakeNetwork()) assert confirmed == 0 async with session_factory() as session: @@ -128,7 +129,7 @@ async def test_payout_confirmation_closes_round(session_factory): await session.commit() client = FakeClient({"tx3": 2}) - confirmed = await poll_once(session_factory, client) + confirmed = await poll_once(session_factory, client, FakeNetwork()) assert confirmed == 1 async with session_factory() as session: @@ -181,7 +182,7 @@ async def test_one_unresolvable_candidate_does_not_block_the_others(session_fact exploding_scripthash = address_to_scripthash(derive_user_address(1)) confirmed = await poll_once( - session_factory, ExplodingClient({"good": 1}, exploding_scripthash=exploding_scripthash) + session_factory, ExplodingClient({"good": 1}, exploding_scripthash=exploding_scripthash), FakeNetwork() ) assert confirmed == 1 # the healthy one still got processed @@ -213,7 +214,7 @@ async def test_bet_confirms_after_an_rbf_bump_changed_the_txid(session_factory): ) await session.commit() - assert await poll_once(session_factory, FakeClient({"bumped-txid": 1})) == 1 + assert await poll_once(session_factory, FakeClient({"bumped-txid": 1}), FakeNetwork()) == 1 async with session_factory() as session: participant = (await session.scalars(select(RoundParticipant).where(RoundParticipant.round_id == 11))).one() @@ -231,7 +232,7 @@ async def test_payout_confirms_after_an_rbf_bump_changed_the_txid(session_factor ) await session.commit() - assert await poll_once(session_factory, FakeClient({"bumped-payout": 1})) == 1 + assert await poll_once(session_factory, FakeClient({"bumped-payout": 1}), FakeNetwork()) == 1 async with session_factory() as session: assert (await session.get(Round, 12)).status == "closed" @@ -264,6 +265,6 @@ async def test_poll_once_caches_history_per_scripthash(session_factory): call_count["n"] += 1 return [{"tx_hash": "tx-a", "height": 0}, {"tx_hash": "tx-b", "height": 0}] - await poll_once(session_factory, CountingClient()) + await poll_once(session_factory, CountingClient(), FakeNetwork()) assert call_count["n"] == 1 diff --git a/tests/unit/test_confirmation_proof.py b/tests/unit/test_confirmation_proof.py new file mode 100644 index 0000000..8fd7831 --- /dev/null +++ b/tests/unit/test_confirmation_proof.py @@ -0,0 +1,449 @@ +"""B-83 (TX-22, TX-06, ELC-25, ELC-26, DEP-02, DEP-08, AUD-18, ADR-07): confirmations and +reinstatements need the quorum, not the active server's word. + +`poll_once` used to mark a transaction confirmed on one server's `height > 0`, and +`reinstate_reappeared_utxos` restored a UTXO on its word — so a hostile active server could +declare a never-relayed payout confirmed (the winner unpaid, the round closed) or put a spent +coin back into a balance. Each test here fails against that. +""" + +import hashlib +import struct + +import pytest +from sqlalchemy import select + +import app.bets.confirmation # noqa: F401 (registers the "bet" handler) +from app.db.models import AuditLog, PendingTransaction, Round, RoundParticipant, User, UtxoEvent +from app.electrum.client import ElectrumEndpoint +from app.electrum.listener import ElectrumListener +from app.electrum.merkle import header_merkle_root, merkle_root_from_branch, proof_matches_header +from app.rounds.draw import header_meets_its_own_target +from app.tx.confirmation import poll_once +from app.wallet.hd import derive_user_address +from tests.unit.fake_network import FakeNetwork +from tests.unit.test_confirmation import FakeClient, session_factory # noqa: F401 (fixture and fake) +from tests.unit.test_electrum_listener import ( # noqa: F401 (fixture and helpers) + _ENDPOINTS, + _EASY_BITS, + _ActiveClient, + _seed_funded_user, + session_factory as listener_session_factory, +) + + +def _sha256d(data: bytes) -> bytes: + return hashlib.sha256(hashlib.sha256(data).digest()).digest() + + +class Block: + """A tiny block: `txids` (display order) under a real merkle tree, and a mined header + that commits to its root.""" + + def __init__(self, txids: list[str], prev_hash: str = "00" * 32): + self.txids = txids + level = [bytes.fromhex(t)[::-1] for t in txids] + self._levels = [level] + while len(level) > 1: + if len(level) % 2: + level = level + [level[-1]] + level = [_sha256d(level[i] + level[i + 1]) for i in range(0, len(level), 2)] + self._levels.append(level) + self.root = level[0] + self.header = self._mine(prev_hash) + + def _mine(self, prev_hash: str) -> str: + for nonce in range(100_000): + header = ( + struct.pack(" dict: + """What `blockchain.transaction.get_merkle` answers.""" + index = self.txids.index(txid) + position, branch = index, [] + for level in self._levels[:-1]: + padded = level + [level[-1]] if len(level) % 2 else level + branch.append(padded[position ^ 1][::-1].hex()) + position //= 2 + return {"block_height": height, "merkle": branch, "pos": index} + + +_TXIDS = [f"{i:02x}" * 32 for i in range(1, 8)] # 7 transactions: an odd level, so a duplicate sibling too +_BLOCK = Block(_TXIDS) + + +# --- the proof itself ----------------------------------------------------------------------- + + +@pytest.mark.parametrize("txid", _TXIDS) +def test_every_transaction_of_a_block_proves_against_its_header(txid): + assert proof_matches_header(txid, 100, _BLOCK.proof(txid, 100), _BLOCK.header) + + +def test_a_single_transaction_block_proves_with_an_empty_branch(): + block = Block(["ab" * 32]) + assert block.proof("ab" * 32, 5)["merkle"] == [] + assert proof_matches_header("ab" * 32, 5, block.proof("ab" * 32, 5), block.header) + + +def test_header_merkle_root_reads_the_committed_root(): + assert header_merkle_root(_BLOCK.header) == _BLOCK.root + + +def test_a_proof_does_not_prove_anything_it_was_not_made_for(): + txid = _TXIDS[2] + proof = _BLOCK.proof(txid, 100) + other_block = Block(["f0" * 32, "f1" * 32]) + assert not proof_matches_header(_TXIDS[3], 100, proof, _BLOCK.header) # another txid + assert not proof_matches_header(txid, 101, proof, _BLOCK.header) # another height + assert not proof_matches_header(txid, 100, proof, other_block.header) # another block + assert not proof_matches_header(txid, 100, {**proof, "pos": proof["pos"] ^ 1}, _BLOCK.header) # another position + assert not proof_matches_header(txid, 100, {**proof, "merkle": proof["merkle"][:-1]}, _BLOCK.header) + tampered = list(proof["merkle"]) + tampered[0] = "00" * 32 + assert not proof_matches_header(txid, 100, {**proof, "merkle": tampered}, _BLOCK.header) + + +@pytest.mark.parametrize( + "proof", + [None, "garbage", {}, {"block_height": 100}, {"block_height": 100, "merkle": "x", "pos": 0}, + {"block_height": 100, "merkle": ["zz"], "pos": 0}, {"block_height": 100, "merkle": ["00" * 32], "pos": 9}], +) +def test_a_malformed_proof_is_simply_a_no(proof): + assert proof_matches_header(_TXIDS[0], 100, proof, _BLOCK.header) is False + + +def test_the_root_computation_matches_a_hand_built_two_leaf_tree(): + a, b = "01" * 32, "02" * 32 + expected = _sha256d(bytes.fromhex(a)[::-1] + bytes.fromhex(b)[::-1]) + assert merkle_root_from_branch(a, [b], 0) == expected + assert merkle_root_from_branch(b, [a], 1) == expected + + +# --- the listener's verdict: confirmation_proven --------------------------------------------- + + +class _ProofServer: + """One other server: `proofs` maps txid -> the get_merkle reply; None = unreachable; + `headers` maps height -> the header it reports.""" + + def __init__(self, proofs=None, headers=None): + self._proofs = proofs + self._headers = headers or {} + + async def connect(self): + if self._proofs is None: + raise ConnectionRefusedError("down") + + async def get_merkle(self, txid, height): + return self._proofs[txid] + + async def get_block_header(self, height): + return self._headers[height] + + async def close(self): + pass + + +def _proof_listener(session_factory, others: dict[str, _ProofServer], *, tip_height=100, tip_header=None, **kwargs): + listener = ElectrumListener(lambda endpoint: others[endpoint.host], session_factory, _ENDPOINTS, **kwargs) + listener.tip_height = tip_height + listener.tip_header_hex = tip_header if tip_header is not None else _BLOCK.header + return listener + + +def _honest(): + txid = _TXIDS[3] + return _ProofServer({txid: _BLOCK.proof(txid, 100)}) + + +def _lying(): + """Claims the tx is in the block, with a proof that does not add up.""" + txid = _TXIDS[3] + proof = _BLOCK.proof(txid, 100) + return _ProofServer({txid: {**proof, "merkle": ["11" * 32] + proof["merkle"][1:]}}) + + +async def test_a_valid_proof_from_the_quorum_confirms(listener_session_factory): + listener = _proof_listener(listener_session_factory, {"second.example": _honest(), "third.example": _honest()}) + assert await listener.confirmation_proven(_TXIDS[3], 100) is True + + +async def test_a_lying_active_server_cannot_confirm_a_transaction(listener_session_factory): + """The active server claims height 100; the other servers' proofs do not check out. + The active server's own word is never part of the verdict.""" + listener = _proof_listener(listener_session_factory, {"second.example": _lying(), "third.example": _lying()}) + assert await listener.confirmation_proven(_TXIDS[3], 100) is False + + +async def test_a_minority_of_liars_does_not_outvote_the_honest_servers(listener_session_factory): + four = [*_ENDPOINTS, ElectrumEndpoint("fourth.example", 50002, True)] + listener = ElectrumListener( + lambda endpoint: {"second.example": _honest(), "third.example": _honest(), "fourth.example": _lying()}[endpoint.host], + listener_session_factory, + four, + ) + listener.tip_height, listener.tip_header_hex = 100, _BLOCK.header + assert await listener.confirmation_proven(_TXIDS[3], 100) is True + + flipped = ElectrumListener( + lambda endpoint: {"second.example": _lying(), "third.example": _lying(), "fourth.example": _honest()}[endpoint.host], + listener_session_factory, + four, + ) + flipped.tip_height, flipped.tip_header_hex = 100, _BLOCK.header + assert await flipped.confirmation_proven(_TXIDS[3], 100) is False + + +async def test_too_few_answers_do_not_confirm(listener_session_factory): + listener = _proof_listener(listener_session_factory, {"second.example": _honest(), "third.example": _ProofServer(None)}) + assert await listener.confirmation_proven(_TXIDS[3], 100) is False # one answer, below the quorum minimum + + +async def test_a_proof_for_another_height_does_not_confirm(listener_session_factory): + txid = _TXIDS[3] + wrong_height = _ProofServer({txid: _BLOCK.proof(txid, 99)}) + listener = _proof_listener(listener_session_factory, {"second.example": wrong_height, "third.example": wrong_height}) + assert await listener.confirmation_proven(txid, 100) is False + + +async def test_a_height_we_have_not_seen_cannot_be_proven(listener_session_factory): + listener = _proof_listener( + listener_session_factory, {"second.example": _honest(), "third.example": _honest()}, tip_height=99 + ) + assert await listener.confirmation_proven(_TXIDS[3], 100) is False # above our tip: no header to check against + + +async def test_an_older_block_is_checked_against_the_header_the_quorum_agrees_on(listener_session_factory): + txid = _TXIDS[3] + proof = _BLOCK.proof(txid, 90) + servers = { + "second.example": _ProofServer({txid: proof}, {90: _BLOCK.header}), + "third.example": _ProofServer({txid: proof}, {90: _BLOCK.header}), + } + listener = _proof_listener(listener_session_factory, servers, tip_height=100, tip_header="00" * 80) + assert await listener.confirmation_proven(txid, 90) is True + + # The servers disagree about that block's header: nothing trusted to check against. + other = Block(["f0" * 32, "f1" * 32]) + split = { + "second.example": _ProofServer({txid: proof}, {90: _BLOCK.header}), + "third.example": _ProofServer({txid: proof}, {90: other.header}), + } + listener = _proof_listener(listener_session_factory, split, tip_height=100, tip_header="00" * 80) + assert await listener.confirmation_proven(txid, 90) is False + + # A header that was never mined (does not meet its own target) is not accepted either. + unmined = "00" * 80 + fake = { + "second.example": _ProofServer({txid: proof}, {90: unmined}), + "third.example": _ProofServer({txid: proof}, {90: unmined}), + } + listener = _proof_listener(listener_session_factory, fake, tip_height=100, tip_header="00" * 80) + assert await listener.confirmation_proven(txid, 90) is False + + +async def test_the_reduced_rule_under_allow_no_quorum_accepts_a_single_proof(listener_session_factory): + listener = _proof_listener( + listener_session_factory, {"second.example": _honest(), "third.example": _ProofServer(None)}, allow_no_quorum=True + ) + assert await listener.confirmation_proven(_TXIDS[3], 100) is True + + +# --- poll_once: the active server only nominates -------------------------------------------------- + + +async def _seed_pending_bet(session_factory, txid: str = "tx1"): + async with session_factory() as session: + user = User(username="u0", password_hash="x", derivation_index=0, address=derive_user_address(0)) + session.add(user) + await session.flush() + session.add(Round(id=1, status="open")) + session.add(RoundParticipant(round_id=1, user_id=user.id, bet_amount_sats=1_000, bet_txid=txid, status="broadcast")) + session.add( + PendingTransaction( + kind="bet", round_id=1, user_id=user.id, current_txid=txid, fee_rate_sat_vb=1, raw_tx_hex="00", status="pending" + ) + ) + await session.commit() + + +async def test_a_height_the_quorum_cannot_prove_leaves_the_row_pending(session_factory): + """A hostile active server reports a bet (or a payout) as mined at some height. The + quorum does not prove it, so nothing is confirmed and no handler runs.""" + await _seed_pending_bet(session_factory) + network = FakeNetwork(proven=False) + + assert await poll_once(session_factory, FakeClient({"tx1": 100}), network) == 0 + + assert network.proofs_asked == [("tx1", 100)] # the claim of the active server was put to the quorum + async with session_factory() as session: + assert (await session.scalars(select(PendingTransaction))).one().status == "pending" + assert (await session.scalars(select(RoundParticipant))).one().status == "broadcast" + + +async def test_a_proven_height_confirms(session_factory): + await _seed_pending_bet(session_factory) + network = FakeNetwork(proven=True) + assert await poll_once(session_factory, FakeClient({"tx1": 100}), network) == 1 + assert network.proofs_asked == [("tx1", 100)] + async with session_factory() as session: + assert (await session.scalars(select(RoundParticipant))).one().status == "confirmed" + + +async def test_nothing_is_confirmed_without_a_network_to_prove_it(session_factory): + await _seed_pending_bet(session_factory) + assert await poll_once(session_factory, FakeClient({"tx1": 100}), None) == 0 + async with session_factory() as session: + assert (await session.scalars(select(PendingTransaction))).one().status == "pending" + + +async def test_an_unconfirmed_transaction_is_not_put_to_the_quorum(session_factory): + await _seed_pending_bet(session_factory) + network = FakeNetwork() + assert await poll_once(session_factory, FakeClient({"tx1": 0}), network) == 0 + assert network.proofs_asked == [] + + +async def test_poll_once_confirms_through_a_real_listener_and_a_lying_active_server(session_factory, listener_session_factory): + """End to end with the real quorum code: the transaction is really in the block, the + other servers prove it, and the same call refuses when they do not.""" + txid = _TXIDS[3] + await _seed_pending_bet(session_factory, txid=txid) + + liars = _proof_listener(listener_session_factory, {"second.example": _lying(), "third.example": _lying()}) + assert await poll_once(session_factory, FakeClient({txid: 100}), liars) == 0 + + honest = _proof_listener(listener_session_factory, {"second.example": _honest(), "third.example": _honest()}) + assert await poll_once(session_factory, FakeClient({txid: 100}), honest) == 1 + + +# --- reinstatement (ELC-25, DEP-02, DEP-08, AUD-18) ------------------------------------------------------ + + +_FLAGGED_TXID = "dd" * 32 +_ADDRESS = "plm1q0xcqpzrky6eff2g52qdye53xkk9jxkvraxkkwd" + + +class _UnspentServer: + def __init__(self, entries): + self._entries = entries + + async def connect(self): + if self._entries is None: + raise ConnectionRefusedError("down") + + async def listunspent(self, scripthash): + return self._entries + + async def close(self): + pass + + +_BACK = {"tx_hash": _FLAGGED_TXID, "tx_pos": 0, "height": 100, "value": 20_000_000} +_OTHER = {"tx_hash": "99" * 32, "tx_pos": 0, "height": 100, "value": 1_000_000} # keeps the reply non-empty + + +async def _flagged_user(session_factory) -> int: + user_id = await _seed_funded_user(session_factory, username="carol", address=_ADDRESS) + async with session_factory() as session: + utxo = (await session.scalars(select(UtxoEvent).where(UtxoEvent.user_id == user_id))).one() + utxo.spent_txid = "external-spend" + user = await session.get(User, user_id) + user.cached_balance_sats = 0 + await session.commit() + return user_id + + +def _reinstate_listener(session_factory, others: dict[str, _UnspentServer], active_entries): + listener = ElectrumListener(lambda endpoint: others[endpoint.host], session_factory, _ENDPOINTS) + listener.client = _ActiveClient(active_entries) + return listener + + +async def _state(session_factory, user_id): + async with session_factory() as session: + utxo = (await session.scalars(select(UtxoEvent).where(UtxoEvent.txid == _FLAGGED_TXID))).one() + events = [e.event_type for e in (await session.scalars(select(AuditLog))).all()] + return utxo.spent_txid, (await session.get(User, user_id)).cached_balance_sats, events + + +async def test_a_reappeared_coin_is_reinstated_once_the_quorum_confirms_it(listener_session_factory): + user_id = await _flagged_user(listener_session_factory) + others = {"second.example": _UnspentServer([_BACK, _OTHER]), "third.example": _UnspentServer([_BACK, _OTHER])} + listener = _reinstate_listener(listener_session_factory, others, [_BACK, _OTHER]) + + await listener.refresh_user(user_id, "scripthash") + + spent, balance, events = await _state(listener_session_factory, user_id) + assert spent is None and balance == 20_000_000 + 1_000_000 # the coin is back (+ the unrelated credit) + assert "utxo_external_spend_reinstated" in events + + +@pytest.mark.parametrize( + "others", + [ + pytest.param({"second.example": [_OTHER], "third.example": [_OTHER]}, id="others-do-not-list-it"), + pytest.param({"second.example": [_BACK, _OTHER], "third.example": [_OTHER]}, id="a-minority-lists-it"), + pytest.param({"second.example": [_BACK, _OTHER], "third.example": None}, id="too-few-answers"), + pytest.param( + {"second.example": [{**_BACK, "value": 1}, _OTHER], "third.example": [{**_BACK, "value": 1}, _OTHER]}, + id="other-amount", + ), + pytest.param( + {"second.example": [{**_BACK, "height": 0}, _OTHER], "third.example": [{**_BACK, "height": 0}, _OTHER]}, + id="unconfirmed", + ), + ], +) +async def test_a_hostile_active_server_cannot_reinstate_a_coin_alone(listener_session_factory, others): + """The active server lists the coin again; the quorum does not bear it out. Before the + fix the reinstatement was written on the active server's word alone.""" + user_id = await _flagged_user(listener_session_factory) + servers = {host: _UnspentServer(entries) for host, entries in others.items()} + listener = _reinstate_listener(listener_session_factory, servers, [_BACK, _OTHER]) + + await listener.refresh_user(user_id, "scripthash") + + spent, balance, events = await _state(listener_session_factory, user_id) + assert spent == "external-spend" # still written off + assert "utxo_external_spend_reinstated" not in events + + +async def test_a_coin_listed_with_another_amount_than_recorded_is_not_even_a_candidate(listener_session_factory): + from app.deposits.service import find_reinstate_candidates + + user_id = await _flagged_user(listener_session_factory) + async with listener_session_factory() as session: + assert await find_reinstate_candidates(session, user_id, [_BACK]) == [_BACK] + assert await find_reinstate_candidates(session, user_id, [{**_BACK, "value": 5}]) == [] + assert await find_reinstate_candidates(session, user_id, [{**_BACK, "height": 0}]) == [] + assert await find_reinstate_candidates(session, user_id, [_OTHER]) == [] + + +async def test_finding_candidates_writes_nothing(listener_session_factory): + from app.deposits.service import find_reinstate_candidates + + user_id = await _flagged_user(listener_session_factory) + async with listener_session_factory() as session: + await find_reinstate_candidates(session, user_id, [_BACK]) + spent, balance, events = await _state(listener_session_factory, user_id) + assert spent == "external-spend" and balance == 0 and "utxo_external_spend_reinstated" not in events + + +async def test_the_reinstatement_write_rechecks_that_the_mark_is_still_there(listener_session_factory): + """DEP-08: a coin whose mark was removed meanwhile (or never carried one) is not touched.""" + from app.deposits.service import reinstate_reappeared_utxos + + user_id = await _seed_funded_user(listener_session_factory, username="dave", address=_ADDRESS) # not flagged + async with listener_session_factory() as session: + assert await reinstate_reappeared_utxos(session, user_id, [_BACK]) == 0 diff --git a/tests/unit/test_reconcile_proof.py b/tests/unit/test_reconcile_proof.py index 6542eb6..4976ba7 100644 --- a/tests/unit/test_reconcile_proof.py +++ b/tests/unit/test_reconcile_proof.py @@ -326,22 +326,23 @@ async def test_an_earlier_version_of_a_failed_row_that_confirms_is_resurrected_t assert row.status == "confirmed" and row.current_txid == "older-txid" -async def test_a_failed_row_the_quorum_does_not_confirm_stays_failed(session_factory): - """Only the active server (or only a minority of the others) says it is mined.""" +async def test_a_failed_row_is_resurrected_only_when_the_quorum_proves_it(session_factory): + """The histories only name a claim (txid, height); the quorum's merkle proofs decide + (B-83, TX-22). A claim made by the active server alone, or one the proofs do not bear + out, leaves the row failed and the lottery running.""" await _seed_failed_withdrawal(session_factory) mined = [{"tx_hash": "wdtxid", "height": 321}] - hostile_active = ScriptedNetwork(active=mined, others=([], [])) - assert await reconcile_once(session_factory, UnknownTxClient(), hostile_active) == 0 - minority = ScriptedNetwork(active=mined, others=(mined, [], []), others_needed=2) - assert await reconcile_once(session_factory, UnknownTxClient(), minority) == 0 - too_few = ScriptedNetwork(active=mined, others=(mined, None), others_needed=2) - assert await reconcile_once(session_factory, UnknownTxClient(), too_few) == 0 + unproven = ScriptedNetwork(active=mined, others=([], []), proven=False) + assert await reconcile_once(session_factory, UnknownTxClient(), unproven) == 0 + assert unproven.proofs_asked == [("wdtxid", 321)] # the claim was put to the quorum + + assert await reconcile_once(session_factory, UnknownTxClient()) == 0 # no network: nothing to prove with assert (await _row(session_factory)).status == "failed" async with session_factory() as session: config = (await session.scalars(select(RoundConfig))).first() - assert config is None or not config.paused # no pause without the quorum + assert config is None or not config.paused # no pause without the proof async def test_a_failed_row_is_watched_for_fourteen_days_and_no_longer(session_factory): diff --git a/tests/unit/test_sqlite_transactions.py b/tests/unit/test_sqlite_transactions.py index abb9e6a..216b236 100644 --- a/tests/unit/test_sqlite_transactions.py +++ b/tests/unit/test_sqlite_transactions.py @@ -788,11 +788,11 @@ async def test_confirmation_poller_writes_under_the_lock(session_factory, funded async with _holding_write_lock(session_factory): with pytest.raises(OperationalError): - await poll_once(session_factory, Client()) + await poll_once(session_factory, Client(), FakeNetwork()) async with session_factory() as session: assert (await session.get(PendingTransaction, row_id)).status == "pending" - assert await poll_once(session_factory, Client()) == 1 + assert await poll_once(session_factory, Client(), FakeNetwork()) == 1 async with session_factory() as session: assert (await session.get(PendingTransaction, row_id)).status == "confirmed" assert (await session.scalar(select(Withdrawal.status))) == "confirmed" diff --git a/tests/unit/test_tx_versions.py b/tests/unit/test_tx_versions.py index bc15fd2..ff8a8a9 100644 --- a/tests/unit/test_tx_versions.py +++ b/tests/unit/test_tx_versions.py @@ -273,7 +273,7 @@ async def test_a_crash_between_the_broadcast_and_the_switch_over_leaves_the_repl # The network confirms the candidate: the poller recognises it and realigns the row. client.history = [{"tx_hash": candidate, "height": 101}] - assert await poll_once(session_factory, client) == 1 + assert await poll_once(session_factory, client, FakeNetwork()) == 1 async with session_factory() as session: row = await session.get(PendingTransaction, seeded["pending_id"]) assert row.status == "confirmed" and row.current_txid == candidate @@ -388,7 +388,7 @@ async def test_the_confirmation_of_an_earlier_version_confirms_the_row_and_retar third = await bump_fee(session_factory, client, seeded["pending_id"]) client.history = [{"tx_hash": seeded["first_txid"], "height": 77}] - assert await poll_once(session_factory, client) == 1 + assert await poll_once(session_factory, client, FakeNetwork()) == 1 async with session_factory() as session: row = await session.get(PendingTransaction, seeded["pending_id"]) @@ -404,7 +404,7 @@ async def test_an_unrelated_txid_in_the_history_does_not_confirm_the_row(session seeded = await _seed_pending_bet(session_factory) client = _Chain() client.history = [{"tx_hash": "ab" * 32, "height": 5}] - assert await poll_once(session_factory, client) == 0 + assert await poll_once(session_factory, client, FakeNetwork()) == 0 async with session_factory() as session: assert (await session.get(PendingTransaction, seeded["pending_id"])).status == "pending"