When we migrate from accounts.db, we use the `account_nonchannel_id`
field. But we can replay the block chain and the channel involved is
still open, we will use the `account_channel_id` field, and our duplicate
detection fails.
As a result, we can end up with duplicate entries in the database, which
make accounting incorrect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `listchainmoves` could contain bogus duplicate entries after 25.09 bookkeeper migration.
For old channels, this can take a while, and it stops everything. But
we are only doing this to save space; it's not a *functional* necessity.
A quick and dirty test with 50,000 htlcs shows the htlc deletion took
450msec. I tried adding an index, and changing it to set hstate to
HTLC_STATE_INVALID instead of deleting entries, but it still took about 350ms.
Whereas the "COUNT(*)" only took 1.7msec, so it's worth keeping.
Reported-by: @michael1011
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: lightningd: we defer deletion of old htlcs on channel close, to avoid pausing for a long time (we clean them on startup)
Fixes: https://github.com/ElementsProject/lightning/issues/7962
This doesn't happen yet, since we delete all HTLCs when we close a channel. But we're
about to change that, so update the wallet_htlcs_first() code to avoid them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
One issue we have in CI is reconnection races: if an incoming
connection arrives while an outgoing one is negotiated, we close the
outgoing one and issue a disconnect, which fails any connect attempts.
By sending a "reconnected" message instead of disconnect/connect we
can avoid disturbing in-progress connection attempts which happens in CI
quite a bit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Lisa *told* me about this on review, and I ignored it. Fool: took an extra day to get the account.db which was triggering this so I could see the problem.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If they ran off master, currency can be null:
```
2025-08-21T10:03:04.566Z **BROKEN** lightningd: bookkeper migration: Accessing a null column e.currency/7 in query SELECT e.id, e.account_id, a.name, e.tag, e.credit, e.debit, e.fees, e.currency, e.payment_id, e.part_id, e.timestamp, e.ev_desc, e.rebalance_id FROM channel_events e LEFT OUTER JOIN accounts a ON a.id = e.account_id ORDER BY e.timestamp, e.id;
```
So allow this, but *also* check if it's a different currency and skip. This won't happen: you had to manually inject events in a different currency.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that this can only happen if you ran a master commit before rc1:
```
2025-08-21T10:03:03.255Z **BROKEN** lightningd: bookkeper migration: Accessing a null column e.ignored/15 in query SELECT e.id, e.account_id, a.name, e.origin, e.tag, e.credit, e.debit, e.output_value, e.currency, e.timestamp, e.blockheight, e.utxo_txid, e.outnum, e.spending_txid, e.payment_id, e.ignored, e.stealable, e.ev_desc, e.spliced, a.closed_count, a.peer_id, a.we_opened FROM chain_events e LEFT OUTER JOIN accounts a ON e.account_id = a.id ORDER BY e.timestamp, e.id;
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you have run recent master, upgrade will fail with:
```
Cannot migrate account database version 18
```
The final migration is:
```
/* We used to send anchors to the wallet, but set ignored tag. Now we send
* them to external. */
{SQL("UPDATE chain_events"
" SET account_id = (SELECT id FROM accounts WHERE name = 'external')"
" WHERE account_id = (SELECT id FROM accounts WHERE name = 'wallet')"
" AND ignored = 1"), NULL},
```
Which is harmless (we do that upgrade ourselves on migration, but if it's done already that will have no effect).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't have an accountdb from bookkeeper:
1. Generate a deposit chain event for every confirmed UTXO.
2. Generate an open chain event for every open, confirmed channel.
3. Generate a push/lease event if necessary.
4. Generate a fixup "journal" entry if balance is different from initial.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We take over the --bookkeeper-dir and --bookkeeper-db options, and
then if we can find the bookkeeper db we extract the records to
initialize our chain_moves and channel_moves tables.
Of course, bookkeeper now needs to not register those options.
When bookkeeper gets invoked the first time, it will reconstruct
everything from listchannelmoves and listcoinmoves. It cannot
preserve manually-added descriptions, so we put those in the datastore
for it ready to go.
Note that the order of onchain_fee changes slightly from the original.
But this is fine.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And gracefully fail for this case.
There's no such thing for Postgres, but that's because dbs need to be
set up by the admin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Only makes sense to wait on creation, since they neither are deleted
nor updated.
We also enhance the list commands to take the standard index options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `wait`: new subsystems `chainmoves` and `channelmoves`.
Iterating through every peer and channel every time can be very slow
for large nodes, when calling wallet_coinmoves_extract for listcoinmoves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We change notify_chain_mvt to wallet_save_chain_mvt, and
notify_channel_mvt to wallet_save_channel_mvt, which save to the db
and call the notifier themselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that bookkeeper de-duplicates chain_moves: we need to too! So we add
an index to make this efficient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
All the core notifications changed over to wrapping the notification
fields in an object with the name of the notification, but notifications
from plugins were missed.
Changelog-Added: Plugins: `channel_hint_update`, `pay_failure` and `pay_success` notifications now have objects of the same name containing the expected fields.
Changelog-Deprecated: Plugins: `channel_hint_update`, `pay_failure` and `pay_success` notification fields outside the same-named object.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to store them in the db this way, so I thought I'd see what it looks like if
we lift that interface all the way through.
We use a struct, so that types are checked strictly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we can keep a pointer to the channel directly, *or* a string.
This avoids gratuitous formatting (on creation) and lookups (later).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than converting to a generic coin_mvt struct, use these directly in
the notification, which is more explicit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Add serializing and deserializing of the extra tlvs to to the
htlc_accepted_hook to allow plugin users to replace the tlv stream that
is attached to the update_add_htlc message on forwards.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
This appends the extra_tlvs to the internal channeld_offer_htlc wire
msg. We also recombine the extra_tlvs with the blinded path key for
forwarding htlcs.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
json_tok_streq(…) and json_get_member(…) are convenience wrappers for
json_tok_strneq(…) and json_get_membern(…) respectively. Unfortunately, using
them incurs a performance penalty in the common case where they are called with
a string literal argument because the compiler is unable to substitute a
compile-time constant in place of the buried call to strlen(…).
For example,
json_get_member(buf, tok, "example");
…will have worse performance than…
json_get_membern(buf, tok, "example", strlen("example"));
…because the former is forced to scan over "example" at run-time to count its
length whereas the latter is able to elide the strlen(…) call at compile time.
Hoist these convenience functions up into common/json_parse_simple.h and mark
them as inline so that the compiler can elide the strlen(…) call in the common
case of calling these functions with a string literal argument.
Changelog-None
We allowed NULL for stub channels, but just don't put the stub scid
into the hash tables. This cleans up all the callers to make it
clear this is a non-optional parameter.
We opencode channel_set_random_local_alias, since there's only one caller now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have a migration which ensures this, but then I discovered that did
*not* address channels without an SCID yet. So fixed the migration, and
simpligied the code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There can be any number of these, and it will be useful to allow
routing by older scids (when other nodes haven't seen our gossip, or
even before we *can* announce the new post-splice channel).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we had to use the number to the db_bind call, these annotations made
sense, but since 0bcff1e76d (for v23.08) we
removed that.
So remove all the counters, which are simple overhead if we want to
change something.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to use a check on the active psbt to see if our splice signature was in it — but now we need to generate the signature early.
So we have to add a field tracking if we’ve sent it and add it to the database, wire protocols, and inflight objects.
Now you can grep for 'sqlite3 version' and see where we would like
to update.
Debian 11 (Bullseye) and Ubuntu 20.04 (Focal) ship with SQLite 3.31.1.
RHEL 9 ships with 3.34.1. Fedora 38+ uses SQLite 3.40+.
Unfortunately, RHEL8 ships with 3.26.0, and is still on maintenance Support
(security fixes, no new features): runs until May 31, 2029.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The spec used to say you had to wait for channel to be ready, *and* 6
depth before exchanging signatures. Now the 6 depth requirement is only
on the actual announcing of the channel: you can send sigs any time.
This means our state machine goes from:
NOT_USABLE -> NOT_DEEP_ENOUGH -> NEED_PEER_SIGS -> ANNOUNCED
to:
NOT_USABLE -> NEED_PEER_SIGS -> NOT_DEEP_ENOUGH -> ANNOUNCED
However, this revealed that our state machine is insufficient, so
rework it to be more general and understandable. In particular,
check for unexpected state transitions, and thus document them.
Note that cg->sent_sigs replaces channel->replied_to_announcement_sigs,
too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now exchange `announcement_signatures` as soon as we're ready, rather than waiting for 6 blocks (as per recent BOLT update)
signmessagewithkey: allows to sign a message with a key associated with
one bitcoin address in our wallet.
Changelog-Added: add a new rpc command signmessagewithkey to sign input messages with keys from our wallet.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
We use this for anchors, in which case we have a minimum value for
change. If we don't take this into account, we end up with a lower
feerate once we actually create the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
