Instead of having a separate field to derive the bip86 base key, we return it in the hsmd init reply once we know that the hsm_secret is of mnemonic type
Simplify wallet address generation by using a unified approach where
the derivation method (BIP86 vs BIP32) is determined by the wallet's
HSM secret type rather than having separate address types.
Add TLV field to hsmd_init_reply_v4 to communicate the HSM secret type
(mnemonic vs legacy) from HSM to lightningd. This allows lightningd to
automatically determine whether to use BIP86 or BIP32 derivation without
needing separate address types.
This simplifies the UTXO type system by removing the separate BIP86
enum value. P2TR addresses will now use unified derivation logic
based on the wallet's HSM secret type rather than having separate
enum values."
This commit fixes an issue where BIP86 addresses were not being
discovered during wallet recovery/rescan operations.
The root cause was that init_txfilter() only populated the transaction
filter with BIP32-derived keys, preventing lightningd from recognizing
BIP86 UTXOs during blockchain scans. Now both BIP32 and BIP86 derived
scripts are included in the filter when BIP86 derivation is enabled.
This ensures that wallets restored from BIP39 mnemonics can properly
discover and display previously funded BIP86 addresses without requiring
manual address generation first.
[ We also move the slightly-lost comment about libbacktrace so it is
where we actually include <backtrace.h> --RR ]
Add the UTXO_P2TR_BIP86 in preparation to add BIP86 wallet functions such as newaddr, listaddr etc. We also add a new index in the database for BIP86 as this is using a completely different derivation path and hsm_secret.
RIP to this commit there's a good chance a lot of this code doesn't even make this into the final PR. Pour one out for the fallen lines of code.
This commit is doing the rest of the derivation. There was a significant overlap between the bip32_pubkey derivation and the bip86_pubkey derivation so that has been refactored in one place.
Changelog-Changed: hsmd: New nodes will now be created with a BIP-39 12-word phrase as their root secret.
Changelog-Deprecated: config: `encrypted-hsm` to require a passphrase (use `hsm-passphrase`).
Changelog-Added: config: `hsm-passphrase` indicates we should use a manual passphrase with the hsm secret.
You can now simply add per-tal-object helpers for memleak, but our older pattern required
calling memleak functions explicitly during memleak handling. Hash tables in particular need
to be dynamically allocated (we override the allocators using htable_set_allocator and assume
this), so it makes sense to have a helper macro that does all three.
This eliminates a huge amount of code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We watch if they are to do with a channel, or have outputs going to us, but otherwise
we didn't, so we never updated the blockheight in the db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `listtransactions` now correctly updates `blockheight` for txs created by `sendpsbt` which have no change outputs.
This happens if the channel is *not* announcable yet. Then we hit the assertion
in funding_depth_cb that the txid is the same as the current funding.txid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: fixed crash when we splice a channel which hasn't been announced yet.
Basically, `devtools/reduce-includes.sh */*.c`.
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
After:
real 0m37.872000-39.974000(39.5466+/-0.59)s
user 3m1.211000-14.968000(12.4556+/-3.9)s
sys 0m35.008000-36.830000(36.4143+/-0.5)s
Build time after touch config.vars (RUST=0):
Before:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
After:
real 0m18.384000-21.307000(20.8605+/-0.92)s
user 2m5.585000-26.843000(23.6017+/-6.7)s
sys 0m19.650000-22.003000(21.4943+/-0.69)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Each header should only include the other headers it needs to compile;
`devtools/reduce-includes.sh */*.h` does this. The C files then need
additional includes if they don't compile.
And remove the entirely useless wire/onion_wire.h, which only serves to include wire/onion_wiregen.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to manually choose what to link against,
which is much of the complexity of our Makefiles: the compiler will
automatically use any object files it needs to link.
We already do this for ccan as libccan.a, now we have libcommon.a.
We don't link against it for *everything*, as some tests require their own
versions.
Notes:
1. I get rid of the weird plugins/test/Makefile2 (accidental commit?)
2. Many tests change due to update-mocks.
3. In some places I added the missing dependency on the Makefile itself, though most are in the next
patch.
Before:
Total program size: 221366528
Total tests size: 364243856
After:
Total program size: 190733656
Total tests size: 337880888
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.227000-44.245000(41.8222+/-1.6)s
user 3m2.105000-33.696000(23.1442+/-8.4)s
sys 0m35.054000-42.269000(39.7231+/-2)s
After:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
Build time after touch config.vars (RUST=0):
Before:
real 0m18.928000-22.776000(21.5084+/-1.1)s
user 2m8.613000-36.567000(27.7281+/-7.7)s
sys 0m20.458000-23.436000(22.3963+/-0.77)s
After:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rusty@rusty-Framework:~/devel/cvs/lightni
Testing a large db shows Postgres slowing down exponentially as
it inserts the channel_events. Rather than updating the index in the
db every time, do it at the end, for spectacular speedup:
```
lightningd-1 2025-10-08T05:39:44.333Z INFO lightningd: Creating database
lightningd-1 2025-10-08T05:39:47.581Z DEBUG lightningd: Transferring 6166 chain_events
lightningd-1 2025-10-08T05:39:48.455Z DEBUG lightningd: Transferring 1660043 channel_events
lightningd-1 2025-10-08T05:39:54.390Z INFO lightningd: Inserted 103100/1660043 channel_events
lightningd-1 2025-10-08T05:40:04.390Z INFO lightningd: Inserted 283280/1660043 channel_events
lightningd-1 2025-10-08T05:40:14.390Z INFO lightningd: Inserted 464065/1660043 channel_events
lightningd-1 2025-10-08T05:40:24.390Z INFO lightningd: Inserted 629559/1660043 channel_events
lightningd-1 2025-10-08T05:40:34.390Z INFO lightningd: Inserted 800659/1660043 channel_events
lightningd-1 2025-10-08T05:40:44.390Z INFO lightningd: Inserted 975433/1660043 channel_events
lightningd-1 2025-10-08T05:40:54.390Z INFO lightningd: Inserted 1134719/1660043 channel_events
lightningd-1 2025-10-08T05:41:04.390Z INFO lightningd: Inserted 1290549/1660043 channel_events
lightningd-1 2025-10-08T05:41:14.390Z INFO lightningd: Inserted 1443304/1660043 channel_events
lightningd-1 2025-10-08T05:41:24.390Z INFO lightningd: Inserted 1590013/1660043 channel_events
lightningd-1 2025-10-08T05:41:29.148Z INFO lightningd: bookkeeper migration complete: migrated 6166 chainmoves, 1660043 channelmoves, 132481 descriptions
```
Now we complete the entire migration in 1 minute 45 seconds.
Thanks to @michael1101 for reporting this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: db: migration from v25.09 on a reasonable size account database could take almost infinite time.
When we migrate from accounts.db, we use the `account_nonchannel_id`
field. But we can replay the block chain and the channel involved is
still open, we will use the `account_channel_id` field, and our duplicate
detection fails.
As a result, we can end up with duplicate entries in the database, which
make accounting incorrect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `listchainmoves` could contain bogus duplicate entries after 25.09 bookkeeper migration.
For old channels, this can take a while, and it stops everything. But
we are only doing this to save space; it's not a *functional* necessity.
A quick and dirty test with 50,000 htlcs shows the htlc deletion took
450msec. I tried adding an index, and changing it to set hstate to
HTLC_STATE_INVALID instead of deleting entries, but it still took about 350ms.
Whereas the "COUNT(*)" only took 1.7msec, so it's worth keeping.
Reported-by: @michael1011
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: lightningd: we defer deletion of old htlcs on channel close, to avoid pausing for a long time (we clean them on startup)
Fixes: https://github.com/ElementsProject/lightning/issues/7962
This doesn't happen yet, since we delete all HTLCs when we close a channel. But we're
about to change that, so update the wallet_htlcs_first() code to avoid them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
One issue we have in CI is reconnection races: if an incoming
connection arrives while an outgoing one is negotiated, we close the
outgoing one and issue a disconnect, which fails any connect attempts.
By sending a "reconnected" message instead of disconnect/connect we
can avoid disturbing in-progress connection attempts which happens in CI
quite a bit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Lisa *told* me about this on review, and I ignored it. Fool: took an extra day to get the account.db which was triggering this so I could see the problem.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If they ran off master, currency can be null:
```
2025-08-21T10:03:04.566Z **BROKEN** lightningd: bookkeper migration: Accessing a null column e.currency/7 in query SELECT e.id, e.account_id, a.name, e.tag, e.credit, e.debit, e.fees, e.currency, e.payment_id, e.part_id, e.timestamp, e.ev_desc, e.rebalance_id FROM channel_events e LEFT OUTER JOIN accounts a ON a.id = e.account_id ORDER BY e.timestamp, e.id;
```
So allow this, but *also* check if it's a different currency and skip. This won't happen: you had to manually inject events in a different currency.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that this can only happen if you ran a master commit before rc1:
```
2025-08-21T10:03:03.255Z **BROKEN** lightningd: bookkeper migration: Accessing a null column e.ignored/15 in query SELECT e.id, e.account_id, a.name, e.origin, e.tag, e.credit, e.debit, e.output_value, e.currency, e.timestamp, e.blockheight, e.utxo_txid, e.outnum, e.spending_txid, e.payment_id, e.ignored, e.stealable, e.ev_desc, e.spliced, a.closed_count, a.peer_id, a.we_opened FROM chain_events e LEFT OUTER JOIN accounts a ON e.account_id = a.id ORDER BY e.timestamp, e.id;
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you have run recent master, upgrade will fail with:
```
Cannot migrate account database version 18
```
The final migration is:
```
/* We used to send anchors to the wallet, but set ignored tag. Now we send
* them to external. */
{SQL("UPDATE chain_events"
" SET account_id = (SELECT id FROM accounts WHERE name = 'external')"
" WHERE account_id = (SELECT id FROM accounts WHERE name = 'wallet')"
" AND ignored = 1"), NULL},
```
Which is harmless (we do that upgrade ourselves on migration, but if it's done already that will have no effect).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't have an accountdb from bookkeeper:
1. Generate a deposit chain event for every confirmed UTXO.
2. Generate an open chain event for every open, confirmed channel.
3. Generate a push/lease event if necessary.
4. Generate a fixup "journal" entry if balance is different from initial.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We take over the --bookkeeper-dir and --bookkeeper-db options, and
then if we can find the bookkeeper db we extract the records to
initialize our chain_moves and channel_moves tables.
Of course, bookkeeper now needs to not register those options.
When bookkeeper gets invoked the first time, it will reconstruct
everything from listchannelmoves and listcoinmoves. It cannot
preserve manually-added descriptions, so we put those in the datastore
for it ready to go.
Note that the order of onchain_fee changes slightly from the original.
But this is fine.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And gracefully fail for this case.
There's no such thing for Postgres, but that's because dbs need to be
set up by the admin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Only makes sense to wait on creation, since they neither are deleted
nor updated.
We also enhance the list commands to take the standard index options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `wait`: new subsystems `chainmoves` and `channelmoves`.
Iterating through every peer and channel every time can be very slow
for large nodes, when calling wallet_coinmoves_extract for listcoinmoves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We change notify_chain_mvt to wallet_save_chain_mvt, and
notify_channel_mvt to wallet_save_channel_mvt, which save to the db
and call the notifier themselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that bookkeeper de-duplicates chain_moves: we need to too! So we add
an index to make this efficient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
All the core notifications changed over to wrapping the notification
fields in an object with the name of the notification, but notifications
from plugins were missed.
Changelog-Added: Plugins: `channel_hint_update`, `pay_failure` and `pay_success` notifications now have objects of the same name containing the expected fields.
Changelog-Deprecated: Plugins: `channel_hint_update`, `pay_failure` and `pay_success` notification fields outside the same-named object.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to store them in the db this way, so I thought I'd see what it looks like if
we lift that interface all the way through.
We use a struct, so that types are checked strictly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we can keep a pointer to the channel directly, *or* a string.
This avoids gratuitous formatting (on creation) and lookups (later).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than converting to a generic coin_mvt struct, use these directly in
the notification, which is more explicit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
