This commit is updating hsmtool and exposesecrets to use the new pattern for storing the secret, which is the secret_data and secret_len, to support both 64 byte and 32 byte seeds.
In the case where we receive a taproot utxo we want to be able to tell if it was derived using a BIP32 seed or a BIP86 seed. Considering we will only be supporting BI86 type wallet addresses for mnemonics we can check if the out secret is 64 bytes long and if it is we can use our BIP86 for the withdrawal.
Add the UTXO_P2TR_BIP86 in preparation to add BIP86 wallet functions such as newaddr, listaddr etc. We also add a new index in the database for BIP86 as this is using a completely different derivation path and hsm_secret.
RIP to this commit there's a good chance a lot of this code doesn't even make this into the final PR. Pour one out for the fallen lines of code.
This commit is doing the rest of the derivation. There was a significant overlap between the bip32_pubkey derivation and the bip86_pubkey derivation so that has been refactored in one place.
BIP86 wants the full 64-byte BIP32 seed (from BIP39). This wires up BIP86
support so the HSM derives the hardened base m/86'/0'/0' inside the box,
and exposes helpers:
• derive_bip86_base_key() // m/86'/0'/0'
• bip86_key(index) // m/86'/0'/0'/0/index
Spoiler: derive_bip86_base_key() and bip86_key() now live in libhsmd.c as they will later be used to check the derived wallet address against hsmd's derivation, this is just to sanity check that we haven't had an accidental bit flip while we have generated this address.
Here's some *foreshadowing* for what's to come. Here's what we're aiming for with our derivation flow:
Derivation split (hardened vs unhardened)
========================================
┌───────────────┐
│ HSM │ (secrets live here)
│ │
│ BIP39 → seed (64B)
│ ↓
│ m/86'/0'/0' ← derive hardened base (private)
│ ↓ (neuter)
│ BIP86 base xpub ← public-only + chain code
│ ↓
│ [send once over wire]
└───────────────┘
│
▼
┌───────────────────────┐
│ lightningd / wallet │
│ │
│ local (unhardened) derivations:
│ /0/i → external
│ /1/i → change
│ │
│ P2TR(BIP86) from pubkey_i
│ (optionally: CHECK with HSM)
└───────────────────────┘
We want to do part of the derivation inside hsmd and then send this base "pubkey" over the wire so our wallet can do the remaining derivation based on the address type and index. This lays the foundation for the base key wire message.
hsmd: plumb length-aware secret into hsmd_init; keep 32B mirror
BIP86 (from BIP39) wants the full 64-byte BIP32 seed. This commit plumbs a variable-length (32/64B) secret into hsmd and uses the accessors from the previous commit. We keep the old 32B hsm_secret mirror and, for now, only use the first 32 bytes so legacy paths keep working.
Spoiler: HKDFs will keep using the 32B seed; only wallet address derivation
will switch to the full 64B in a follow-up.
BIP86 derivation requires the full 64-byte seed that comes from the BIP39 mnemonic. The first 32 bytes goes towards to master seed material and the nest 32 bytes go towards the chaincode, so we need the entire 64 bytes for deterministic derivations. I've kept the old secret struct in for now for backwards compatibility and also added some accessors which will eventually die in this branch's git multiverse but that's a spoiler, they're on the ride for the next few commits at least to help us migrate to this length aware API throughout the rest of the code without making a lot of breaking changes.
Update the exposesecret plugin to work with the new unified HSM secret
format that supports BIP39 mnemonics.
Changelog-Added - exposesecret now has a mnemonic field
In several places we were using tal_wally_end(tmpctx), which assumes
that libwally isn't using any of those allocations.
Make an explicit "tal_wally_discard" which asserts that there are no
outstanding libwally allocations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: hsmd: New nodes will now be created with a BIP-39 12-word phrase as their root secret.
Changelog-Deprecated: config: `encrypted-hsm` to require a passphrase (use `hsm-passphrase`).
Changelog-Added: config: `hsm-passphrase` indicates we should use a manual passphrase with the hsm secret.
Changelog-Added: `hsmtool` now supports hsm_secret files using a 12-word mnemonic.
Changelog-Removed: hsmtool support for mnemonics in non-english languages removed.
This is a newer API to replace hsm_encryption.c and hsm_encryption.c, this tidies up the API to be used and also cleans things up to support our new formats. Our hsm_secret formats now include:
- Legacy 32-byte plain format
- Legacy 73-byte encrypted format
- New mnemonic format without passphrase (32 zero bytes + mnemonic)
- New mnemonic format with passphrase (32-byte hash + mnemonic)
This commit includes support to detect the format based on the file size and content structure. The hsm will store mnemonics in the hsm_secret file as:
`passphraseHash`mnemonic`
Incorporate a time: this covers the restart case as well. And make it time_mono(),
which doesn't get overridden when we override normal wall time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You can now simply add per-tal-object helpers for memleak, but our older pattern required
calling memleak functions explicitly during memleak handling. Hash tables in particular need
to be dynamically allocated (we override the allocators using htable_set_allocator and assume
this), so it makes sense to have a helper macro that does all three.
This eliminates a huge amount of code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It now simply renames tal names, so it's harmless to do even if we're
not going to do memleak detection.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We watch if they are to do with a channel, or have outputs going to us, but otherwise
we didn't, so we never updated the blockheight in the db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `listtransactions` now correctly updates `blockheight` for txs created by `sendpsbt` which have no change outputs.
I got a NULL deref on `infcopy->remote_funding = *inflight->funding->splice_remote_funding`
at once point in testing, so this should prevent that from happening,
yet still allow us to catch it in CI if it happens again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happens if the channel is *not* announcable yet. Then we hit the assertion
in funding_depth_cb that the txid is the same as the current funding.txid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: fixed crash when we splice a channel which hasn't been announced yet.
I use .clangd to configure compiler flags for the lsp (language server)
I use in nvim.
Changelog-None
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
This is a trick from bcli: we ask bitcoind for the block, and it hands
us a 2MB hex blob (which we read in multiple parts). Our parser wades
through it all, but a quick search for '}' makes it much faster.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The efficient way to do this is to use membuf, which handles the buffer
control (only using memmove when necessary). We have multiple places
where we opencoded this, some of which did not use membuf at all.
So now we create common infrastructure. I tried making it a single
function but the various users are quite different, so instead I opted
for a toolbox approach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They were never referenced, and saving the toks is questionable since their
lifetime is not guaranteed to live beyond this call (at least the buffer was
duplicated, but that also assumed it was at the start of the object).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And add a check for new uses creeping in, since it got cut & paste
everywhere.
This means "this is a valid string, but truncate it to this many characters"
vs "%.*s" which means "only read this many characters of string":
```
['lightningd-3 2025-10-23T02:31:40.890Z **BROKEN** plugin-funder: Plugin marked as important, shutting down lightningd!']
--------------------------- Captured stderr teardown ---------------------------
#0 0x557da58ad1dc in printf_common(void*, char const*, __va_list_tag*) asan_interceptors.cpp.o
#1 0x557da5aff814 in json_out_addv /home/runner/work/lightning/lightning/ccan/ccan/json_out/json_out.c:239:11
#2 0x557da59740ce in plugin_logv /home/runner/work/lightning/lightning/plugins/libplugin.c:1777:2
#3 0x557da5969b6f in plugin_log /home/runner/work/lightning/lightning/plugins/libplugin.c:1934:2
#4 0x557da595c4f6 in datastore_del_success /home/runner/work/lightning/lightning/plugins/funder.c:161:2
#5 0x557da598b837 in handle_rpc_reply /home/runner/work/lightning/lightning/plugins/libplugin.c:1072:10
#6 0x557da598a4b0 in rpc_conn_read_response /home/runner/work/lightning/lightning/plugins/libplugin.c:1361:3
#7 0x557da5adbea5 in next_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:60:9
#8 0x557da5ae06ff in do_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:422:8
#9 0x557da5adfb58 in io_ready /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:439:10
#10 0x557da5aec2ce in io_loop /home/runner/work/lightning/lightning/ccan/ccan/io/poll.c:455:5
#11 0x557da59757ac in plugin_main /home/runner/work/lightning/lightning/plugins/libplugin.c:2409:3
#12 0x557da594fe23 in main /home/runner/work/lightning/lightning/plugins/funder.c:1723:2
#13 0x7f6572229d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#14 0x7f6572229e3f in __libc_start_main csu/../csu/libc-start.c:392:3
#15 0x557da588b584 in _start (/home/runner/work/lightning/lightning/plugins/funder+0x10d584) (BuildId: 71ba63ab577fc6fa60573d3e8555f6db7d5c584d)
0x624000009d28 is located 0 bytes to the right of 7208-byte region [0x624000008100,0x624000009d28)
allocated by thread T0 here:
#0 0x557da590e7f6 in __interceptor_realloc (/home/runner/work/lightning/lightning/plugins/funder+0x1907f6) (BuildId: 71ba63ab577fc6fa60573d3e8555f6db7d5c584d)
#1 0x557da5b2149b in tal_resize_ /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:755:13
#2 0x557da59f2032 in membuf_tal_resize /home/runner/work/lightning/lightning/common/utils.c:203:2
#3 0x557da5b03934 in membuf_prepare_space_ /home/runner/work/lightning/lightning/ccan/ccan/membuf/membuf.c:45:12
#4 0x557da59d4289 in jsonrpc_io_read_ /home/runner/work/lightning/lightning/common/jsonrpc_io.c:127:2
#5 0x557da598a635 in rpc_conn_read_response /home/runner/work/lightning/lightning/plugins/libplugin.c:1366:9
#6 0x557da5adbea5 in next_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:60:9
#7 0x557da5ae06ff in do_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:422:8
#8 0x557da5adfb58 in io_ready /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:439:10
#9 0x557da5aec2ce in io_loop /home/runner/work/lightning/lightning/ccan/ccan/io/poll.c:455:5
#10 0x557da59757ac in plugin_main /home/runner/work/lightning/lightning/plugins/libplugin.c:2409:3
#11 0x557da594fe23 in main /home/runner/work/lightning/lightning/plugins/funder.c:1723:2
#12 0x7f6572229d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
SUMMARY: AddressSanitizer: heap-buffer-overflow asan_interceptors.cpp.o in printf_common(void*, char const*, __va_list_tag*)
Shadow bytes around the buggy address:
0x0c487fff9350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c487fff93a0: 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa
0x0c487fff93b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==26122==ABORTING
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we can make sure the compile and run in normal builds.
Side note: various tests call common_setup(), which means we called it
twice in unit testing mode, so we conditionalize those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happens in the fuzzer corpora, but that doesn't check for take()
leaks. Our unit tests do:
```
fuzz-initial_channel: outstanding taken(): 0x626c3b3affc8
make: *** [Makefile:823: unittest/tests/fuzz/fuzz-initial_channel] Error 1
```
This doesn't matter in real life, since we exit the subdaemon if this
fails, but it's still a bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
