Only in developer mode, ofc.
Notes:
1. We have to move the initialization before the lightningd main trace_start,
since that uses pseudorand().
2. To make the results stable, we need to use per-caller values to randbytes().
Otherwise external timing changes the call order.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They are invalid! This is because our BOLT11_FIELD_BYTE_LIMIT is not the limit,
it's one greater than the limit.
Reported-by: https://github.com/noblepayne
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `invoice` no longer accepts 640-byte descriptions (it would produce malformed invoices).
We call it once at the end, but calling on each allocation is
excessive, and it shows when dealing with large PSBTS. Testing a
700-input PSBT was unusably slow without this: after this the entire
test ran in 9 seconds.
Changelog-Fixed: JSON-RPC: Dealing with giant PSBTs (700 inputs!) is now much faster.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No idea if it works, we don't test it and nobody runs it. I guess not.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Config: non-functioning litecoin support (who knew we even had that?)
Instead of having a separate field to derive the bip86 base key, we return it in the hsmd init reply once we know that the hsm_secret is of mnemonic type
Add TLV field to hsmd_init_reply_v4 to communicate the HSM secret type
(mnemonic vs legacy) from HSM to lightningd. This allows lightningd to
automatically determine whether to use BIP86 or BIP32 derivation without
needing separate address types.
This simplifies the UTXO type system by removing the separate BIP86
enum value. P2TR addresses will now use unified derivation logic
based on the wallet's HSM secret type rather than having separate
enum values."
This commit fixes an issue where BIP86 addresses were not being
discovered during wallet recovery/rescan operations.
The root cause was that init_txfilter() only populated the transaction
filter with BIP32-derived keys, preventing lightningd from recognizing
BIP86 UTXOs during blockchain scans. Now both BIP32 and BIP86 derived
scripts are included in the filter when BIP86 derivation is enabled.
This ensures that wallets restored from BIP39 mnemonics can properly
discover and display previously funded BIP86 addresses without requiring
manual address generation first.
[ We also move the slightly-lost comment about libbacktrace so it is
where we actually include <backtrace.h> --RR ]
This commit is updating hsmtool and exposesecrets to use the new pattern for storing the secret, which is the secret_data and secret_len, to support both 64 byte and 32 byte seeds.
In the case where we receive a taproot utxo we want to be able to tell if it was derived using a BIP32 seed or a BIP86 seed. Considering we will only be supporting BI86 type wallet addresses for mnemonics we can check if the out secret is 64 bytes long and if it is we can use our BIP86 for the withdrawal.
Add the UTXO_P2TR_BIP86 in preparation to add BIP86 wallet functions such as newaddr, listaddr etc. We also add a new index in the database for BIP86 as this is using a completely different derivation path and hsm_secret.
BIP86 derivation requires the full 64-byte seed that comes from the BIP39 mnemonic. The first 32 bytes goes towards to master seed material and the nest 32 bytes go towards the chaincode, so we need the entire 64 bytes for deterministic derivations. I've kept the old secret struct in for now for backwards compatibility and also added some accessors which will eventually die in this branch's git multiverse but that's a spoiler, they're on the ride for the next few commits at least to help us migrate to this length aware API throughout the rest of the code without making a lot of breaking changes.
Update the exposesecret plugin to work with the new unified HSM secret
format that supports BIP39 mnemonics.
Changelog-Added - exposesecret now has a mnemonic field
In several places we were using tal_wally_end(tmpctx), which assumes
that libwally isn't using any of those allocations.
Make an explicit "tal_wally_discard" which asserts that there are no
outstanding libwally allocations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: hsmd: New nodes will now be created with a BIP-39 12-word phrase as their root secret.
Changelog-Deprecated: config: `encrypted-hsm` to require a passphrase (use `hsm-passphrase`).
Changelog-Added: config: `hsm-passphrase` indicates we should use a manual passphrase with the hsm secret.
Changelog-Added: `hsmtool` now supports hsm_secret files using a 12-word mnemonic.
Changelog-Removed: hsmtool support for mnemonics in non-english languages removed.
This is a newer API to replace hsm_encryption.c and hsm_encryption.c, this tidies up the API to be used and also cleans things up to support our new formats. Our hsm_secret formats now include:
- Legacy 32-byte plain format
- Legacy 73-byte encrypted format
- New mnemonic format without passphrase (32 zero bytes + mnemonic)
- New mnemonic format with passphrase (32-byte hash + mnemonic)
This commit includes support to detect the format based on the file size and content structure. The hsm will store mnemonics in the hsm_secret file as:
`passphraseHash`mnemonic`
You can now simply add per-tal-object helpers for memleak, but our older pattern required
calling memleak functions explicitly during memleak handling. Hash tables in particular need
to be dynamically allocated (we override the allocators using htable_set_allocator and assume
this), so it makes sense to have a helper macro that does all three.
This eliminates a huge amount of code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It now simply renames tal names, so it's harmless to do even if we're
not going to do memleak detection.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a trick from bcli: we ask bitcoind for the block, and it hands
us a 2MB hex blob (which we read in multiple parts). Our parser wades
through it all, but a quick search for '}' makes it much faster.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The efficient way to do this is to use membuf, which handles the buffer
control (only using memmove when necessary). We have multiple places
where we opencoded this, some of which did not use membuf at all.
So now we create common infrastructure. I tried making it a single
function but the various users are quite different, so instead I opted
for a toolbox approach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we can make sure the compile and run in normal builds.
Side note: various tests call common_setup(), which means we called it
twice in unit testing mode, so we conditionalize those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happens in the fuzzer corpora, but that doesn't check for take()
leaks. Our unit tests do:
```
fuzz-initial_channel: outstanding taken(): 0x626c3b3affc8
make: *** [Makefile:823: unittest/tests/fuzz/fuzz-initial_channel] Error 1
```
This doesn't matter in real life, since we exit the subdaemon if this
fails, but it's still a bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Basically, `devtools/reduce-includes.sh */*.c`.
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
After:
real 0m37.872000-39.974000(39.5466+/-0.59)s
user 3m1.211000-14.968000(12.4556+/-3.9)s
sys 0m35.008000-36.830000(36.4143+/-0.5)s
Build time after touch config.vars (RUST=0):
Before:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
After:
real 0m18.384000-21.307000(20.8605+/-0.92)s
user 2m5.585000-26.843000(23.6017+/-6.7)s
sys 0m19.650000-22.003000(21.4943+/-0.69)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Each header should only include the other headers it needs to compile;
`devtools/reduce-includes.sh */*.h` does this. The C files then need
additional includes if they don't compile.
And remove the entirely useless wire/onion_wire.h, which only serves to include wire/onion_wiregen.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. $(JSMN_OBJS) is not set anywhere.
2. You don't need to depend on CCAN_HEADERS, COMMON_HEADERS or JSMN_HEADERS: the top level Makefile has all object depedning on it.
3. Similarly, CCAN_OBJS.
4. Every object file should be rebuilt if its Makefile changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to manually choose what to link against,
which is much of the complexity of our Makefiles: the compiler will
automatically use any object files it needs to link.
We already do this for ccan as libccan.a, now we have libcommon.a.
We don't link against it for *everything*, as some tests require their own
versions.
Notes:
1. I get rid of the weird plugins/test/Makefile2 (accidental commit?)
2. Many tests change due to update-mocks.
3. In some places I added the missing dependency on the Makefile itself, though most are in the next
patch.
Before:
Total program size: 221366528
Total tests size: 364243856
After:
Total program size: 190733656
Total tests size: 337880888
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.227000-44.245000(41.8222+/-1.6)s
user 3m2.105000-33.696000(23.1442+/-8.4)s
sys 0m35.054000-42.269000(39.7231+/-2)s
After:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
Build time after touch config.vars (RUST=0):
Before:
real 0m18.928000-22.776000(21.5084+/-1.1)s
user 2m8.613000-36.567000(27.7281+/-7.7)s
sys 0m20.458000-23.436000(22.3963+/-0.77)s
After:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rusty@rusty-Framework:~/devel/cvs/lightni
Changes:
* Fields renumbered to their draft values + billion.
* offer_recurrence now comes in compulsory or optional (backwards compat) flavors.
* `proportional_amount` is now inside `offer_recurrence_base` not `offer_recurrence_paywindow`.
* New field `invreq_recurrence_cancel`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Draft specification for recurring offers changed: old recurring offers will no longer work.
This should detect partial writes more robustly, since we make a
separate pwrite() call to update this flag after the record is written.
Previously we were playing a bit loose with synchronization assumptions,
which seemed to work on Linux ext4, but not so well elsewhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
