This makes a big difference for large tables. Consider 1.6M channelmoves,
which took 82 seconds to populate, now takes 17 seconds:
Before:
plugin-sql: Time to call listchannelmoves: 10.380341485 seconds
plugin-sql: Time to refresh channelmoves: 82.311287310 seconds
After:
plugin-sql: Time to call listchannelmoves: 9.962815480 seconds
plugin-sql: Time to refresh channelmoves: 15.711549299 seconds
plugin-sql: Time to refresh + create indices for channelmoves: 17.100151235 seconds
tests/test_coinmoves.py::test_generate_coinmoves (50,000):
Time (from start to end of l2 node): 27 seconds
Worst latency: 16.0 seconds
Changelog-Changed: Plugins: `sql` initial load for tables is much faster (e.g 82 to 17 seconds for very large channelmoves table).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we're synchronous, these only reach lightningd after we're done:
in the case of 1.6M channelmoves, that can give it major heartburn.
In practice, this reduces the first bkpr command on a fresh upgrade
from 349 to 235 seconds (but this was before other improvements we did
this release).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `bookkeeper` reduced logging for large imports to increase speed.
```
lightningd-1 2025-10-27T11:26:04.285Z **BROKEN** plugin-bcli: bitcoin-cli exec failed: Argument list too long
```
Use -stdin to bitcoin-cli: we can then handle arguments of arbitrary length.
Fixes: https://github.com/ElementsProject/lightning/issues/8634
Changelog-Fixed: plugins: `bcli` would fail with "Argument list too long" when sending a giant tx.
We have another report of looping. This maxparts code is being completely
rewritten, but it's good to have a catchall for any other cases which might
emerge.
I had to make it customizable since our tests under valgrind are SLOW!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. We would find a flow.
2. refine_flow would reduce it so it doesn't deliver enough.
3. So we need to find another, but we are at the limit.
4. So we remove the flow we found.
5. Goto 1.
This can be fixed by disabling a channel which we caused us to reduce the flow,
so we should always make forward progress.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `askrene` could enter an infinite loop when maxparts is restricted.
This commit is updating hsmtool and exposesecrets to use the new pattern for storing the secret, which is the secret_data and secret_len, to support both 64 byte and 32 byte seeds.
Update the exposesecret plugin to work with the new unified HSM secret
format that supports BIP39 mnemonics.
Changelog-Added - exposesecret now has a mnemonic field
You can now simply add per-tal-object helpers for memleak, but our older pattern required
calling memleak functions explicitly during memleak handling. Hash tables in particular need
to be dynamically allocated (we override the allocators using htable_set_allocator and assume
this), so it makes sense to have a helper macro that does all three.
This eliminates a huge amount of code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They were never referenced, and saving the toks is questionable since their
lifetime is not guaranteed to live beyond this call (at least the buffer was
duplicated, but that also assumed it was at the start of the object).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And add a check for new uses creeping in, since it got cut & paste
everywhere.
This means "this is a valid string, but truncate it to this many characters"
vs "%.*s" which means "only read this many characters of string":
```
['lightningd-3 2025-10-23T02:31:40.890Z **BROKEN** plugin-funder: Plugin marked as important, shutting down lightningd!']
--------------------------- Captured stderr teardown ---------------------------
#0 0x557da58ad1dc in printf_common(void*, char const*, __va_list_tag*) asan_interceptors.cpp.o
#1 0x557da5aff814 in json_out_addv /home/runner/work/lightning/lightning/ccan/ccan/json_out/json_out.c:239:11
#2 0x557da59740ce in plugin_logv /home/runner/work/lightning/lightning/plugins/libplugin.c:1777:2
#3 0x557da5969b6f in plugin_log /home/runner/work/lightning/lightning/plugins/libplugin.c:1934:2
#4 0x557da595c4f6 in datastore_del_success /home/runner/work/lightning/lightning/plugins/funder.c:161:2
#5 0x557da598b837 in handle_rpc_reply /home/runner/work/lightning/lightning/plugins/libplugin.c:1072:10
#6 0x557da598a4b0 in rpc_conn_read_response /home/runner/work/lightning/lightning/plugins/libplugin.c:1361:3
#7 0x557da5adbea5 in next_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:60:9
#8 0x557da5ae06ff in do_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:422:8
#9 0x557da5adfb58 in io_ready /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:439:10
#10 0x557da5aec2ce in io_loop /home/runner/work/lightning/lightning/ccan/ccan/io/poll.c:455:5
#11 0x557da59757ac in plugin_main /home/runner/work/lightning/lightning/plugins/libplugin.c:2409:3
#12 0x557da594fe23 in main /home/runner/work/lightning/lightning/plugins/funder.c:1723:2
#13 0x7f6572229d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#14 0x7f6572229e3f in __libc_start_main csu/../csu/libc-start.c:392:3
#15 0x557da588b584 in _start (/home/runner/work/lightning/lightning/plugins/funder+0x10d584) (BuildId: 71ba63ab577fc6fa60573d3e8555f6db7d5c584d)
0x624000009d28 is located 0 bytes to the right of 7208-byte region [0x624000008100,0x624000009d28)
allocated by thread T0 here:
#0 0x557da590e7f6 in __interceptor_realloc (/home/runner/work/lightning/lightning/plugins/funder+0x1907f6) (BuildId: 71ba63ab577fc6fa60573d3e8555f6db7d5c584d)
#1 0x557da5b2149b in tal_resize_ /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:755:13
#2 0x557da59f2032 in membuf_tal_resize /home/runner/work/lightning/lightning/common/utils.c:203:2
#3 0x557da5b03934 in membuf_prepare_space_ /home/runner/work/lightning/lightning/ccan/ccan/membuf/membuf.c:45:12
#4 0x557da59d4289 in jsonrpc_io_read_ /home/runner/work/lightning/lightning/common/jsonrpc_io.c:127:2
#5 0x557da598a635 in rpc_conn_read_response /home/runner/work/lightning/lightning/plugins/libplugin.c:1366:9
#6 0x557da5adbea5 in next_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:60:9
#7 0x557da5ae06ff in do_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:422:8
#8 0x557da5adfb58 in io_ready /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:439:10
#9 0x557da5aec2ce in io_loop /home/runner/work/lightning/lightning/ccan/ccan/io/poll.c:455:5
#10 0x557da59757ac in plugin_main /home/runner/work/lightning/lightning/plugins/libplugin.c:2409:3
#11 0x557da594fe23 in main /home/runner/work/lightning/lightning/plugins/funder.c:1723:2
#12 0x7f6572229d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
SUMMARY: AddressSanitizer: heap-buffer-overflow asan_interceptors.cpp.o in printf_common(void*, char const*, __va_list_tag*)
Shadow bytes around the buggy address:
0x0c487fff9350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c487fff93a0: 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa
0x0c487fff93b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==26122==ABORTING
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Basically, `devtools/reduce-includes.sh */*.c`.
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
After:
real 0m37.872000-39.974000(39.5466+/-0.59)s
user 3m1.211000-14.968000(12.4556+/-3.9)s
sys 0m35.008000-36.830000(36.4143+/-0.5)s
Build time after touch config.vars (RUST=0):
Before:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
After:
real 0m18.384000-21.307000(20.8605+/-0.92)s
user 2m5.585000-26.843000(23.6017+/-6.7)s
sys 0m19.650000-22.003000(21.4943+/-0.69)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Each header should only include the other headers it needs to compile;
`devtools/reduce-includes.sh */*.h` does this. The C files then need
additional includes if they don't compile.
And remove the entirely useless wire/onion_wire.h, which only serves to include wire/onion_wiregen.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. $(JSMN_OBJS) is not set anywhere.
2. You don't need to depend on CCAN_HEADERS, COMMON_HEADERS or JSMN_HEADERS: the top level Makefile has all object depedning on it.
3. Similarly, CCAN_OBJS.
4. Every object file should be rebuilt if its Makefile changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to manually choose what to link against,
which is much of the complexity of our Makefiles: the compiler will
automatically use any object files it needs to link.
We already do this for ccan as libccan.a, now we have libcommon.a.
We don't link against it for *everything*, as some tests require their own
versions.
Notes:
1. I get rid of the weird plugins/test/Makefile2 (accidental commit?)
2. Many tests change due to update-mocks.
3. In some places I added the missing dependency on the Makefile itself, though most are in the next
patch.
Before:
Total program size: 221366528
Total tests size: 364243856
After:
Total program size: 190733656
Total tests size: 337880888
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.227000-44.245000(41.8222+/-1.6)s
user 3m2.105000-33.696000(23.1442+/-8.4)s
sys 0m35.054000-42.269000(39.7231+/-2)s
After:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
Build time after touch config.vars (RUST=0):
Before:
real 0m18.928000-22.776000(21.5084+/-1.1)s
user 2m8.613000-36.567000(27.7281+/-7.7)s
sys 0m20.458000-23.436000(22.3963+/-0.77)s
After:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rusty@rusty-Framework:~/devel/cvs/lightni
If the fees are not *all* of the fees (as we do in next patch), the
query would be wrong. Plus, as the FIXME suggests, we should just save
it as we're getting the fee_sums, not do a whole new query!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we read all of them, we might get 1.6M at once (after initial
migration). Then we submit a few hundred thousand simultaneous
requests to lightningd, and it gets upset, queueing them all on the
xpay command hook and running out of memory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: plugins: bookkeeper first invocation after migration from prior to 25.09 with very large databases will not crash.
This significantly speeds up the query which bookkeeper often does:
"SELECT created_index"
" FROM channelmoves"
" WHERE payment_hash = X'%s'"
" AND credit_msat = %"PRIu64
" AND created_index <= %"PRIu64,
On large databases this scan is expensive, and a payment_hash index
cuts it down a great deal. It does take longer to load the channelmoves
in the first place though (about 3x).
Before:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
169505 /tmp/bkpr-progress
196010 /tmp/bkpr-progress
219370 /tmp/bkpr-progress
235671 /tmp/bkpr-progress
244242 /tmp/bkpr-progress
255362 /tmp/bkpr-progress
265636 /tmp/bkpr-progress
276966 /tmp/bkpr-progress
284451 /tmp/bkpr-progress
288836 /tmp/bkpr-progress
296578 /tmp/bkpr-progress
304571 /tmp/bkpr-progress
After:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
161421 /tmp/bkpr-progress
238273 /tmp/bkpr-progress
281185 /tmp/bkpr-progress
305787 /tmp/bkpr-progress
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: plugins: the sql plugin now keeps an index on `channelmoves` by `payment_hash`.
Note that we create a destructor on the command to reset request->cmd
pointer if the cmd is freed (so we know not to call the callback).
But attaching hundreds of thousands of them is slow: it's a
single-linked list, which is iterated in several places.
But that's redundant: the request is now allocated off the cmd, so freeing the command
will free the request anyway.
Hacking in something to print progress to a file, here's the number of
requests processed every 10 seconds before and after:
Before:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
181529 /tmp/bkpr-progress
195994 /tmp/bkpr-progress
207083 /tmp/bkpr-progress
226336 /tmp/bkpr-progress
234319 /tmp/bkpr-progress
241514 /tmp/bkpr-progress
247421 /tmp/bkpr-progress
255292 /tmp/bkpr-progress
261367 /tmp/bkpr-progress
269085 /tmp/bkpr-progress
276953 /tmp/bkpr-progress
282233 /tmp/bkpr-progress
286193 /tmp/bkpr-progress
290930 /tmp/bkpr-progress
295276 /tmp/bkpr-progress
301086 /tmp/bkpr-progress
After:
169505 /tmp/bkpr-progress
196010 /tmp/bkpr-progress
219370 /tmp/bkpr-progress
235671 /tmp/bkpr-progress
244242 /tmp/bkpr-progress
255362 /tmp/bkpr-progress
265636 /tmp/bkpr-progress
276966 /tmp/bkpr-progress
284451 /tmp/bkpr-progress
288836 /tmp/bkpr-progress
296578 /tmp/bkpr-progress
304571 /tmp/bkpr-progress
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Deprecated in 24.11, disabled in 25.05.
Changelog-Removed: JSON-RPC: `decode` field `blinding` (use `first_path_key` as per modern BOLT naming)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In this case, we make an immediately-expiring invoice. This correctly blocks
any successive requests for invoices, as per the spec requirement.
This means we have to handle invoice_requests without reply_path, amounts
or quantity *if* they specify invreq_recurrence_cancel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`fetchinvoice` variant, for setting invreq_recurrence_cancel instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `cancelrecurringinvoice` command to send new "don't expect any more invoice requests" msg to recurring bolt12 invoices.
We added this to the recurrence spec: the offer expiration only applies to the
first request, not subsequent ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you're doing just-in-time stock management, it would be annoying to
discover this wasn't allowed!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changes:
* Fields renumbered to their draft values + billion.
* offer_recurrence now comes in compulsory or optional (backwards compat) flavors.
* `proportional_amount` is now inside `offer_recurrence_base` not `offer_recurrence_paywindow`.
* New field `invreq_recurrence_cancel`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Draft specification for recurring offers changed: old recurring offers will no longer work.
Offer_absolute_expiry should be used if you want to require starting at the start.
Changelog-EXPERIMENTAL: Protocol: BOLT 12 recurrence `start_any_period` removed, use expiry if you need to restrict when they can start using the offer.
all these changelogs only apply to the Docker image.
Changelog-Added: added verification of GPG keys for the bitcoin and litecoin tarballs.
Changelog-Fixed: fixed compilation on all target architectures; each had their own bugs (poetry, missing packages...).
Changelog-Fixed: fixed cargo cross compilation. it was mistakenly using QEMU before.
Changelog-Fixed: fixed CPU compatibility bug described in issue 8456
Changelog-Changed: improve build time by 8.8x
Changelog-Changed: improve image size by 2.07x
more detailed changelog can be found on the PR: https://github.com/ElementsProject/lightning/pull/8429
