Commit Graph

17550 Commits

Author SHA1 Message Date
Rusty Russell
acb8a8cc15 gossipd: dev-compact-gossip-store to manually invoke compaction.
And tests!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
88f3f97b7c gossipd: reset dying_channels array after compact.
Reported-by: @daywalker90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
912b40aeff gossipd: compact when gossip store is 80% deleted records.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `gossipd` now uses a `lightning_gossip_compactd` helper to compact the gossip_store on demand, keeping it under about 210MB.
2026-02-16 17:23:33 +10:30
Rusty Russell
15696d97bd gossipd: code to invoke compactd and reopen store.
This isn't called anywhere yet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
f56f8adcdf gossipd: lightningd/lightning_gossip_compactd
A new subprocess run by gossipd to create a compacted gossip store.

It's pretty simple: a linear compaction of the file.  Once it's done the amount it
was told to, then gossipd waits until it completes the last bit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
a966dd71ad common: expose gossip_store "header and type" single-read struct.
gossip_store.c uses this to avoid two reads, and we want to use it
elsewhere too.

Also fix old comment on gossip_store_readhdr().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
1fb4da075f gossipd: put the last_writes array inside struct gossip_store.
This is the file responsible for all the writing, so it should be
responsible for the rewriting if necessary (rather than
gossmap_manage).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
facf24b6ee devtools/gossmap-compress: create latest gossip_store version
This saves gossipd from converting it:

```
lightningd-1 2026-02-02T00:50:49.505Z DEBUG   gossipd: Time to convert version 14 store: 890 msec
```

Reducing node startup time from 1.4 seconds to 0.5 seconds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
445bcd040a gossipd: don't compact on startup.
We now only need to walk it if we're doing an upgrade.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: `gossipd` no longer compacts gossip_store on startup (improving start times significantly).
2026-02-16 17:23:33 +10:30
Rusty Russell
dfc4ce21de gossipd: don't gather dying channels during compaction.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
900fd08455 gossipd: use gossmap to load the dying entries.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
7d70e8baf2 gossmap: keep stats on live/deleted records.
This way gossmap_manage can decide when to compact.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
1ad8ca9603 gossmap: add callback for gossipd to see dying messages.
gossmap doesn't care, so gossipd currently has to iterate through the
store to find them at startup.  Create a callback for gossipd to use
instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
e8fd235d4e common: move gossip_store_wire.csv into common/ from gossipd/
It's used by common/gossip_store.c, which is used by many things other than
gossipd.  This file belongs in common.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
5dcf39867c gossipd: write uuid record on startup.
This is the first record, and ignored by everything else.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
25131d2ea2 common/gossmap: use the UUID record on reopen.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
b1055aa0ac gossip_store: add UUID entry at front of the store.
We also put this in the store_ended message, too: so you can
tell if the equivalent_offset there really refers to this new
entry (or if two or more rewrites have happened).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
d4c62f8c4c tools: delete gossip_store of needed for downgrade even if db hasn't changed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
5e1bbb08c7 gossmap: reduce load times by 20%
It's actually quite quick to load a cache-hot 308,874,377 byte
gossip_store (normal -Og build), but perf does show time spent
in siphash(), which is a bit overkill here, so drop that:

Before:
	Time to load: 66718983-78037766(7.00553e+07+/-2.8e+06)nsec
	
After: 
	Time to load: 54510433-57991725(5.61457e+07+/-1e+06)nsec

We could save maybe 10% more by disabling checksums, but having
that assurance is nice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
120c9d8ce4 devtools/gossmap-compress: generate better scids.
Our poor scid generation clashes badly with simplified hashing (the
next patch) leading to l1's startup time when using a generated map
moving from 4 seconds to 14 seconds.  Under CI it actually timed out
several tests.

Fixing our fake scids to be more "random" reduces it to 1.5 seconds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
ae957161e6 pytest: fix bogus test_gossip_store_compact_noappend test.
It didn't do anything, since the dev_compact_gossip_store command was
removed.  When we make it do something, it crashes since old_len is 0:

```
gossipd: gossip_store_compact: bad version
gossipd: FATAL SIGNAL 6 (version v25.12rc3-1-g9e6c715-modded)
...
gossipd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x7119bd8288fe
gossipd: backtrace: ./assert/assert.c:96 (__assert_fail_base) 0x7119bd82881a
gossipd: backtrace: ./assert/assert.c:105 (__assert_fail) 0x7119bd83b516
gossipd: backtrace: gossipd/gossip_store.c:52 (append_msg) 0x56294de240eb
gossipd: backtrace: gossipd/gossip_store.c:358 (gossip_store_compact) 0x56294
gossipd: backtrace: gossipd/gossip_store.c:395 (gossip_store_new) 0x56294de24
gossipd: backtrace: gossipd/gossmap_manage.c:455 (setup_gossmap) 0x56294de255
gossipd: backtrace: gossipd/gossmap_manage.c:488 (gossmap_manage_new) 0x56294
gossipd: backtrace: gossipd/gossipd.c:400 (gossip_init) 0x56294de22de9
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
e5e5998cd8 devtools: enhance dump-gossipstore to show some details of messages.
Not a complete decode, just the highlights (what channel was announced
or updated, what node was announced).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-16 17:23:33 +10:30
Rusty Russell
09781bd381 lightningd: don't assume peer existrs in peer_connected_serialize.
It's always true for the first hook invocation, but if there is more
than one plugin, it could vanish between the two!  In the default configuration, this can't happen.

This bug has been around since v23.02.

Note: we always tell all the plugins about the peer, even if it's
already gone.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightningd: possible crash when peers disconnected if there was more than one plugin servicing the `peer_connected` hook.
Reported-by: https://github.com/santyr
Fixes: https://github.com/ElementsProject/lightning/issues/8858
2026-02-12 09:08:10 +10:30
Rusty Russell
eaf6fabf04 pytest: reproduce crash when node disconnects between hooks:
```
lightningd-2 2026-02-09T00:41:35.196Z TRACE   lightningd: Plugin peer_connected_logger_a.py returned from peer_connected hook call
lightningd-2 2026-02-09T00:41:35.196Z TRACE   lightningd: Calling peer_connected hook of plugin peer_connected_logger_b.py
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: FATAL SIGNAL 11 (version v25.12-257-g2a5fbd1-modded)
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (send_backtrace) 0x5b2abd7f29bd
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: common/daemon.c:83 (crashdump) 0x5b2abd7f2a0c
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x75950d84532f
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: lightningd/peer_control.c:1333 (peer_connected_serialize) 0x5b2abd79c964
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:359 (plugin_hook_call_next) 0x5b2abd7ae14a
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:299 (plugin_hook_callback) 0x5b2abd7ae38f
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:701 (plugin_response_handle) 0x5b2abd7a7e28
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:790 (plugin_read_json) 0x5b2abd7ace9c
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x5b2abd81dada
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x5b2abd81def6
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x5b2abd81dfb3
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:470 (io_loop) 0x5b2abd81f0db
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x5b2abd77c13b
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1495 (main) 0x5b2abd781c6a
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x75950d82a1c9
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:360 (__libc_start_main_impl) 0x75950d82a28a
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x5b2abd752964
lightningd-2 2026-02-09T00:41:35.293Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-12 09:08:10 +10:30
Rusty Russell
1b1274df7e CI: reduce parallelism for pytest.
In November 2022 we seemed to increase parallelism from 2 and 3 to 10!
That is a huge load for these CI boxes, and does explain some of our
flakes.

We only run in parallel because some tests sleep, but it's diminishing
returns (GH runners have 4 VCPUs, 16GB RAM).

This reduces it so:
- Normal runs are -n 4
- Valgrind runs are -n 2
- Sanitizer runs are -n 3

If I use my beefy build box (64BG RAM) but reduce it to 4 CPUs:

Time for pytest -n 5:
Time for pytest -n 4:
Time for pytest -n 3:

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-09 19:46:22 +10:30
Rusty Russell
939aec3b61 pytest: make hold_timeout.py test plugin release on a prompt, not timeout.
Avoids guessing what the timeout should be, use a file trigger.  This
is more optimal, and should reduce a flake in test_sql under valgrind.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-09 14:48:15 +10:30
Rusty Russell
1ad72fdafd pytest: fix test_xpay flake.
```
>       assert len(layers['layers']) == 1
E       AssertionError: assert 2 == 1
E        +  where 2 = len([{'layer': 'xpay', 'persistent': True, 'disabled_nodes': [], 'created_channels': [], 'channel_updates': [], 'constraints': [{'short_channel_id_dir': '45210x2134x44171/0', 'timestamp': 1770341134, 'minimum_msat': 289153519}, {'short_channel_id_dir': '1895x7x1895/1', 'timestamp': 1770341134, 'minimum_msat': 289007015}, {'short_channel_id_dir': '1906x1039x1906/1', 'timestamp': 1770341134, 'minimum_msat': 289008304}, {'short_channel_id_dir': '10070x60x10063/1', 'timestamp': 1770341134, 'minimum_msat': 289005726}, {'short_channel_id_dir': '18772x60x18743/0', 'timestamp': 1770341134, 'minimum_msat': 289005726}, {'short_channel_id_dir': '18623x208x18594/0', 'timestamp': 1770341134, 'minimum_msat': 289004859}, {'short_channel_id_dir': '33935x826x33727/1', 'timestamp': 1770341134, 'maximum_msat': 491501488}], 'biases': [], 'node_biases': []}, {'layer': 'xpay-94', 'persistent': False, 'disabled_nodes': [], 'created_channels': [], 'channel_updates': [], 'constraints': [], 'biases': [], 'node_biases': []}])
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-09 14:48:15 +10:30
Rusty Russell
2a5fbd1730 pytest: speed up test_sql significantly.
It uses the hold_invoice plugin to ensure that an HTLC is in flight, but
it tells it to hold the HTLC for "TIMEOUT * 2" which is a big number under CI.

Reduce it to sqrt(TIMEOUT + 1) * 2, which works for local testing (I run
with TIMEOUT=10) and still should be enough for CI (TIMEOUT=180).

Christian reported that the test took 763.00s (!!) under CI.

On my build machine (TIMEOUT=90):

Before:
	383.00s

After:
	64.38s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-07 07:29:28 +10:30
Rusty Russell
96adac48ab pytest: update "real gossip map" tests to a recent snapshot.
We delete the test_xpay_maxfee test which required the specific
topology.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-05 08:36:28 +10:30
Rusty Russell
95b63e4738 Makefile: add "canned-gossmap" target
Taked /tmp/gossip_store and created canned gossmap for testing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-05 08:36:28 +10:30
Rusty Russell
c90c21301f pytest: speed up channeld_fakenet tests.
Reduce randome delay from 0.1 - 1 seconds, to 0.01 to 0.1 seconds.

Running tests/test_xpay.py::test_xpay_fake_channeld[False]

Before:
	348.41s
After:
	76.76s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-05 08:36:28 +10:30
Rusty Russell
23202b7e85 pytest: fix channeld_fakenet divide by zero bug.
1. If max was 0, we crashed with SIGFPE due to % 0.
2. If min was non-zero, logic was incorrect (but all callers had min == 0).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-05 08:36:28 +10:30
Rusty Russell
13545124eb pytest: fix flake in test_channel_lease_unilat_closes
```
2026-01-30T05:55:13.6654636Z         # Note that l3 has the whole lease delay (minus blocks already mined)
2026-01-30T05:55:13.6655396Z         _, _, l3blocks = l3.wait_for_onchaind_tx('OUR_DELAYED_RETURN_TO_WALLET',
2026-01-30T05:55:13.6656086Z                                                  'OUR_UNILATERAL/DELAYED_OUTPUT_TO_US')
2026-01-30T05:55:13.6656618Z >       assert l3blocks == 4032 - 6 - 2 - 1
2026-01-30T05:55:13.6657033Z E       assert 4025 == (((4032 - 6) - 2) - 1)
```

Turns out that 4342043382 (tests: de-flake test that was failing on
cltv expiry) added a line to mine two more blocks, but the hardcoded
110 was not changed to 112, so we weren't actually waiting correctly.

Remove hardcoded numbers in favor of calculation, and do the same in
test_channel_lease_post_expiry (which was correct, for now).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
63497b3180 pytest: fix flake in test_even_sendcustommsg
We need to make sure the message is fully processed before
removing the plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
db2a560b9c lightningd: fix spurious memleak in peer_connected_serialize.
Of course stream will be freed soon, too, but if run at the exact
right time, memleak will get upset: use tmpctx.

```
lightningd: MEMLEAK: 0x5616f720bc48
lightningd:   label=common/wireaddr.c:255:char[]
lightningd:   alloc:
lightningd:     /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:488 (tal_alloc_)
lightningd:     /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:517 (tal_alloc_arr_)
lightningd:     /home/runner/work/lightning/lightning/ccan/ccan/tal/str/str.c:81 (tal_vfmt_)
lightningd:     /home/runner/work/lightning/lightning/ccan/ccan/tal/str/str.c:37 (tal_fmt_)
lightningd:     /home/runner/work/lightning/lightning/common/wireaddr.c:255 (fmt_wireaddr_without_port)
lightningd:     /home/runner/work/lightning/lightning/common/wireaddr.c:276 (fmt_wireaddr)
lightningd:     /home/runner/work/lightning/lightning/common/wireaddr.c:232 (fmt_wireaddr_internal)
lightningd:     /home/runner/work/lightning/lightning/lightningd/peer_control.c:1327 (peer_connected_serialize)
lightningd:     /home/runner/work/lightning/lightning/lightningd/plugin_hook.c:359 (plugin_hook_call_next)
lightningd:     /home/runner/work/lightning/lightning/lightningd/plugin_hook.c:395 (plugin_hook_call_)
lightningd:     /home/runner/work/lightning/lightning/lightningd/peer_control.c:1753 (plugin_hook_call_peer_connected)
lightningd:     /home/runner/work/lightning/lightning/lightningd/peer_control.c:1885 (handle_peer_connected)
lightningd:     /home/runner/work/lightning/lightning/lightningd/connect_control.c:563 (connectd_msg)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
9e8e85de99 pytest: allow pushed after onchain_fee in test_bookkeeping_missed_chans_pushed
It can happen, and it's perfectly reasonable.  If this happens in other places, we might need to allow
arbitrary reordering?

```
2026-01-29T05:55:58.5474967Z         exp_events = [{'tag': 'channel_open', 'credit_msat': open_amt * 1000, 'debit_msat': 0},
2026-01-29T05:55:58.5475765Z                       {'tag': 'pushed', 'credit_msat': 0, 'debit_msat': push_amt},
2026-01-29T05:55:58.5476454Z                       {'tag': 'onchain_fee', 'credit_msat': 4927000, 'debit_msat': 0},
2026-01-29T05:55:58.5477168Z                       {'tag': 'invoice', 'credit_msat': 0, 'debit_msat': invoice_msat}]
2026-01-29T05:55:58.5477797Z >       check_events(l1, channel_id, exp_events)
2026-01-29T05:55:58.5478120Z 
2026-01-29T05:55:58.5478282Z tests/test_bookkeeper.py:402: 
2026-01-29T05:55:58.5478777Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2026-01-29T05:55:58.5479162Z 
2026-01-29T05:55:58.5479396Z node = <fixtures.LightningNode object at 0x7fa3660a6140>
2026-01-29T05:55:58.5480158Z channel_id = 'a4e913b2d143efc3d90cfa66a56aeed3eb9e1533b350c8e84124bdec37bcf74a'
2026-01-29T05:55:58.5481929Z exp_events = [{'credit_msat': 10000000000, 'debit_msat': 0, 'tag': 'channel_open'}, {'credit_msat': 0, 'debit_msat': 1000000000, 'tag': 'pushed'}, {'credit_msat': 4927000, 'debit_msat': 0, 'tag': 'onchain_fee'}, {'credit_msat': 0, 'debit_msat': 11000000, 'tag': 'invoice'}]
2026-01-29T05:55:58.5483442Z 
2026-01-29T05:55:58.5483671Z     def check_events(node, channel_id, exp_events):
2026-01-29T05:55:58.5484551Z         chan_events = [ev for ev in node.rpc.bkpr_listaccountevents()['events'] if ev['account'] == channel_id]
2026-01-29T05:55:58.5485684Z         stripped = [{k: d[k] for k in ('tag', 'credit_msat', 'debit_msat') if k in d} for d in chan_events]
2026-01-29T05:55:58.5486455Z >       assert stripped == exp_events
2026-01-29T05:55:58.5489277Z E       AssertionError: assert [{'tag': 'channel_open', 'credit_msat': 10000000000, 'debit_msat': 0}, {'tag': 'onchain_fee', 'credit_msat': 4927000, 'debit_msat': 0}, {'tag': 'pushed', 'credit_msat': 0, 'debit_msat': 1000000000}, {'tag': 'invoice', 'credit_msat': 0, 'debit_msat': 11000000}] == [{'tag': 'channel_open', 'credit_msat': 10000000000, 'debit_msat': 0}, {'tag': 'pushed', 'credit_msat': 0, 'debit_msat': 1000000000}, {'tag': 'onchain_fee', 'credit_msat': 4927000, 'debit_msat': 0}, {'tag': 'invoice', 'credit_msat': 0, 'debit_msat': 11000000}]
2026-01-29T05:55:58.5492021Z E         
2026-01-29T05:55:58.5492767Z E         At index 1 diff: {'tag': 'onchain_fee', 'credit_msat': 4927000, 'debit_msat': 0} != {'tag': 'pushed', 'credit_msat': 0, 'debit_msat': 1000000000}
2026-01-29T05:55:58.5493812Z E         
2026-01-29T05:55:58.5494078Z E         Full diff:
2026-01-29T05:55:58.5494373Z E           [
2026-01-29T05:55:58.5494863Z E               {
2026-01-29T05:55:58.5495166Z E                   'credit_msat': 10000000000,
2026-01-29T05:55:58.5495565Z E                   'debit_msat': 0,
2026-01-29T05:55:58.5495946Z E                   'tag': 'channel_open',
2026-01-29T05:55:58.5496330Z E         -     },
2026-01-29T05:55:58.5496613Z E         -     {
2026-01-29T05:55:58.5496906Z E         -         'credit_msat': 0,
2026-01-29T05:55:58.5497285Z E         -         'debit_msat': 1000000000,
2026-01-29T05:55:58.5497900Z E         -         'tag': 'pushed',
2026-01-29T05:55:58.5498264Z E               },
2026-01-29T05:55:58.5498531Z E               {
2026-01-29T05:55:58.5498818Z E                   'credit_msat': 4927000,
2026-01-29T05:55:58.5499200Z E                   'debit_msat': 0,
2026-01-29T05:55:58.5499563Z E                   'tag': 'onchain_fee',
2026-01-29T05:55:58.5499925Z E               },
2026-01-29T05:55:58.5500190Z E               {
2026-01-29T05:55:58.5500477Z E                   'credit_msat': 0,
2026-01-29T05:55:58.5500863Z E         +         'debit_msat': 1000000000,
2026-01-29T05:55:58.5501255Z E         +         'tag': 'pushed',
2026-01-29T05:55:58.5501592Z E         +     },
2026-01-29T05:55:58.5501853Z E         +     {
2026-01-29T05:55:58.5502141Z E         +         'credit_msat': 0,
2026-01-29T05:55:58.5502511Z E                   'debit_msat': 11000000,
2026-01-29T05:55:58.5502889Z E                   'tag': 'invoice',
2026-01-29T05:55:58.5503424Z E               },
2026-01-29T05:55:58.5503698Z E           ]
2026-01-29T05:55:58.5503861Z 
2026-01-29T05:55:58.5504027Z tests/test_bookkeeper.py:29: AssertionError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
18a416be0d connectd: unify IO logging calls.
Normally, connectd forwards messages and then the subds do logging,
but it logs manually for msgs which are handled internally.

Clarify this logic in one place for all callers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
e36564f6dc pytest: make test_connect_ratelimit more robust
We were sending SIGSTOP to the lightningds, but that doesn't always
mean the connectd would stop it seems:

```
lightningd-1 2026-01-27T04:49:16.979Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: Initializing important peer with 1 addresses
lightningd-1 2026-01-27T04:49:16.979Z DEBUG   connectd: Got 10 bad cupdates, ignoring them (expected on mainnet)
lightningd-1 2026-01-27T04:49:16.979Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: Connected out, starting crypto
lightningd-1 2026-01-27T04:49:16.980Z DEBUG   038194b5f32bdf0aa59812c86c4ef7ad2f294104fa027d1ace9b469bb6f88cf37b-hsmd: Got WIRE_HSMD_ECDH_REQ
lightningd-1 2026-01-27T04:49:16.981Z DEBUG   hsmd: Client: Received message 1 from client
lightningd-1 2026-01-27T04:49:16.985Z TRACE   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-gossipd: handle_recv_gossip: WIRE_CHANNEL_ANNOUNCEMENT
lightningd-1 2026-01-27T04:49:16.985Z TRACE   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-gossipd: handle_recv_gossip: WIRE_CHANNEL_UPDATE
lightningd-1 2026-01-27T04:49:16.985Z TRACE   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-gossipd: handle_recv_gossip: WIRE_CHANNEL_UPDATE
lightningd-1 2026-01-27T04:49:16.985Z TRACE   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-gossipd: handle_recv_gossip: WIRE_NODE_ANNOUNCEMENT
lightningd-1 2026-01-27T04:49:16.985Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: Connect OUT
lightningd-1 2026-01-27T04:49:16.986Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: peer_out WIRE_INIT
lightningd-1 2026-01-27T04:49:16.986Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: peer_in WIRE_INIT
lightningd-1 2026-01-27T04:49:16.986Z TRACE   lightningd: Calling peer_connected hook of plugin chanbackup
lightningd-1 2026-01-27T04:49:16.986Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: Handed peer, entering loop
lightningd-1 2026-01-27T04:49:16.986Z DEBUG   03cecbfdc68544cc596223b68ce0710c9e5d2c9cb317ee07822d95079acc703d31-connectd: Initializing important peer with 1 addresses
lightningd-1 2026-01-27T04:49:16.986Z DEBUG   033845802d25b4e074ccfd7cd8b339a41dc75bf9978a034800444b51d42b07799a-connectd: Initializing important peer with 1 addresses
lightningd-1 2026-01-27T04:49:16.987Z DEBUG   033845802d25b4e074ccfd7cd8b339a41dc75bf9978a034800444b51d42b07799a-connectd: Too many connections, waiting...
lightningd-1 2026-01-27T04:49:16.987Z DEBUG   02186115cb7e93e2cb4d9d9fe7a9cf5ff7a5784bfdda4f164ff041655e4bcd4fd0-connectd: Initializing important peer with 1 addresses
lightningd-1 2026-01-27T04:49:16.987Z DEBUG   02186115cb7e93e2cb4d9d9fe7a9cf5ff7a5784bfdda4f164ff041655e4bcd4fd0-connectd: Too many connections, waiting...
lightningd-1 2026-01-27T04:49:16.987Z DEBUG   02287bfac8b99b35477ebe9334eede1e32b189e24644eb701c079614712331cec0-connectd: Initializing important peer with 1 addresses
lightningd-1 2026-01-27T04:49:16.987Z DEBUG   02287bfac8b99b35477ebe9334eede1e32b189e24644eb701c079614712331cec0-connectd: Too many connections, waiting...
lightningd-1 2026-01-27T04:49:16.987Z DEBUG   03cecbfdc68544cc596223b68ce0710c9e5d2c9cb317ee07822d95079acc703d31-connectd: Connected out, starting crypto
lightningd-1 2026-01-27T04:49:16.989Z DEBUG   038194b5f32bdf0aa59812c86c4ef7ad2f294104fa027d1ace9b469bb6f88cf37b-hsmd: Got WIRE_HSMD_ECDH_REQ
lightningd-1 2026-01-27T04:49:16.989Z DEBUG   hsmd: Client: Received message 1 from client
lightningd-1 2026-01-27T04:49:16.990Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: peer_in WIRE_GOSSIP_TIMESTAMP_FILTER
lightningd-1 2026-01-27T04:49:16.991Z DEBUG   03cecbfdc68544cc596223b68ce0710c9e5d2c9cb317ee07822d95079acc703d31-connectd: Connect OUT
lightningd-1 2026-01-27T04:49:16.991Z DEBUG   03cecbfdc68544cc596223b68ce0710c9e5d2c9cb317ee07822d95079acc703d31-connectd: peer_out WIRE_INIT
lightningd-1 2026-01-27T04:49:16.991Z DEBUG   0258f3ff3e0853ccc09f6fe89823056d7c0c55c95fab97674df5e1ad97a72f6265-connectd: peer_out WIRE_PEER_STORAGE
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
cf3583ff83 pytest: mark renepay's self-pay test flaky
Not sure this is worth getting to the bottom of, if we're going for xpay?

```
2026-01-20T05:25:55.4375717Z lightningd-1 2026-01-20T05:22:41.062Z DEBUG   plugin-cln-renepay: sendpay_failure notification: {"sendpay_failure":{"code":202,"message":"Malformed error reply","data":{"created_index":2,"id":2,"payment_hash":"8447592dc3786c3181746bd7bf17fe9e84d3de3dfff847f9f2b8c1a65c2d3ff6","groupid":1,"partid":1,"destination":"038194b5f32bdf0aa59812c86c4ef7ad2f294104fa027d1ace9b469bb6f88cf37b","amount_msat":10000,"amount_sent_msat":10000,"created_at":1768886560,"status":"pending","bolt11":"lnbcrt100n1p5k7yfqsp5g072rrl6jn7wu8nlxk279zpkacz34hr22n089j4elguccfxmg0gqpp5s3r4jtwr0pkrrqt5d0tm79l7n6zd8h3alluy070jhrq6vhpd8lmqdqgw3jhxapjxqyjw5qcqp99qxpqysgq6dd74sj0vpyjy9he8pep73tt8pljhv6mc74y28rr8995yjyahgh9kc2jqwqz62h6u6jpqz0u9x6gcahdw3pe2x8r5eyppp53mlpru8sp4p4nf6","onionreply":"c5a02ce5372635959d87818f723767efe3137bbc40568fb04e1f2eedee0a6149000e400f00000000000027100000006c00f20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}}}
2026-01-20T05:25:55.4386158Z lightningd-1 2026-01-20T05:22:41.067Z **BROKEN** plugin-cln-renepay: Unable to parse sendpay_failure: {"code":202,"message":"Malformed error reply","data":{"created_index":2,"id":2,"payment_hash":"8447592dc3786c3181746bd7bf17fe9e84d3de3dfff847f9f2b8c1a65c2d3ff6","groupid":1,"partid":1,"destination":"038194b5f32bdf0aa59812c86c4ef7ad2f294104fa027d1ace9b469bb6f88cf37b","amount_msat":10000,"amount_sent_msat":10000,"created_at":1768886560,"status":"pending","bolt11":"lnbcrt100n1p5k7yfqsp5g072rrl6jn7wu8nlxk279zpkacz34hr22n089j4elguccfxmg0gqpp5s3r4jtwr0pkrrqt5d0tm79l7n6zd8h3alluy070jhrq6vhpd8lmqdqgw3jhxapjxqyjw5qcqp99qxpqysgq6dd74sj0vpyjy9he8pep73tt8pljhv6mc74y28rr8995yjyahgh9kc2jqwqz62h6u6jpqz0u9x6gcahdw3pe2x8r5eyppp53mlpru8sp4p4nf6","onionreply":"c5a02ce5372635959d87818f723767efe3137bbc40568fb04e1f2eedee0a6149000e400f00000000000027100000006c00f20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}}
```

```
2026-01-20T05:25:55.4465729Z         with pytest.raises(RpcError, match=r"Unknown invoice") as excinfo:
2026-01-20T05:25:55.4466111Z >           l1.rpc.call("renepay", {"invstring": inv2})
2026-01-20T05:25:55.4466310Z 
2026-01-20T05:25:55.4466402Z tests/test_renepay.py:423: 
...
2026-01-20T05:25:55.4489732Z         elif "error" in resp:
2026-01-20T05:25:55.4490005Z >           raise RpcError(method, payload, resp['error'])
2026-01-20T05:25:55.4492135Z E           pyln.client.lightning.RpcError: RPC call failed: method: renepay, payload: {'invstring': 'lnbcrt100n1p5k7yfqsp5g072rrl6jn7wu8nlxk279zpkacz34hr22n089j4elguccfxmg0gqpp5s3r4jtwr0pkrrqt5d0tm79l7n6zd8h3alluy070jhrq6vhpd8lmqdqgw3jhxapjxqyjw5qcqp99qxpqysgq6dd74sj0vpyjy9he8pep73tt8pljhv6mc74y28rr8995yjyahgh9kc2jqwqz62h6u6jpqz0u9x6gcahdw3pe2x8r5eyppp53mlpru8sp4p4nf6'}, error: {'code': -4, 'message': 'Plugin terminated before replying to RPC call.'}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
f799c4c556 pytest: fix flake in test_sql.
If l2 hasn't seen l1's node_announcement yet:

```
        # Correctly handles missing object.
>       assert l2.rpc.sql("SELECT option_will_fund_lease_fee_base_msat,"
                          " option_will_fund_lease_fee_basis,"
                          " option_will_fund_funding_weight,"
                          " option_will_fund_channel_fee_max_base_msat,"
                          " option_will_fund_channel_fee_max_proportional_thousandths,"
                          " option_will_fund_compact_lease"
                          " FROM nodes WHERE HEX(nodeid) = '{}';".format(l1.info['id'].upper())) == {'rows': [[None] * 6]}
E       AssertionError: assert {'rows': []} == {'rows': [[None, None, None, None, None, None]]}
E         
E         Differing items:
E         {'rows': []} != {'rows': [[None, None, None, None, None, None]]}
E         
E         Full diff:
E           {
E         -     'rows': [
E         +     'rows': [],
E         ?              ++
E         -         [
E         -             None,
E         -             None,
E         -             None,
E         -             None,
E         -             None,
E         -             None,
E         -         ],
E         -     ],
E           }

tests/test_plugin.py:4131: AssertionError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 16:12:04 +10:30
Rusty Russell
5c31d33786 common: remove legacy onion translation.
This was added in 24.05, but LND since 0.18.3 no longer ever creates
such onions, and even that version (September 2024) is now a long way
behind.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: we no longer support legacy onions (never sent by LND >= 0.18.3, which was the last)
2026-02-03 15:35:36 +10:30
Rusty Russell
2c0214f246 common: remove unused type field.
We always use modern TLV style: in fact, this field was never set!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 15:35:36 +10:30
Rusty Russell
6a937307b1 decode: don't treat every failure as a rune.
If we can't decode something, and it decodes as a rune (and all bech32
strings do!), then we would usually just complain it was a malformed
rune.  Be a big more useful, when the parameter looks like somthing else.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `decode` is now more informative with malformed strings (won't claim everything is a malformed rune!).
2026-02-03 13:05:23 +10:30
Rusty Russell
5053e2e3f6 common: enforce currency-requires-amount check.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 13:05:23 +10:30
Rusty Russell
1de4e1dc28 pytest: test for amountless offer with currency.
Should fail: see https://github.com/lightning/bolts/pull/1279

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-03 13:05:23 +10:30
Tatiana Moroz
0173610527 Fix Reckless search command not finding partial matches
The Reckless search command was only returning a result if you
searched a perfect match, which is not too helpful.  This updates the
command so that partial search matches return a result.

Before:
reckless search bolt
Search exhausted all sources
reckless search bol
Search exhausted all sources
reckless search bolt12-pris
Search exhausted all sources

After:
reckless search bolt
Plugins matching 'bolt':
  bolt12-prism (https://github.com/lightningd/plugins)
reckless search bol
Plugins matching 'bol':
  bolt12-prism (https://github.com/lightningd/plugins)
reckless search bolt12-pris
Plugins matching 'bolt12-pris':
  bolt12-prism (https://github.com/lightningd/plugins)

Changelog-Fixed: reckless search now returns partial matches instead of requiring exact plugin names.
2026-02-02 13:01:00 -06:00
Christian Decker
ff34e485bc feat: Enhance JUnit XML reporting with environment variables and add import to conftest.py
Changelog-None
2026-02-02 11:41:09 +01:00
21M4TW
d76e4e7149 lightningd: don't allow enableoffer on single-use offer.
Changelog-Fixed: enableoffer: Adding an error when trying to activate an used single use offer (don't crash!)
2026-02-02 10:31:02 +10:30
Rusty Russell
e2d17cea0c pytest: test for crash when enableoffer called on a used single-use offer.
Assertion happens here:

	newstatus = offer_status_in_db(s | OFFER_STATUS_ACTIVE_F);

Since OFFER_STATUS_SINGLE_F|OFFER_STATUS_USED_F|OFFER_STATUS_ACTIVE_F
is not a valid combination:

```
lightningd-3 2026-01-28T04:45:21.184Z **BROKEN** lightningd: offer_status_in_db: 7 is invalid
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v25.12-92-g7fff32d-modded)
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: common/daemon.c:83 (crashdump) 0x5a883759dbb7
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x79a2b0c4532f
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:44 (__pthread_kill_implementation) 0x79a2b0c9eb2c
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:78 (__pthread_kill_internal) 0x79a2b0c9eb2c
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:89 (__GI___pthread_kill) 0x79a2b0c9eb2c
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ../sysdeps/posix/raise.c:26 (__GI_raise) 0x79a2b0c4527d
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x79a2b0c288fe
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/log.c:1054 (fatal_vfmt) 0x5a8837509557
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/log.c:1064 (fatal) 0x5a88375095fe
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ./wallet/wallet.h:1451 (offer_status_in_db) 0x5a88375491dc
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: wallet/wallet.c:6160 (offer_status_in_db) 0x5a8837555388
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: wallet/wallet.c:6162 (wallet_offer_enable) 0x5a8837555388
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/offer.c:288 (json_enableoffer) 0x5a8837540939
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:769 (command_exec) 0x5a8837503198
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:910 (rpc_command_hook_final) 0x5a8837503198
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:884 (rpc_command_hook_final) 0x5a8837503198
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:243 (hook_done) 0x5a8837535383
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:343 (plugin_hook_call_next) 0x5a8837535383
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:998 (plugin_hook_call_rpc_command) 0x5a8837503c4f
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1123 (parse_request) 0x5a8837503c4f
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1217 (read_json) 0x5a8837503c4f
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x5a88375eca38
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x5a88375eca38
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x5a88375eca38
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:470 (io_loop) 0x5a88375eead5
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x5a8837501f8e
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1492 (main) 0x5a88374d3c27
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x79a2b0c2a1c9
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: ../csu/libc-start.c:360 (__libc_start_main_impl) 0x79a2b0c2a28a
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x5a88374d5aa4
lightningd-3 2026-01-28T04:45:21.260Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2026-02-02 10:31:02 +10:30
Lagrang3
f1bf66c91f askrene: fixed a timeout corner case
Changelog-Fixed: askrene: fixed a class of corner cases that cause askrene main loop to timeout instead of quickly failing, thus wasting runtime.

Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
2026-02-02 10:23:27 +10:30