Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `gossipd` now uses a `lightning_gossip_compactd` helper to compact the gossip_store on demand, keeping it under about 210MB.
A new subprocess run by gossipd to create a compacted gossip store.
It's pretty simple: a linear compaction of the file. Once it's done the amount it
was told to, then gossipd waits until it completes the last bit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
gossip_store.c uses this to avoid two reads, and we want to use it
elsewhere too.
Also fix old comment on gossip_store_readhdr().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the file responsible for all the writing, so it should be
responsible for the rewriting if necessary (rather than
gossmap_manage).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This saves gossipd from converting it:
```
lightningd-1 2026-02-02T00:50:49.505Z DEBUG gossipd: Time to convert version 14 store: 890 msec
```
Reducing node startup time from 1.4 seconds to 0.5 seconds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now only need to walk it if we're doing an upgrade.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: `gossipd` no longer compacts gossip_store on startup (improving start times significantly).
gossmap doesn't care, so gossipd currently has to iterate through the
store to find them at startup. Create a callback for gossipd to use
instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's used by common/gossip_store.c, which is used by many things other than
gossipd. This file belongs in common.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We also put this in the store_ended message, too: so you can
tell if the equivalent_offset there really refers to this new
entry (or if two or more rewrites have happened).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's actually quite quick to load a cache-hot 308,874,377 byte
gossip_store (normal -Og build), but perf does show time spent
in siphash(), which is a bit overkill here, so drop that:
Before:
Time to load: 66718983-78037766(7.00553e+07+/-2.8e+06)nsec
After:
Time to load: 54510433-57991725(5.61457e+07+/-1e+06)nsec
We could save maybe 10% more by disabling checksums, but having
that assurance is nice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our poor scid generation clashes badly with simplified hashing (the
next patch) leading to l1's startup time when using a generated map
moving from 4 seconds to 14 seconds. Under CI it actually timed out
several tests.
Fixing our fake scids to be more "random" reduces it to 1.5 seconds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It didn't do anything, since the dev_compact_gossip_store command was
removed. When we make it do something, it crashes since old_len is 0:
```
gossipd: gossip_store_compact: bad version
gossipd: FATAL SIGNAL 6 (version v25.12rc3-1-g9e6c715-modded)
...
gossipd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x7119bd8288fe
gossipd: backtrace: ./assert/assert.c:96 (__assert_fail_base) 0x7119bd82881a
gossipd: backtrace: ./assert/assert.c:105 (__assert_fail) 0x7119bd83b516
gossipd: backtrace: gossipd/gossip_store.c:52 (append_msg) 0x56294de240eb
gossipd: backtrace: gossipd/gossip_store.c:358 (gossip_store_compact) 0x56294
gossipd: backtrace: gossipd/gossip_store.c:395 (gossip_store_new) 0x56294de24
gossipd: backtrace: gossipd/gossmap_manage.c:455 (setup_gossmap) 0x56294de255
gossipd: backtrace: gossipd/gossmap_manage.c:488 (gossmap_manage_new) 0x56294
gossipd: backtrace: gossipd/gossipd.c:400 (gossip_init) 0x56294de22de9
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not a complete decode, just the highlights (what channel was announced
or updated, what node was announced).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's always true for the first hook invocation, but if there is more
than one plugin, it could vanish between the two! In the default configuration, this can't happen.
This bug has been around since v23.02.
Note: we always tell all the plugins about the peer, even if it's
already gone.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightningd: possible crash when peers disconnected if there was more than one plugin servicing the `peer_connected` hook.
Reported-by: https://github.com/santyr
Fixes: https://github.com/ElementsProject/lightning/issues/8858
In November 2022 we seemed to increase parallelism from 2 and 3 to 10!
That is a huge load for these CI boxes, and does explain some of our
flakes.
We only run in parallel because some tests sleep, but it's diminishing
returns (GH runners have 4 VCPUs, 16GB RAM).
This reduces it so:
- Normal runs are -n 4
- Valgrind runs are -n 2
- Sanitizer runs are -n 3
If I use my beefy build box (64BG RAM) but reduce it to 4 CPUs:
Time for pytest -n 5:
Time for pytest -n 4:
Time for pytest -n 3:
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Avoids guessing what the timeout should be, use a file trigger. This
is more optimal, and should reduce a flake in test_sql under valgrind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It uses the hold_invoice plugin to ensure that an HTLC is in flight, but
it tells it to hold the HTLC for "TIMEOUT * 2" which is a big number under CI.
Reduce it to sqrt(TIMEOUT + 1) * 2, which works for local testing (I run
with TIMEOUT=10) and still should be enough for CI (TIMEOUT=180).
Christian reported that the test took 763.00s (!!) under CI.
On my build machine (TIMEOUT=90):
Before:
383.00s
After:
64.38s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. If max was 0, we crashed with SIGFPE due to % 0.
2. If min was non-zero, logic was incorrect (but all callers had min == 0).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
2026-01-30T05:55:13.6654636Z # Note that l3 has the whole lease delay (minus blocks already mined)
2026-01-30T05:55:13.6655396Z _, _, l3blocks = l3.wait_for_onchaind_tx('OUR_DELAYED_RETURN_TO_WALLET',
2026-01-30T05:55:13.6656086Z 'OUR_UNILATERAL/DELAYED_OUTPUT_TO_US')
2026-01-30T05:55:13.6656618Z > assert l3blocks == 4032 - 6 - 2 - 1
2026-01-30T05:55:13.6657033Z E assert 4025 == (((4032 - 6) - 2) - 1)
```
Turns out that 4342043382 (tests: de-flake test that was failing on
cltv expiry) added a line to mine two more blocks, but the hardcoded
110 was not changed to 112, so we weren't actually waiting correctly.
Remove hardcoded numbers in favor of calculation, and do the same in
test_channel_lease_post_expiry (which was correct, for now).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Normally, connectd forwards messages and then the subds do logging,
but it logs manually for msgs which are handled internally.
Clarify this logic in one place for all callers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If l2 hasn't seen l1's node_announcement yet:
```
# Correctly handles missing object.
> assert l2.rpc.sql("SELECT option_will_fund_lease_fee_base_msat,"
" option_will_fund_lease_fee_basis,"
" option_will_fund_funding_weight,"
" option_will_fund_channel_fee_max_base_msat,"
" option_will_fund_channel_fee_max_proportional_thousandths,"
" option_will_fund_compact_lease"
" FROM nodes WHERE HEX(nodeid) = '{}';".format(l1.info['id'].upper())) == {'rows': [[None] * 6]}
E AssertionError: assert {'rows': []} == {'rows': [[None, None, None, None, None, None]]}
E
E Differing items:
E {'rows': []} != {'rows': [[None, None, None, None, None, None]]}
E
E Full diff:
E {
E - 'rows': [
E + 'rows': [],
E ? ++
E - [
E - None,
E - None,
E - None,
E - None,
E - None,
E - None,
E - ],
E - ],
E }
tests/test_plugin.py:4131: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was added in 24.05, but LND since 0.18.3 no longer ever creates
such onions, and even that version (September 2024) is now a long way
behind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: we no longer support legacy onions (never sent by LND >= 0.18.3, which was the last)
If we can't decode something, and it decodes as a rune (and all bech32
strings do!), then we would usually just complain it was a malformed
rune. Be a big more useful, when the parameter looks like somthing else.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `decode` is now more informative with malformed strings (won't claim everything is a malformed rune!).
The Reckless search command was only returning a result if you
searched a perfect match, which is not too helpful. This updates the
command so that partial search matches return a result.
Before:
reckless search bolt
Search exhausted all sources
reckless search bol
Search exhausted all sources
reckless search bolt12-pris
Search exhausted all sources
After:
reckless search bolt
Plugins matching 'bolt':
bolt12-prism (https://github.com/lightningd/plugins)
reckless search bol
Plugins matching 'bol':
bolt12-prism (https://github.com/lightningd/plugins)
reckless search bolt12-pris
Plugins matching 'bolt12-pris':
bolt12-prism (https://github.com/lightningd/plugins)
Changelog-Fixed: reckless search now returns partial matches instead of requiring exact plugin names.
