gossmap doesn't care, so gossipd currently has to iterate through the
store to find them at startup. Create a callback for gossipd to use
instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's used by common/gossip_store.c, which is used by many things other than
gossipd. This file belongs in common.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
common/test/run-param.c:381:8: runtime error: applying zero offset to null pointer
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior common/test/run-param.c:381:8
```
Probably because CI now on 24.04, so more recent clang. But the test really does
want to see what happens when the callback is NULL, so workaround.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Only in developer mode, ofc.
Notes:
1. We have to move the initialization before the lightningd main trace_start,
since that uses pseudorand().
2. To make the results stable, we need to use per-caller values to randbytes().
Otherwise external timing changes the call order.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The efficient way to do this is to use membuf, which handles the buffer
control (only using memmove when necessary). We have multiple places
where we opencoded this, some of which did not use membuf at all.
So now we create common infrastructure. I tried making it a single
function but the various users are quite different, so instead I opted
for a toolbox approach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Basically, `devtools/reduce-includes.sh */*.c`.
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
After:
real 0m37.872000-39.974000(39.5466+/-0.59)s
user 3m1.211000-14.968000(12.4556+/-3.9)s
sys 0m35.008000-36.830000(36.4143+/-0.5)s
Build time after touch config.vars (RUST=0):
Before:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
After:
real 0m18.384000-21.307000(20.8605+/-0.92)s
user 2m5.585000-26.843000(23.6017+/-6.7)s
sys 0m19.650000-22.003000(21.4943+/-0.69)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Each header should only include the other headers it needs to compile;
`devtools/reduce-includes.sh */*.h` does this. The C files then need
additional includes if they don't compile.
And remove the entirely useless wire/onion_wire.h, which only serves to include wire/onion_wiregen.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to manually choose what to link against,
which is much of the complexity of our Makefiles: the compiler will
automatically use any object files it needs to link.
We already do this for ccan as libccan.a, now we have libcommon.a.
We don't link against it for *everything*, as some tests require their own
versions.
Notes:
1. I get rid of the weird plugins/test/Makefile2 (accidental commit?)
2. Many tests change due to update-mocks.
3. In some places I added the missing dependency on the Makefile itself, though most are in the next
patch.
Before:
Total program size: 221366528
Total tests size: 364243856
After:
Total program size: 190733656
Total tests size: 337880888
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.227000-44.245000(41.8222+/-1.6)s
user 3m2.105000-33.696000(23.1442+/-8.4)s
sys 0m35.054000-42.269000(39.7231+/-2)s
After:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
Build time after touch config.vars (RUST=0):
Before:
real 0m18.928000-22.776000(21.5084+/-1.1)s
user 2m8.613000-36.567000(27.7281+/-7.7)s
sys 0m20.458000-23.436000(22.3963+/-0.77)s
After:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rusty@rusty-Framework:~/devel/cvs/lightni
Prior to 23.05, we used this tag to mark onchain to-self inputs we didn't
wait for (because they were too small). This fixes migration if that happened
(and we are debating whether we should re-introduce this!).
```
lightningd: FATAL SIGNAL 6 (version v25.09rc2)
0x100c8683 send_backtrace
common/daemon.c:33
0x100c876f crashdump
common/daemon.c:78
0x7fffb2080493 ???
???:0
0x7fffb1ab0cac ???
__pthread_kill_implementation+0x1bc:0
0x7fffb1a48a5b ???
__GI_raise+0x2b:0
0x7fffb1a2a3db ???
__GI_abort+0x153:0
0x100935b7 migrate_from_account_db
wallet/account_migration.c:424
0x10093ff7 db_migrate
wallet/db.c:1139
0x10096763 db_setup
wallet/db.c:1185
0x100a1bcb wallet_new
wallet/wallet.c:223
0x1004485f main
lightningd/lightningd.c:1311
0x7fffb1a2aba3 ???
__libc_start_call_main+0x93:0
0x7fffb1a2adeb ???
__libc_start_main_alias_1+0x1ab:0
0xffffffffffffffff ???
???:0
lightningd: Died with signal 6
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8484
If we don't have an accountdb from bookkeeper:
1. Generate a deposit chain event for every confirmed UTXO.
2. Generate an open chain event for every open, confirmed channel.
3. Generate a push/lease event if necessary.
4. Generate a fixup "journal" entry if balance is different from initial.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows the bookkeeper plugin to know it's not actually a channel account.
Remove the "ignored" tag from the schema too: we removed it previously.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids making an extra copy of the escaped string.
Note that jsonrpc_command_add() no longer accepts usage strings
containing invalid escape sequences. (Previously, it would quietly
accept such a string without unescaping anything.)
Changelog-None
bookkeeper used to generate these as channel events, now lightningd does.
We also add a "journal" event, which we will need later too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to store them in the db this way, so I thought I'd see what it looks like if
we lift that interface all the way through.
We use a struct, so that types are checked strictly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we can keep a pointer to the channel directly, *or* a string.
This avoids gratuitous formatting (on creation) and lookups (later).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make the payment secret field ('s') mandatory for BOLT11 payment requests,
implementing the requirement specified in BOLT11 spec PR #1242
(https://github.com/lightning/bolts/pull/1242).
This security enhancement prevents payment probing attacks by requiring
all invoices to include payment secrets. Changes include:
1. Adding validation in bolt11_decode_nosig() to reject invoices without
the 's' field
2. Adding payment secrets to all test vectors
3. Updating expected encoded values in test cases to include payment secrets
4. Adding a specific test case that verifies proper rejection of invoices
missing the payment secret field
Changelog-Changed: Made payment secret ('s' field) mandatory in BOLT11 payment requests for improved security.
This is much faster to give 64 bits of data, and we don't need
cryptographic randomness.
This brings us back to 413ns per trace.
Before:
real 0m5.819000-6.472000(6.2064+/-0.26)s
user 0m3.779000-4.101000(3.956+/-0.12)s
sys 0m2.040000-2.431000(2.2496+/-0.15)s
After:
real 0m3.981000-4.247000(4.1276+/-0.11)s
user 0m3.979000-4.245000(4.126+/-0.11)s
sys 0m0.000000-0.002000(0.001+/-0.00063)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightingd: trimmed overhead of tracing infrastructure.
With an average runtime of 18.7674, this implies 1876ns
per trace, which is far in excess of the 370ns claimed in
doc/developers-guide/tracing-cln-performance.md.
We also add a tag in there, so we measure that!
Results on my laptop:
real 0m18.524000-19.100000(18.7674+/-0.21)s
user 0m16.171000-16.833000(16.424+/-0.26)s
sys 0m2.259000-2.400000(2.337+/-0.059)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we included the spec for it, this is a good time to implement
it.
I also asked chatgpt to write some unit tests. I had to mangle them a
bit, but it probably saved me a few minutes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Default goes to stderr for LOG_UNUSUAL and higher.
We have to whitelist more cases in map_catchup so we don't spam the logs
with perfectly-expected (but ignored) messages though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The updated API requires typed htables to explicitly state whether they
allow duplicates: for most cases we don't, but we've had issues in the
past.
This is a big patch, but mainly mechanical.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I wanted to make sure we didn't have a bug in our htable routine,
so I wrote this test. We don't, but leave it in.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The `send_outreq` function is a good place to suspend and resume
traces, since these are usually the places where we hand off control
back to the `io_loop`. This assumes that we do not continue doing
heavy liftin after we have queued an `outreq` call, but that is most
likely the case anyway. This frees us from having to track suspensions
whenever we call the RPC from a plugin.
No code changes, just catching up with the BOLT changes which rework our
blinded path terminology (for the better!).
Another patch will sweep the rest of our internal names, this tries only to
make things compile and fix up the BOLT quotes.
1. Inside payload: current_blinding_point -> current_path_key
2. Inside update_add_htlc TLV: blinding_point -> blinded_path
3. Inside blinded_path: blinding -> first_path_key
4. Inside onion_message: blinding -> path_key.
5. Inside encrypted_data_tlv: next_blinding_override -> next_path_key_override
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, this lets you find the exact htlc_maximum_msat/htlc_minimum_msat
values.
This means we actually create real channel_updates for local mods, which
requires a second "local" scratch region.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
