This happens in the fuzzer corpora, but that doesn't check for take()
leaks. Our unit tests do:
```
fuzz-initial_channel: outstanding taken(): 0x626c3b3affc8
make: *** [Makefile:823: unittest/tests/fuzz/fuzz-initial_channel] Error 1
```
This doesn't matter in real life, since we exit the subdaemon if this
fails, but it's still a bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Basically, `devtools/reduce-includes.sh */*.c`.
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
After:
real 0m37.872000-39.974000(39.5466+/-0.59)s
user 3m1.211000-14.968000(12.4556+/-3.9)s
sys 0m35.008000-36.830000(36.4143+/-0.5)s
Build time after touch config.vars (RUST=0):
Before:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
After:
real 0m18.384000-21.307000(20.8605+/-0.92)s
user 2m5.585000-26.843000(23.6017+/-6.7)s
sys 0m19.650000-22.003000(21.4943+/-0.69)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Each header should only include the other headers it needs to compile;
`devtools/reduce-includes.sh */*.h` does this. The C files then need
additional includes if they don't compile.
And remove the entirely useless wire/onion_wire.h, which only serves to include wire/onion_wiregen.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. $(JSMN_OBJS) is not set anywhere.
2. You don't need to depend on CCAN_HEADERS, COMMON_HEADERS or JSMN_HEADERS: the top level Makefile has all object depedning on it.
3. Similarly, CCAN_OBJS.
4. Every object file should be rebuilt if its Makefile changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to manually choose what to link against,
which is much of the complexity of our Makefiles: the compiler will
automatically use any object files it needs to link.
We already do this for ccan as libccan.a, now we have libcommon.a.
We don't link against it for *everything*, as some tests require their own
versions.
Notes:
1. I get rid of the weird plugins/test/Makefile2 (accidental commit?)
2. Many tests change due to update-mocks.
3. In some places I added the missing dependency on the Makefile itself, though most are in the next
patch.
Before:
Total program size: 221366528
Total tests size: 364243856
After:
Total program size: 190733656
Total tests size: 337880888
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.227000-44.245000(41.8222+/-1.6)s
user 3m2.105000-33.696000(23.1442+/-8.4)s
sys 0m35.054000-42.269000(39.7231+/-2)s
After:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
Build time after touch config.vars (RUST=0):
Before:
real 0m18.928000-22.776000(21.5084+/-1.1)s
user 2m8.613000-36.567000(27.7281+/-7.7)s
sys 0m20.458000-23.436000(22.3963+/-0.77)s
After:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rusty@rusty-Framework:~/devel/cvs/lightni
This was changing all the time when I tried to make
autogenerate-rpc-examples.py reproducible. Turns out it was being
corrupted (it does suspicious things with pointers); rather than try
to diagnose it, I simply rewrote the code to create it only when we
need it.
```
Valgrind error file: valgrind-errors.34506
==34506== Uninitialised byte(s) found during client check request
==34506== at 0x241732: memcheck_ (mem.h:247)
==34506== by 0x2417BC: towire (towire.c:17)
==34506== by 0x24185C: towire_u16 (towire.c:28)
==34506== by 0x20C8E4: towire_tlv_scb_tlvs_remote_to_self_delay (scb_wiregen.c:213)
==34506== by 0x240E78: towire_tlv (tlvstream.c:342)
==34506== by 0x20C99F: towire_tlv_scb_tlvs (scb_wiregen.c:234)
==34506== by 0x20C298: towire_modern_scb_chan (scb_wiregen.c:89)
==34506== by 0x1A6CF3: json_add_scb (peer_control.c:2488)
==34506== by 0x1A6E0C: json_staticbackup (peer_control.c:2519)
==34506== by 0x177E3F: command_exec (jsonrpc.c:799)
==34506== by 0x1785AE: rpc_command_hook_final (jsonrpc.c:945)
==34506== by 0x1BEC2D: plugin_hook_call_next (plugin_hook.c:199)
==34506== Address 0x1ffeffe736 is on thread 1's stack
==34506== in frame #2, created by towire_u16 (towire.c:26)
==34506==
{
<insert_a_suppression_name_here>
Memcheck:User
fun:memcheck_
fun:towire
fun:towire_u16
fun:towire_tlv_scb_tlvs_remote_to_self_delay
fun:towire_tlv
fun:towire_tlv_scb_tlvs
fun:towire_modern_scb_chan
fun:json_add_scb
fun:json_staticbackup
fun:command_exec
fun:rpc_command_hook_final
fun:plugin_hook_call_next
}
==34506== Uninitialised byte(s) found during client check request
==34506== at 0x241732: memcheck_ (mem.h:247)
==34506== by 0x2417BC: towire (towire.c:17)
==34506== by 0x240EF0: towire_tlv (tlvstream.c:354)
==34506== by 0x20C99F: towire_tlv_scb_tlvs (scb_wiregen.c:234)
==34506== by 0x20C298: towire_modern_scb_chan (scb_wiregen.c:89)
==34506== by 0x1A6CF3: json_add_scb (peer_control.c:2488)
==34506== by 0x1A6E0C: json_staticbackup (peer_control.c:2519)
==34506== by 0x177E3F: command_exec (jsonrpc.c:799)
==34506== by 0x1785AE: rpc_command_hook_final (jsonrpc.c:945)
==34506== by 0x1BEC2D: plugin_hook_call_next (plugin_hook.c:199)
==34506== by 0x1BEBA8: plugin_hook_callback (plugin_hook.c:186)
==34506== by 0x1B771E: plugin_response_handle (plugin.c:705)
==34506== Address 0x7bd1d08 is 40 bytes inside a block of size 42 alloc'd
==34506== at 0x484DCD3: realloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==34506== by 0x3C8614: tal_resize_ (tal.c:755)
==34506== by 0x2417A2: towire (towire.c:14)
==34506== by 0x24185C: towire_u16 (towire.c:28)
==34506== by 0x20C8E4: towire_tlv_scb_tlvs_remote_to_self_delay (scb_wiregen.c:213)
==34506== by 0x240E78: towire_tlv (tlvstream.c:342)
==34506== by 0x20C99F: towire_tlv_scb_tlvs (scb_wiregen.c:234)
==34506== by 0x20C298: towire_modern_scb_chan (scb_wiregen.c:89)
==34506== by 0x1A6CF3: json_add_scb (peer_control.c:2488)
==34506== by 0x1A6E0C: json_staticbackup (peer_control.c:2519)
==34506== by 0x177E3F: command_exec (jsonrpc.c:799)
==34506== by 0x1785AE: rpc_command_hook_final (jsonrpc.c:945)
==34506==
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The equality check will fail, but it will show is what is missing, rather than:
FAILED tests/test_coinmoves.py::test_coinmoves_unilateral_htlc_fulfill - ValueError: Timeout while waiting for <function check_chain_moves.<locals>.<lambda> at 0x7f7800941ab0>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
when bash expands bitcoin/*.h, it returns the files in lexicographically sorted order by default this is not necessarily the case for macOS so it has be explicitly sorted. I get a fairly uninformative error like this:
make: *** [check-bitcoin-makefile] Error 1
The error is now more informative and does not error on a clean branch on macOS:
BITCOIN_HEADERS missing: bitcoin/signature.h bitcoin/tx_parts.h bitcoin/tx.h bitcoin/varint.h
make: *** [check-bitcoin-makefile] Error 1
If the fees are not *all* of the fees (as we do in next patch), the
query would be wrong. Plus, as the FIXME suggests, we should just save
it as we're getting the fee_sums, not do a whole new query!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using `tal_dup_or_null` does not set the tal context to the included
bytes array. Luckily we already have a function tha deeply duplicates
a `feature_set`: `feature_set_dup`
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Workflow error `gpg: using "4129A994AA7E9852" is thrown due to incorrect gpg parsing. Update the awk parsing logic to properly locate and extract the key fingerprint within the gpgconf --list-options output structure, ensuring automated signing uses the correct key.
Changelog-None.
Enable fuzzing support on macOS by configuring the build system to use Homebrew LLVM toolchain and handle macOS-specific linking requirements.
The `make check-fuzz` command was failing on macOS because:
- System clang lacks fuzzer runtime library support
- Linking issues with fuzzer targets
- Test script attempts to execute debug symbol files
This PR adds macOS-specific configuration to:
- Use Homebrew LLVM toolchain for fuzzer support
- Explicitly link fuzzer libraries
- Exclude `.dSYM` directories from test discovery
All 76 fuzzer targets now build and pass on macOS.
If we read all of them, we might get 1.6M at once (after initial
migration). Then we submit a few hundred thousand simultaneous
requests to lightningd, and it gets upset, queueing them all on the
xpay command hook and running out of memory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: plugins: bookkeeper first invocation after migration from prior to 25.09 with very large databases will not crash.
This significantly speeds up the query which bookkeeper often does:
"SELECT created_index"
" FROM channelmoves"
" WHERE payment_hash = X'%s'"
" AND credit_msat = %"PRIu64
" AND created_index <= %"PRIu64,
On large databases this scan is expensive, and a payment_hash index
cuts it down a great deal. It does take longer to load the channelmoves
in the first place though (about 3x).
Before:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
169505 /tmp/bkpr-progress
196010 /tmp/bkpr-progress
219370 /tmp/bkpr-progress
235671 /tmp/bkpr-progress
244242 /tmp/bkpr-progress
255362 /tmp/bkpr-progress
265636 /tmp/bkpr-progress
276966 /tmp/bkpr-progress
284451 /tmp/bkpr-progress
288836 /tmp/bkpr-progress
296578 /tmp/bkpr-progress
304571 /tmp/bkpr-progress
After:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
161421 /tmp/bkpr-progress
238273 /tmp/bkpr-progress
281185 /tmp/bkpr-progress
305787 /tmp/bkpr-progress
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: plugins: the sql plugin now keeps an index on `channelmoves` by `payment_hash`.
Note that we create a destructor on the command to reset request->cmd
pointer if the cmd is freed (so we know not to call the callback).
But attaching hundreds of thousands of them is slow: it's a
single-linked list, which is iterated in several places.
But that's redundant: the request is now allocated off the cmd, so freeing the command
will free the request anyway.
Hacking in something to print progress to a file, here's the number of
requests processed every 10 seconds before and after:
Before:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
181529 /tmp/bkpr-progress
195994 /tmp/bkpr-progress
207083 /tmp/bkpr-progress
226336 /tmp/bkpr-progress
234319 /tmp/bkpr-progress
241514 /tmp/bkpr-progress
247421 /tmp/bkpr-progress
255292 /tmp/bkpr-progress
261367 /tmp/bkpr-progress
269085 /tmp/bkpr-progress
276953 /tmp/bkpr-progress
282233 /tmp/bkpr-progress
286193 /tmp/bkpr-progress
290930 /tmp/bkpr-progress
295276 /tmp/bkpr-progress
301086 /tmp/bkpr-progress
After:
169505 /tmp/bkpr-progress
196010 /tmp/bkpr-progress
219370 /tmp/bkpr-progress
235671 /tmp/bkpr-progress
244242 /tmp/bkpr-progress
255362 /tmp/bkpr-progress
265636 /tmp/bkpr-progress
276966 /tmp/bkpr-progress
284451 /tmp/bkpr-progress
288836 /tmp/bkpr-progress
296578 /tmp/bkpr-progress
304571 /tmp/bkpr-progress
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use some GNU sed features and it's easier to require gsed than fix
them all (and keep noticing when they break). MacOS comes with gsed
already!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We fix the configure test which was bogus, and breaks on 32-bit gcc with optimization:
```
checking for unaligned access to int... ccan/tools/configurator/configurator: Test for HAVE_UNALIGNED_ACCESS did not compile:
configuratortest.c: In function 'main':
configuratortest.c:8:22: error: array subscript 'int[0]' is partly outside array bounds of 'char[4]' [-Werror=array-bounds]
8 | return *x == *y;
| ^~
configuratortest.c:5:11: note: at offset 1 into object 'pad' of size 4
5 | char pad[sizeof(int *) * 1];
| ^~~
cc1: all warnings being treated as errors
```
And also removes sprintf() calls, which apparently MacOS no longer likes:
Environment: Sequoia 15.6.1 (24G90)
SDK Version: MacOSX15.sdk
```
ccan/ccan/closefrom/closefrom.c:83:2: error: 'sprintf' is deprecated: This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Werror,-Wdeprecated-declarations]
83 | sprintf(dnam, "/proc/%ld/fd", (long) getpid());
| ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:278:1: note: 'sprintf' has been explicitly marked deprecated here
278 | deprecated_msg("This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
| ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro 'deprecated_msg'
218 | #define deprecated_msg(_msg) attribute((deprecated(_msg)))
| ^
ccan/ccan/closefrom/closefrom.c:162:2: error: 'sprintf' is deprecated: This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Werror,-Wdeprecated-declarations]
162 | sprintf(dnam, "/proc/%ld/fd", (long) getpid());
| ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/_stdio.h:278:1: note: 'sprintf' has been explicitly marked deprecated here
278 | deprecated_msg("This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.")
| ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro 'deprecated_msg'
218 | #define deprecated_msg(_msg) attribute((deprecated(_msg)))
| ^
2 errors generated.
make: *** [ccan-closefrom.o] Error 1
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: Sangbida
Fixes: https://github.com/ElementsProject/lightning/issues/8581
Reported-by: https://github.com/Raimo33
Fixes: https://github.com/ElementsProject/lightning/issues/8501
Changelog-Fixed: build: we now build on MacOS without errors on the latest Command Line Tools (macOS 15 SDK).
Changelog-Fixes: build: fix build with -O2 on 32 bit arm (armhf)
Testing a large db shows Postgres slowing down exponentially as
it inserts the channel_events. Rather than updating the index in the
db every time, do it at the end, for spectacular speedup:
```
lightningd-1 2025-10-08T05:39:44.333Z INFO lightningd: Creating database
lightningd-1 2025-10-08T05:39:47.581Z DEBUG lightningd: Transferring 6166 chain_events
lightningd-1 2025-10-08T05:39:48.455Z DEBUG lightningd: Transferring 1660043 channel_events
lightningd-1 2025-10-08T05:39:54.390Z INFO lightningd: Inserted 103100/1660043 channel_events
lightningd-1 2025-10-08T05:40:04.390Z INFO lightningd: Inserted 283280/1660043 channel_events
lightningd-1 2025-10-08T05:40:14.390Z INFO lightningd: Inserted 464065/1660043 channel_events
lightningd-1 2025-10-08T05:40:24.390Z INFO lightningd: Inserted 629559/1660043 channel_events
lightningd-1 2025-10-08T05:40:34.390Z INFO lightningd: Inserted 800659/1660043 channel_events
lightningd-1 2025-10-08T05:40:44.390Z INFO lightningd: Inserted 975433/1660043 channel_events
lightningd-1 2025-10-08T05:40:54.390Z INFO lightningd: Inserted 1134719/1660043 channel_events
lightningd-1 2025-10-08T05:41:04.390Z INFO lightningd: Inserted 1290549/1660043 channel_events
lightningd-1 2025-10-08T05:41:14.390Z INFO lightningd: Inserted 1443304/1660043 channel_events
lightningd-1 2025-10-08T05:41:24.390Z INFO lightningd: Inserted 1590013/1660043 channel_events
lightningd-1 2025-10-08T05:41:29.148Z INFO lightningd: bookkeeper migration complete: migrated 6166 chainmoves, 1660043 channelmoves, 132481 descriptions
```
Now we complete the entire migration in 1 minute 45 seconds.
Thanks to @michael1101 for reporting this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: db: migration from v25.09 on a reasonable size account database could take almost infinite time.
This allows compatibility with python 3.14.0 which coincurve 21.0.0 did
not support. The next coincurve release should restore compatibility.
Fixes: #8591
Changelog-changed: pyln-testing requires python>=3.9.2
We need to remove the feature bits set via a plugins get_manifest
response when the init response disables the plugin.
Changelog-Fixed Remove feature bits set by a plugin when the plugin
disables itself during init.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
The bitcoin.rpc.DEFAULT_HTTP_TIMEOUT of 30 seconds may not be enough
time to generate a block when the test machine is under load. Pass
pyln.testing.utils.TIMEOUT to bitcoin.rpc.RawProxy to allow extra time:
currently 60 seconds by default or 180 seconds if SLOW_MACHINE is set.
Changelog-None
When we migrate from accounts.db, we use the `account_nonchannel_id`
field. But we can replay the block chain and the channel involved is
still open, we will use the `account_channel_id` field, and our duplicate
detection fails.
As a result, we can end up with duplicate entries in the database, which
make accounting incorrect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `listchainmoves` could contain bogus duplicate entries after 25.09 bookkeeper migration.
For old channels, this can take a while, and it stops everything. But
we are only doing this to save space; it's not a *functional* necessity.
A quick and dirty test with 50,000 htlcs shows the htlc deletion took
450msec. I tried adding an index, and changing it to set hstate to
HTLC_STATE_INVALID instead of deleting entries, but it still took about 350ms.
Whereas the "COUNT(*)" only took 1.7msec, so it's worth keeping.
Reported-by: @michael1011
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: lightningd: we defer deletion of old htlcs on channel close, to avoid pausing for a long time (we clean them on startup)
Fixes: https://github.com/ElementsProject/lightning/issues/7962
