A new subprocess run by gossipd to create a compacted gossip store.
It's pretty simple: a linear compaction of the file. Once it's done the amount it
was told to, then gossipd waits until it completes the last bit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's used by common/gossip_store.c, which is used by many things other than
gossipd. This file belongs in common.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows further reduction in binary sizes.
Size of all user before:
text data bss dec hex filename
64277932 904968 317576 65500476 3e7753c (TOTALS)
After:
52443036 729560 271720 53444316 32f7edc (TOTALS)
Some are dramatic, such as lightning-cli not pulling in libbacktrace:
174454 2880 296 177630 2b5de cli/lightning-cli
56633 2608 232 59473 e851 cli/lightning-cli
But most things get a 25% trim:
3972339 363568 21760 4357667 427e23 lightningd/lightningd
3300337 247768 21664 3569769 367869 lightningd/lightningd
Changelog-Changed: Build: most binaries are now about 20% smaller.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Users care about memory more than disk, and they care about programs
they run more than our test programs and tools.
Also, fix dependencies on print-binary-sizes, as spotted by
https://github.com/Lagrang3
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means:
1. downgrade changes (we no longer fail due to node biases).
2. various deprecations no longer are
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This commit adds automated Python API documentation generation for all
workspace packages using pdoc3:
- Add contrib/api/generate-python-docs.py script to generate docs
- Add Makefile targets: python-docs and python-docs-clean
- Add GitHub Actions workflow for nightly documentation generation
- Documents 5 packages: pyln.client, pyln.proto, pyln.grpc, pyln.testing, pyln.spec.bolt7
- Creates beautiful index page with cards linking to each package
- Stores generated docs as artifacts with 90-day retention
- Add pdoc3 and markdown to dev dependencies
Bug fix:
- Fix pyln-client version.py: __all__ must contain strings, not class objects
This was causing "TypeError: attribute name must be string, not 'type'" in pdoc3
Documentation is generated to docs/python/ which is excluded from version control.
Run 'make python-docs' to generate locally, or download from nightly workflow artifacts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit introduces a modern coverage infrastructure for Core Lightning:
- Migrate from ad-hoc coverage script to integrated Makefile targets
- Add LLVM source-based coverage support with per-test profraw organization
- Integrate coverage collection into pytest framework via TailableProc
- Add GitHub Actions workflow for nightly coverage reports
- Add Taskfile.yml for convenient task automation
- Add codecov.yml for Codecov integration
- Add comprehensive coverage documentation in COVERAGE.md
- Update contributor workflow docs with new coverage script path
- Add coverage data files to .gitignore (*.profraw, *.profdata)
- Remove obsolete contrib/clang-coverage-report.sh
- Remove obsolete tests/conftest.py (now using pyln-testing markers)
- Update pyproject.toml to include pyln-testing in main dependencies
The new infrastructure automatically collects coverage data when CLN_COVERAGE_DIR
is set, organizing profraw files by test name for granular analysis.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: tools: `lightningd-downgrade` can downgrade your database from v25.12 to v25.09 if something goes wrong.
When installed, the name is `lightning-hsmtool`. We actually copy
`tools/hsmtool` to `tools/lightning-hsmtool` but that's a silly step
which we should get rid of.
So:
1. Make sure our documentation always refers to it as lightning-hsmtool.
2. Make sure our tests invoke it as `lightning-hsmtool`.
3. Rename the C file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're removing the hardcoded homebrew paths for openssl and sqlite as those folder paths may not be correct for all macOS users. Instead we are now calling brew --prefix to get the correct path and use that to populate the LDFLAGS and CPPFLAGS
And add a check for new uses creeping in, since it got cut & paste
everywhere.
This means "this is a valid string, but truncate it to this many characters"
vs "%.*s" which means "only read this many characters of string":
```
['lightningd-3 2025-10-23T02:31:40.890Z **BROKEN** plugin-funder: Plugin marked as important, shutting down lightningd!']
--------------------------- Captured stderr teardown ---------------------------
#0 0x557da58ad1dc in printf_common(void*, char const*, __va_list_tag*) asan_interceptors.cpp.o
#1 0x557da5aff814 in json_out_addv /home/runner/work/lightning/lightning/ccan/ccan/json_out/json_out.c:239:11
#2 0x557da59740ce in plugin_logv /home/runner/work/lightning/lightning/plugins/libplugin.c:1777:2
#3 0x557da5969b6f in plugin_log /home/runner/work/lightning/lightning/plugins/libplugin.c:1934:2
#4 0x557da595c4f6 in datastore_del_success /home/runner/work/lightning/lightning/plugins/funder.c:161:2
#5 0x557da598b837 in handle_rpc_reply /home/runner/work/lightning/lightning/plugins/libplugin.c:1072:10
#6 0x557da598a4b0 in rpc_conn_read_response /home/runner/work/lightning/lightning/plugins/libplugin.c:1361:3
#7 0x557da5adbea5 in next_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:60:9
#8 0x557da5ae06ff in do_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:422:8
#9 0x557da5adfb58 in io_ready /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:439:10
#10 0x557da5aec2ce in io_loop /home/runner/work/lightning/lightning/ccan/ccan/io/poll.c:455:5
#11 0x557da59757ac in plugin_main /home/runner/work/lightning/lightning/plugins/libplugin.c:2409:3
#12 0x557da594fe23 in main /home/runner/work/lightning/lightning/plugins/funder.c:1723:2
#13 0x7f6572229d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#14 0x7f6572229e3f in __libc_start_main csu/../csu/libc-start.c:392:3
#15 0x557da588b584 in _start (/home/runner/work/lightning/lightning/plugins/funder+0x10d584) (BuildId: 71ba63ab577fc6fa60573d3e8555f6db7d5c584d)
0x624000009d28 is located 0 bytes to the right of 7208-byte region [0x624000008100,0x624000009d28)
allocated by thread T0 here:
#0 0x557da590e7f6 in __interceptor_realloc (/home/runner/work/lightning/lightning/plugins/funder+0x1907f6) (BuildId: 71ba63ab577fc6fa60573d3e8555f6db7d5c584d)
#1 0x557da5b2149b in tal_resize_ /home/runner/work/lightning/lightning/ccan/ccan/tal/tal.c:755:13
#2 0x557da59f2032 in membuf_tal_resize /home/runner/work/lightning/lightning/common/utils.c:203:2
#3 0x557da5b03934 in membuf_prepare_space_ /home/runner/work/lightning/lightning/ccan/ccan/membuf/membuf.c:45:12
#4 0x557da59d4289 in jsonrpc_io_read_ /home/runner/work/lightning/lightning/common/jsonrpc_io.c:127:2
#5 0x557da598a635 in rpc_conn_read_response /home/runner/work/lightning/lightning/plugins/libplugin.c:1366:9
#6 0x557da5adbea5 in next_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:60:9
#7 0x557da5ae06ff in do_plan /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:422:8
#8 0x557da5adfb58 in io_ready /home/runner/work/lightning/lightning/ccan/ccan/io/io.c:439:10
#9 0x557da5aec2ce in io_loop /home/runner/work/lightning/lightning/ccan/ccan/io/poll.c:455:5
#10 0x557da59757ac in plugin_main /home/runner/work/lightning/lightning/plugins/libplugin.c:2409:3
#11 0x557da594fe23 in main /home/runner/work/lightning/lightning/plugins/funder.c:1723:2
#12 0x7f6572229d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
SUMMARY: AddressSanitizer: heap-buffer-overflow asan_interceptors.cpp.o in printf_common(void*, char const*, __va_list_tag*)
Shadow bytes around the buggy address:
0x0c487fff9350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c487fff9390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c487fff93a0: 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa
0x0c487fff93b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c487fff93f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==26122==ABORTING
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we can make sure the compile and run in normal builds.
Side note: various tests call common_setup(), which means we called it
twice in unit testing mode, so we conditionalize those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to manually choose what to link against,
which is much of the complexity of our Makefiles: the compiler will
automatically use any object files it needs to link.
We already do this for ccan as libccan.a, now we have libcommon.a.
We don't link against it for *everything*, as some tests require their own
versions.
Notes:
1. I get rid of the weird plugins/test/Makefile2 (accidental commit?)
2. Many tests change due to update-mocks.
3. In some places I added the missing dependency on the Makefile itself, though most are in the next
patch.
Before:
Total program size: 221366528
Total tests size: 364243856
After:
Total program size: 190733656
Total tests size: 337880888
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.227000-44.245000(41.8222+/-1.6)s
user 3m2.105000-33.696000(23.1442+/-8.4)s
sys 0m35.054000-42.269000(39.7231+/-2)s
After:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
Build time after touch config.vars (RUST=0):
Before:
real 0m18.928000-22.776000(21.5084+/-1.1)s
user 2m8.613000-36.567000(27.7281+/-7.7)s
sys 0m20.458000-23.436000(22.3963+/-0.77)s
After:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rusty@rusty-Framework:~/devel/cvs/lightni
Enable fuzzing support on macOS by configuring the build system to use Homebrew LLVM toolchain and handle macOS-specific linking requirements.
The `make check-fuzz` command was failing on macOS because:
- System clang lacks fuzzer runtime library support
- Linking issues with fuzzer targets
- Test script attempts to execute debug symbol files
This PR adds macOS-specific configuration to:
- Use Homebrew LLVM toolchain for fuzzer support
- Explicitly link fuzzer libraries
- Exclude `.dSYM` directories from test discovery
All 76 fuzzer targets now build and pass on macOS.
all these changelogs only apply to the Docker image.
Changelog-Added: added verification of GPG keys for the bitcoin and litecoin tarballs.
Changelog-Fixed: fixed compilation on all target architectures; each had their own bugs (poetry, missing packages...).
Changelog-Fixed: fixed cargo cross compilation. it was mistakenly using QEMU before.
Changelog-Fixed: fixed CPU compatibility bug described in issue 8456
Changelog-Changed: improve build time by 8.8x
Changelog-Changed: improve image size by 2.07x
more detailed changelog can be found on the PR: https://github.com/ElementsProject/lightning/pull/8429
clnrest's `utoipa-swagger-ui` library has an indirect `rust-embed` dependency which by default includes timestamps in build. It results in non-deterministic build for clnrest. Using environment variable `SOURCE_DATE_EPOCH` with fixed value will enforce a consistent timestamp for builds.
Also adding the `--locked` flag to ensure the release build uses exact dependencies from Cargo.lock. The `--locked` flag is particularly important for deterministic builds as it prevents Cargo from updating the lockfile.
Fixes#8288.
Changelog-Fixed: Core lightning builds for Ubuntu Focal, Jammy and Noble are deterministic again.
1. We need to replace versions in __init__.py
2. We need to run uv after changing versions, so it updates uv.lock
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Extract package versions from pyproject.toml directly
instead of using poetry commands. Use `uv run` to execute flake8,
pytest and other Python tools consistently.
Add new make commands for uv builds
When installing core lightning on macOS I found that the debug symbols were not being preserved leading to "no debug info in
Mach-O executable". This is because the .dSYM files, which contain the debug information, is generated in the build directory and could not be found by the installed binaries.
Changes:
- Add -fno-standalone-debug flag to CDEBUGFLAGS on macOS to reduce
dependency on absolute source paths
- Modify install-program target to copy .dSYM bundles alongside
binaries for BIN_PROGRAMS, PKGLIBEXEC_PROGRAMS, and PLUGINS
Testing:
1. ./configure --reconfigure
2. make install PREFIX=/tmp/lightning-install
3. make clean (removes all .dSYM files from build directory)
4. /tmp/lightning-install/bin/lightningd --help
5. Verified stack traces now work correctly without "no debug info" errors
On macOS, libbacktrace was failing to find debug information due to:
1. Debug symbols not being properly linked with dsymutil
2. Apple Clang 17.0.0 generating DWARF 5 which libbacktrace couldn't parse
In this commit we address both issues:
Debug symbol accessibility:
- Add dsymutil integration in Makefile to properly link debug symbols
- Use -fno-standalone-debug to embed debug info inline in executable
DWARF format compatibility:
- Force -gdwarf-4 instead of default DWARF 5 to avoid "DW_FORM_addrx value out of range" errors
Changelog-added: libbacktrace works with macOS
Changelog-Changed: wss-proxy.py was replaced by a rust version with support for multiple `wss-bind-addr`. If you install CLN from pre-compiled binaries you must remove the old wss-proxy directory first before installing CLN, usually
it is located in `/usr/local/libexec/c-lightning/plugins/wss-proxy`. If you compile from source `make` will take care of this automatically.
These checks are a SHOULD, but implementing them helps avoid anyone
making such weird things in future.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Doesn't change anything for us, since we will already fulfull the incoming
HTLCs if we can, but good to note.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They weren't formatted correctly for bolts/tools/extract-formats.py
until this commit, so we had to patch them in manually.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When the command string run through $(call VERBOSE,…) contains a single-quote
character followed by shell metacharacters, the shell executing the VERBOSE
template's $(ECHO) command will attempt to interpret those metacharacters. This
could be disastrous, such as if someone were to put in a Makefile recipe:
$(call VERBOSE, , $(ECHO) 'Will not `rm -rf ~`')
When run with V=1, which causes VERBOSE to be defined as…
VERBOSE = $(ECHO) '$(2)'; $(2)
…Make would evaluate the above call into:
echo 'echo 'Will not `rm -rf ~`''; echo 'Will not `rm -rf ~`'
And oops, there goes the neighborhood.
The real-world motivating case for this fix is the sed call in the recipe for
doc/index.rst in doc/Makefile. It contains a sed expression enclosed in single
quotes, containing parentheses. When run through VERBOSE with V=1, the single
quotes around the sed expression actually escape _out_ of the single-quoted
string that is intended to be the whole command line, and you get:
/bin/sh: -c: line 1: syntax error near unexpected token `('
The fix is for VERBOSE to escape any single quotes embedded in the command line
argument when echoing it:
VERBOSE = $(ECHO) '$(subst ','\'',$(2))'; $(2)
Note that this is still wrong, as it will not do the right thing if $(2) happens
to begin with a hyphen, but I didn't want to introduce a new "PRINTF" variable
(or do something unsavory like calling cat with a here-doc) to squash a bug that
currently has no known manifestations.
Changelog-None
Unfortunately a spec typo means the data fields are missing (PR pending),
so we still patch those in.
The message "your_peer_storage" got renamed to "peer_storage_retrieval",
and the option "want_peer_backup_storage" was removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `experimental-peer-storage` now only advertizes feature 43, not 41.
We now have to explicitly enable various deprecated commando commands, and now
when deprecations are disabled, we honour missing MPP option in bolt12 invoices.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
