In November 2022 we seemed to increase parallelism from 2 and 3 to 10!
That is a huge load for these CI boxes, and does explain some of our
flakes.
We only run in parallel because some tests sleep, but it's diminishing
returns (GH runners have 4 VCPUs, 16GB RAM).
This reduces it so:
- Normal runs are -n 4
- Valgrind runs are -n 2
- Sanitizer runs are -n 3
If I use my beefy build box (64BG RAM) but reduce it to 4 CPUs:
Time for pytest -n 5:
Time for pytest -n 4:
Time for pytest -n 3:
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means:
1. downgrade changes (we no longer fail due to node biases).
2. various deprecations no longer are
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3974806e5a added this:
CI: Try not running group 2/10 UBSAN in parallel.
It's being killed with signal 143, which means docker isn't happy; too much memory consumption?
But since we're now at 12 groups, that probably doesn't apply (it
might not have even before, in the two years since that commit since
so may things have been added). And it caused this shard to take over
2 hours and timed out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- ReadMe API v2 now requires category.uri instead of category.id.
- ReadMe v2 page responses include a renderable field indicating whether the document is MDX-compatible and can be rendered successfully. The script now checks this field and prints detailed compilation errors if rendering fails.
Add a GROUP matrix variable to valgrind and sanitizer test configurations
with simple numeric values. Use matrix.GROUP in artifact names instead of
sanitizing the descriptive NAME, which contains special characters.
This is cleaner and more maintainable than string replacement logic.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The valgrind and sanitizer test matrix names contain parentheses and slashes
(e.g., 'Valgrind (01/10)', 'ASan/UBSan (01/12)') which are invalid in artifact
filenames. Replace these characters with hyphens to ensure reliable artifact uploads.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Adds upload-artifact steps after each test run to preserve test
results as GitHub artifacts. Each job uploads its report.xml with
a unique name based on the job and matrix configuration.
Artifacts are uploaded even when tests fail (if: always()) to
ensure test results are available for debugging.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replaces GLOBAL_PYTEST_OPTS with PYTEST_OPTS_BASE that includes all
common pytest options: --reruns=10, -vvv, --junit-xml=report.xml,
--timeout=1800, and --durations=10.
Removes redundant option specifications from individual jobs and
simplifies pytest command invocations by removing direct references
to GLOBAL_PYTEST_OPTS (now inherited via PYTEST_OPTS).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changelog-None
This commit creates a comprehensive documentation publishing system that
combines coverage reports, Python API docs, and project documentation into
a unified GitHub Pages site.
Changes:
- Update coverage-nightly.yaml to support workflow_call trigger
- Update python-docs-nightly.yaml to support workflow_call trigger
- Add docs-nightly.yaml workflow for project documentation
- Add publish-site.yaml orchestrator workflow
The publish-site workflow:
- Triggers all three documentation workflows in parallel
- Collects artifacts from each workflow
- Organizes them into a unified site structure:
- / (root) - Beautiful landing page with navigation
- /docs/ - Project documentation
- /python/ - Python API reference (pdoc3)
- /coverage/ - Code coverage reports
- Deploys to GitHub Pages with proper permissions
- Runs nightly at 5 AM UTC, after all other workflows complete
Each workflow can be:
- Triggered manually via workflow_dispatch
- Called from other workflows via workflow_call
- Run on schedule (coverage: 2 AM, python-docs: 3 AM, docs: 4 AM)
The site includes:
- Modern, responsive landing page with gradient design
- Navigation cards for each documentation section
- 404 error page
- .nojekyll file to prevent Jekyll processing
- Automatic timestamp updates
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changelog-None
This commit adds automated Python API documentation generation for all
workspace packages using pdoc3:
- Add contrib/api/generate-python-docs.py script to generate docs
- Add Makefile targets: python-docs and python-docs-clean
- Add GitHub Actions workflow for nightly documentation generation
- Documents 5 packages: pyln.client, pyln.proto, pyln.grpc, pyln.testing, pyln.spec.bolt7
- Creates beautiful index page with cards linking to each package
- Stores generated docs as artifacts with 90-day retention
- Add pdoc3 and markdown to dev dependencies
Bug fix:
- Fix pyln-client version.py: __all__ must contain strings, not class objects
This was causing "TypeError: attribute name must be string, not 'type'" in pdoc3
Documentation is generated to docs/python/ which is excluded from version control.
Run 'make python-docs' to generate locally, or download from nightly workflow artifacts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit introduces a modern coverage infrastructure for Core Lightning:
- Migrate from ad-hoc coverage script to integrated Makefile targets
- Add LLVM source-based coverage support with per-test profraw organization
- Integrate coverage collection into pytest framework via TailableProc
- Add GitHub Actions workflow for nightly coverage reports
- Add Taskfile.yml for convenient task automation
- Add codecov.yml for Codecov integration
- Add comprehensive coverage documentation in COVERAGE.md
- Update contributor workflow docs with new coverage script path
- Add coverage data files to .gitignore (*.profraw, *.profdata)
- Remove obsolete contrib/clang-coverage-report.sh
- Remove obsolete tests/conftest.py (now using pyln-testing markers)
- Update pyproject.toml to include pyln-testing in main dependencies
The new infrastructure automatically collects coverage data when CLN_COVERAGE_DIR
is set, organizing profraw files by test name for granular analysis.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
It breaks, but more importantly we don't need to install lowdown any more,
since the check in build-release.sh has been removed.
```
Run sudo apt-get install -y lowdown
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
lowdown
0 upgraded, 1 newly installed, 0 to remove and 21 not upgraded.
Need to get 129 kB of archives.
After this operation, 314 kB of additional disk space will be used.
Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
Get:2 http://azure.archive.ubuntu.com/ubuntu noble/universe amd64 lowdown amd64 1.1.0-1 [129 kB]
Fetched 129 kB in 0s (2971 kB/s)
Selecting previously unselected package lowdown.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 216225 files and directories currently installed.)
Preparing to unpack .../lowdown_1.1.0-1_amd64.deb ...
Unpacking lowdown (1.1.0-1) ...
Setting up lowdown (1.1.0-1) ...
Processing triggers for man-db (2.12.0-4build2) ...
Not building database; man-db/auto-update is not 'true'.
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
checking for getpagesize() in <unistd.h>... yes
checking for isblank() in <ctype.h>... yes
checking for little endian... yes
checking for memmem in <string.h>... yes
checking for memrchr in <string.h>... yes
checking for mmap() declaration... yes
checking for /proc/self/maps exists... yes
checking for qsort_r cmp takes trailing arg... yes
checking for __attribute__((section)) and __start/__stop... yes
checking for stack grows upwards... no
checking for statement expression support... yes
checking for <sys/filio.h>... no
checking for <sys/termios.h>... yes
checking for <sys/unistd.h>... yes
checking for __typeof__ support... yes
checking for unaligned access to int... yes
checking for utime() declaration... yes
checking for __attribute__((warn_unused_result))... yes
checking for #pragma omp and -fopenmp support... yes
checking for <valgrind/memcheck.h>... no
checking for working <ucontext.h... yes
checking for passing pointers via makecontext()... yes
checking for __builtin_cpu_supports()... yes
checking for closefrom() offered by system... yes
checking for F_CLOSEM defined for fctnl.... no
checking for close_range syscall available as __NR_close_range.... yes
checking for F_MAXFD defined for fcntl.... no
checking for zlib support... yes
checking for libsodium with IETF chacha20 variants... no
checking for sqlite3... yes
checking for postgres... yes
checking for User Statically-Defined Tracing (USDT)... no
checking for compiler is GCC... yes
checking for GCC version is 7 or above... yes
Writing variables to config.vars.2200... yes
Writing header to ccan/config.h.2200... yes
checking for python3-mako... not found
checking for lowdown... found
checking for sha256sum... found
checking for jq... found
Setting PREFIX... /usr/local
Setting CC... cc
Setting CONFIGURATOR_CC... cc
Setting CWARNFLAGS... -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -Werror -Wno-maybe-uninitialized -Wshadow=local
Setting CDEBUGFLAGS... -std=gnu11 -g -fstack-protector-strong
Setting COPTFLAGS... -Og
CSANFLAGS not found
FUZZFLAGS not found
FUZZER_LIB not found
LLVM_LDFLAGS not found
SQLITE3_CFLAGS not found
Setting SQLITE3_LDLIBS... -lsqlite3
Setting POSTGRES_INCLUDE... -I/usr/include/postgresql
Setting POSTGRES_LDLIBS... -L/usr/lib/x86_64-linux-gnu -lpq
SODIUM_CFLAGS not found
SODIUM_LDLIBS not found
Setting VALGRIND... 0
Setting DEBUGBUILD... 0
Setting COMPAT... 1
Setting PYTEST... python3 -m pytest
Setting STATIC... 0
Setting CLANG_COVERAGE... 0
Setting ASAN... 0
Setting UBSAN... 0
Setting TEST_NETWORK... regtest
Setting HAVE_PYTHON3_MAKO... 0
Setting SHA256SUM... sha256sum
Setting FUZZING... 0
Setting RUST... 1
Setting PYTHON... python3
Setting SED... sed
*** We need a libsodium >= 1.0.4 (released 2015-06-11).
Error: Process completed with exit code 1.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Switched to nixpkgs-unstable
Replaced postgresql dependency with much smaller libpq.
Utilise new inputs.self.submodules feature to simplify flake use
Moved apps to a separate file
Changelog-None
It takes over an hour to run, because there's no FreeBSD wheel for the cryptography Python
package, and it builds it, in Rust, from scratch.
And eventually, takes over 125 minutes and times out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows us to show what tests are slowest, by showing the duration for anything
which took 10 seconds or longer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We shipped our own because Ubuntu xenial (16.4) had an ancient one.
Changelog-Changed: Build: libsodium version >= 1.0.4 now required (released 2015-06-11)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Every distribution we have packages this now.
Changelog-Changed: Build: lowdown is now required (we no longer bundle our own).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The Publish distribution stage was failing because it executed the update-pyln-versions script from within the WORKDIR, which created an invalid context. To resolve this, we have decoupled the process, separating the updating of version state into its own step that runs from the root directory before the publish operation.
Changelog-None.
Telling users to rename it is a poor idea, and if you have multiple releases in your release/ dir
it will get confusing. So always append -v25.09 and update docs accordingly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Imagine you've burned the midnight oil trying to get your feature into this latest release that was due yesterday. Sweat is dripping off your brow, but you're proud, you're finally ready to push all your changes and merge!
.
.
.
Only for Github Gandalf the Grey to go "YOU SHALL NOT PASS". Why? Because you did not add a "Changelog-None" to any of commit messages. So you have to go back and try add a full stop here or a new line there and wait for the CI overlords to bless your PR. :'(
Well now, you can just add a Changelog-None to your PR description and github hopefully will take mercy on you and run your CI workflow again :)
Fixes conflict error in Release action when trying to fetch the commit and tag both together. Like
```
fatal: Cannot fetch both da7d3057ed and refs/tags/v25.09rc1 to refs/tags/v25.09rc1
```
Changelog-None.
It was breaking a lot, due to Cargo dependencies not being published,
no surprise there, we are just publishing them with the changes it is
complaining about.
