Commit Graph

17287 Commits

Author SHA1 Message Date
Rusty Russell
cef11ea361 lightningd: fix db constraint error when fixing up old blocks.
It's very hard to reproduce, since we only consider UTXOs we are watching, but scanning
from the first block we know about is wrong, because we don't care about blocks which
we only put in the db in response to old gossip queries.

I'm not sure how Sjors got into the state where they see their own UTXO spend in a block
they don't have in the database, but we shouldn't crash:

```
2025-12-07T11:25:18.163Z **BROKEN** lightningd: Error executing statement: wallet/wallet.c:4913: UPDATE outputs SET spend_height = ?,  status = ? WHERE prev_out_tx = ? AND prev_out_index = ?: FOREIGN KEY constraint failed
2025-12-07T11:25:18.163Z **BROKEN** lightningd: Error executing statement: wallet/wallet.c:4913: UPDATE outputs SET spend_height = ?,  status = ? WHERE prev_out_tx = ? AND prev_out_index = ?: FOREIGN KEY constraint failed
2025-12-07T11:25:18.179Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v25.12)
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (send_backtrace) 0x5e263feb9292
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: common/daemon.c:83 (crashdump) 0x5e263feb92e1
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x70ee6f64532f
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x70ee6f69eb2c
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x70ee6f64527d
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x70ee6f6288fe
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/log.c:1054 (fatal_vfmt) 0x5e263fe38e10
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: wallet/db.c:95 (db_error) 0x5e263fe6f7ef
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: db/utils.c:326 (db_fatal) 0x5e263feaa797
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: db/utils.c:200 (db_exec_prepared_v2) 0x5e263feaab85
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: wallet/wallet.c:4924 (wallet_outpoint_spend) 0x5e263fe7de2f
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:929 (topo_update_spends) 0x5e263fe0af81
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:1532 (fixup_scan_block) 0x5e263fe0c268
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/bitcoind.c:503 (getrawblockbyheight_callback) 0x5e263fe09280
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:701 (plugin_response_handle) 0x5e263fe5bf2b
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:790 (plugin_read_json) 0x5e263fe610e7
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x5e263fef3629
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x5e263fef3afa
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x5e263fef3bb7
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:470 (io_loop) 0x5e263fef55c5
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x5e263fe308e9
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1492 (main) 0x5e263fe363ce
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x70ee6f62a1c9
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x70ee6f62a28a
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x5e263fe06bf4
2025-12-07T11:25:18.180Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff
```

Changelog-Fixes: lightningd: crash on fixup scan with old blocks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-12-11 12:16:42 +10:30
Rusty Russell
22f7a620e6 lightningd: fix crash on fixup scan if block unavailable.
```
lightningd: FATAL SIGNAL 11 (version v25.12rc3-1-g498c5b6)
0x5cc2f620ce4c send_backtrace
        common/daemon.c:38
0x5cc2f620cee8 crashdump
        common/daemon.c:83
0x7e3ac1e4532f ???
        ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x5cc2f615f186 fixup_scan_block
        lightningd/chaintopology.c:1531
0x5cc2f615c22c getrawblockbyheight_callback
        lightningd/bitcoind.c:484
0x5cc2f61aee87 plugin_response_handle
        lightningd/plugin.c:701
0x5cc2f61b4043 plugin_read_json
        lightningd/plugin.c:790
0x5cc2f6248d8b next_plan
        ccan/ccan/io/io.c:60
0x5cc2f624925c do_plan
        ccan/ccan/io/io.c:422
0x5cc2f6249319 io_ready
        ccan/ccan/io/io.c:439
0x5cc2f624ad24 io_loop
        ccan/ccan/io/poll.c:470
0x5cc2f618381a io_loop_with_timers
        lightningd/io_loop_with_timers.c:22
0x5cc2f61892ff main
```

This happens intermittantly on in a few tests:

tests/test_invoices.py::test_invoice_botched_migration
tests/test_pay.py::test_pay_bolt11_metadata
tests/test_runes.py::test_id_migration

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: lightningd: potential crash on startup if bitcoind isn't up-to-date.
2025-12-10 14:43:45 +10:30
Madeline Paech
b270d915d3 updated pull request for next cln release timeline 2025-12-09 19:11:04 +10:30
Lakshya Singh
62300da7b8 chore: remove rustfmt check in configure
rustfmt isn’t needed for production builds and cargo is sufficient

[ Also, rustfmt --version exits with status 1 due to deprecation --RR ]
Changelog-None
2025-12-09 13:50:16 +10:30
Christian Decker
686ce98906 git: Remove generated binary 2025-12-08 16:37:02 +01:00
Christian Decker
16735c95b7 chore: Downgrade tool diff 2025-12-08 16:37:02 +01:00
Christian Decker
8db09f6d11 Add GitHub Pages documentation site with orchestrated workflows
This commit creates a comprehensive documentation publishing system that
combines coverage reports, Python API docs, and project documentation into
a unified GitHub Pages site.

Changes:
- Update coverage-nightly.yaml to support workflow_call trigger
- Update python-docs-nightly.yaml to support workflow_call trigger
- Add docs-nightly.yaml workflow for project documentation
- Add publish-site.yaml orchestrator workflow

The publish-site workflow:
- Triggers all three documentation workflows in parallel
- Collects artifacts from each workflow
- Organizes them into a unified site structure:
  - / (root) - Beautiful landing page with navigation
  - /docs/ - Project documentation
  - /python/ - Python API reference (pdoc3)
  - /coverage/ - Code coverage reports
- Deploys to GitHub Pages with proper permissions
- Runs nightly at 5 AM UTC, after all other workflows complete

Each workflow can be:
- Triggered manually via workflow_dispatch
- Called from other workflows via workflow_call
- Run on schedule (coverage: 2 AM, python-docs: 3 AM, docs: 4 AM)

The site includes:
- Modern, responsive landing page with gradient design
- Navigation cards for each documentation section
- 404 error page
- .nojekyll file to prevent Jekyll processing
- Automatic timestamp updates

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Changelog-None
2025-12-08 16:37:02 +01:00
Christian Decker
5c9d3884bb Add Python API documentation generation with pdoc3
This commit adds automated Python API documentation generation for all
workspace packages using pdoc3:

- Add contrib/api/generate-python-docs.py script to generate docs
- Add Makefile targets: python-docs and python-docs-clean
- Add GitHub Actions workflow for nightly documentation generation
- Documents 5 packages: pyln.client, pyln.proto, pyln.grpc, pyln.testing, pyln.spec.bolt7
- Creates beautiful index page with cards linking to each package
- Stores generated docs as artifacts with 90-day retention
- Add pdoc3 and markdown to dev dependencies

Bug fix:
- Fix pyln-client version.py: __all__ must contain strings, not class objects
  This was causing "TypeError: attribute name must be string, not 'type'" in pdoc3

Documentation is generated to docs/python/ which is excluded from version control.
Run 'make python-docs' to generate locally, or download from nightly workflow artifacts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-08 16:37:02 +01:00
Christian Decker
4b9cffe183 Add comprehensive coverage infrastructure with clang source-based coverage
This commit introduces a modern coverage infrastructure for Core Lightning:

- Migrate from ad-hoc coverage script to integrated Makefile targets
- Add LLVM source-based coverage support with per-test profraw organization
- Integrate coverage collection into pytest framework via TailableProc
- Add GitHub Actions workflow for nightly coverage reports
- Add Taskfile.yml for convenient task automation
- Add codecov.yml for Codecov integration
- Add comprehensive coverage documentation in COVERAGE.md
- Update contributor workflow docs with new coverage script path
- Add coverage data files to .gitignore (*.profraw, *.profdata)
- Remove obsolete contrib/clang-coverage-report.sh
- Remove obsolete tests/conftest.py (now using pyln-testing markers)
- Update pyproject.toml to include pyln-testing in main dependencies

The new infrastructure automatically collects coverage data when CLN_COVERAGE_DIR
is set, organizing profraw files by test name for granular analysis.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-08 16:37:02 +01:00
Matt Whitlock
7a1a363330 test_renepay.py: remove data dump into /tmp
Changelog-None
2025-12-08 17:47:26 +10:30
Christian Decker
5c0827e417 ci: Add 2 shards to the UBSAN / ASAN runs 2025-12-06 12:32:38 +01:00
Christian Decker
2a2def32ab ci: Remove nix check from the critical path in ci.yaml 2025-12-06 12:32:38 +01:00
Christian Decker
5c749fe4ee ci: Mark some tests as rerun 2025-12-06 12:32:38 +01:00
Christian Decker
fa31e6e4c2 git: Ignore two more binaries 2025-12-06 12:32:38 +01:00
Christian Decker
7c28fdb689 ci: Run with rerunfailures rather than flaky 2025-12-06 12:32:38 +01:00
Christian Decker
287abfbd90 ci: Add a simple plugin to report test results to our falkiness tracker
Changelog-None
2025-12-06 12:32:38 +01:00
ShahanaFarooqui
d18efbb0b7 doc: Add instructions to run CLN node with remote_hsmd_socket 2025-12-05 10:25:19 -08:00
ShahanaFarooqui
a5268e5780 ci: Add new version-vls tag for Docker image
Changelog-Added: Added a new `version-vls` tag with includes  VLS's `remote_hsmd_socket` binary with the Core Lightning image.
2025-12-05 10:25:19 -08:00
ShahanaFarooqui
be471e598f docker: Add vls remote_hsmd_socket binary
This Dockerfile builds the VLS binaries during the builder stage and copies only the `remote_hsmd_socket` binary into the `lightningd-vls-signer` target.
2025-12-05 10:25:19 -08:00
ShahanaFarooqui
c03288cff7 doc: Update release instructions 2025-12-04 16:29:39 -08:00
ShahanaFarooqui
bc64d92b23 tools: Capture Shasums for current version only
Fixes #8753.

Changelog-None: Release process fix.
2025-12-04 16:29:39 -08:00
Madeline Paech
ed439772a8 release branch for 25.12 release 2025-12-04 10:44:56 +10:30
ShahanaFarooqui
af5d02a9fa gitignore: Ignore lightning-downgrade and check-bolt tools
Changelog-None.
2025-12-03 06:24:04 -08:00
Rusty Russell
e02f76bdd8 CHANGELOG: update for 25.12rc3
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-29 10:32:47 +10:30
Dusty Daemon
377802ce9c contrib: Update startup_regtest for taproot
Wallet address default to taproot now — update the startup_regtest.sh script to use taproot addresses.

Changelog-None
2025-11-29 08:35:49 +10:30
Matt Whitlock
6db80536cc tests/test_misc.py: fix another "AF_UNIX path too long"
Also, avoid resource leak by explicitly closing socket.

Changelog-None
2025-11-29 08:35:09 +10:30
wqxoxo
3c08a8bdb9 xpay: fix misleading "route hint" label in error messages
describe_scidd() was labeling any channel as "the invoice's route hint"
when there was a single-hop route hint, even if the error occurred on
an unrelated intermediate channel.

Now we check the channel actually matches the route hint's
short_channel_id before using that label.

Fixes: #8252
Changelog-Fixed: xpay: error messages no longer incorrectly label intermediate channels as "the invoice's route hint".
2025-11-29 08:33:37 +10:30
Rusty Russell
9d4a150b7f pytest: add test that we correctly scan back for missed UTXOs.
I took the node from the original test which missed the UTXO, and
added some more blocks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-29 08:18:20 +10:30
Rusty Russell
6c60375651 lightningd: scan back to seek missing UTXOs.
We only do this once, and not on new nodes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-29 08:18:20 +10:30
Rusty Russell
1e66414b9e wallet: make sure we re-watch outpoints after blocks are rolled back.
At startup, we load the outpoints to watch, *then* roll back 15
blocks.  If there were things in those blocks we wanted to watch, we
no longer do!

1. We load the utxoset into memory: everything in the utxoset table
   which has spendheight null.
2. We roll back 15 blocks to re-read.  Deleting a block from the
   database causes the utxo spentheights referring to it to be set
   to null.
3. We roll forward, but we didn't update the in-memory utxoset,
   so we're not watching those utxos which are spent.

The main symptom of this is that we spam peers with obsolete gossip
(if we get sent a channel announcement for a closed channel, we can
think it isn't spent yet).  But it could *also* mean we don't notice
onchain txs, if we restart at the wrong time!

Changelog-Fixed: lightningd: we could miss tx spends which happened in the past blocks when we restarted.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-29 08:18:20 +10:30
Rusty Russell
8caffde0fa pytest: test for watching utxos over restart reorg.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-29 08:18:20 +10:30
ShahanaFarooqui
534e3ae8f8 ci: Remove lowdown and configure from the sign release job step
Changelog-None: Fixes draft release job creation in CI.
2025-11-28 20:27:02 +10:30
Rusty Russell
7e9ec3ef2d tools: remove accidentally-committed obsolete hsmtool binary.
And keep it in the .gitignore, as others may have old ones lying around too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 15:45:06 +10:30
Madeline Paech
c59d9beab3 second release candidate for 25.12 2025-11-28 10:48:43 +10:30
Rusty Russell
385b6a9e2f doc: generate documentation for askrene-bias-node.
I noticed it was missing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
640888aa91 CI: add test that we can downgrade the node and run it under v25.09.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
39a92116a8 doc: add a manual page.
In particular, document when downgrades are not possible.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
4d8f1d33e9 pytest: add failing downgrade test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
a45c907200 lightningd-downgrade: downgrade askrene's layer info too.
If they had a channel bias, and ran xpay, it will update the bias
to a v2 bias (with a timestamp).  We must downgrade that, or the
older version won't load!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
eb83a205b9 askrene: move datastore wire functions into separate file for lightning-downgrade to use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
2a066b92fc wallet: separate datastore access functions for the lightning-downgrade tool to access.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
9981e238a3 tools/lightning-downgrade: tool to downgrade (offline) v25.12 to v25.09.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: tools: `lightningd-downgrade` can downgrade your database from v25.12 to v25.09 if something goes wrong.
2025-11-28 09:16:53 +10:30
Rusty Russell
76ad32ec88 check-bolt: move to devtools, clean up tools/
And leave $(TOOLS) as user-visible tools.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
23f2141683 global: use lightning-hsmtool not hsmtool.
When installed, the name is `lightning-hsmtool`.  We actually copy
`tools/hsmtool` to `tools/lightning-hsmtool` but that's a silly step
which we should get rid of.

So:
1. Make sure our documentation always refers to it as lightning-hsmtool.
2. Make sure our tests invoke it as `lightning-hsmtool`.
3. Rename the C file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
6dcf4e7bdd wallet: separate migration table into its own source file.
This will make it easier to share with the downgrade tool.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
378293d268 wallet: add optional sql and functions to downgrade db.
This will allow a downgrade tool to attempt to downgrade a node to the
previous version.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
f052408832 wire: add explicit-length fromwire_peektype variant.
Useful for continuous streams.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-28 09:16:53 +10:30
Rusty Russell
fc23b538cd lightningd: print UNUSUAL log message if a command is very slow.
We don't expect an internal command to take 5 seconds to service
without explicitly pausing: if it does, log at a higher level.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-27 15:04:20 +10:30
Rusty Russell
acc41ddc0c connectd: don't log at INFO level for known issue.
We get spammed by this, because we somehow missed occasional channel closes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-11-27 15:04:20 +10:30
ShahanaFarooqui
ff0ee6dfa0 doc: Fix mdx format errors generated due to Readme v2 migrations
Changelog-None: Documentation fixes only.
2025-11-27 14:10:18 +10:30