Commit Graph

2084 Commits

Author SHA1 Message Date
Rusty Russell
343d30ae5b bookkeeper: fix reloading of rebalances.
Parse key correctly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: bookkeeper: failed reload of rebalances on restart.
2025-10-01 15:35:50 +09:30
Rusty Russell
ba75f7bcf1 offers: remove blinding from decode JSON-RPC.
Deprecated in 24.11, disabled in 25.05.

Changelog-Removed: JSON-RPC: `decode` field `blinding` (use `first_path_key` as per modern BOLT naming)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:32:54 +09:30
Rusty Russell
512f328730 wire: update to slight spec neating.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
5e53af7d3f plugins/offers: handle invoice_request with invreq_recurrence_cancel
In this case, we make an immediately-expiring invoice.  This correctly blocks
any successive requests for invoices, as per the spec requirement.

This means we have to handle invoice_requests without reply_path, amounts
or quantity *if* they specify invreq_recurrence_cancel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
0d18b82dae plugins: cancelrecurringinvoice command.
`fetchinvoice` variant, for setting invreq_recurrence_cancel instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `cancelrecurringinvoice` command to send new "don't expect any more invoice requests" msg to recurring bolt12 invoices.
2025-10-01 15:31:30 +09:30
Rusty Russell
5a73bd34e5 plugins/fetchinvoice: allow send_message() to say "don't expect a reply"
We're going to want this for sending invreq_recurrence_cancel invoice_requests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
4e7c2fcf32 plugins/fetchinvoice: extract bip353 parsing to param helper.
We're going to want this for cancelrecurringinvoice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
551ecdf7ab plugins/fetchinvoice: extract recurrence invreq_metadata routine.
We're going to want this for cancelrecurringinvoice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
b30e063432 plugins/fetchinvoice: allow use of expired offers *for recurrence*.
We added this to the recurrence spec: the offer expiration only applies to the
first request, not subsequent ones.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
559e3fc447 common: fix bolt12 quotes to bring them up-to-date.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
75e6c49989 offers: allow quantity_max = 1
If you're doing just-in-time stock management, it would be annoying to
discover this wasn't allowed!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
6476347c79 offers: use param_check() for more thorough parameter validation.
And fix incorrect comment on its use in the header!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 15:31:30 +09:30
Rusty Russell
54444e4337 BOLT12: Update recurrence to latest spec draft.
Changes:
* Fields renumbered to their draft values + billion.
* offer_recurrence now comes in compulsory or optional (backwards compat) flavors.
* `proportional_amount` is now inside `offer_recurrence_base` not `offer_recurrence_paywindow`.
* New field `invreq_recurrence_cancel`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Draft specification for recurring offers changed: old recurring offers will no longer work.
2025-10-01 15:31:30 +09:30
Rusty Russell
e7ea57e130 BOLT12: Remove start_any_period from recurrence_base.
Offer_absolute_expiry should be used if you want to require starting at the start.

Changelog-EXPERIMENTAL: Protocol: BOLT 12 recurrence `start_any_period` removed, use expiry if you need to restrict when they can start using the offer.
2025-10-01 15:31:30 +09:30
Rusty Russell
05005475e3 BOLT12: Remove years from recurrence.
Use months instead.

Changelog-EXPERIMENTAL: Protocol: BOLT 12 recurrence `years` removed; use 12 x months.
2025-10-01 15:31:30 +09:30
Rusty Russell
907f7c6698 unit tests: update all the gossmaps to have the GOSSIP_STORE_COMPLETED_BIT set.
Mostly this meant running them, then running devtools/convert-gossmap and replacing the code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-10-01 13:29:33 +09:30
Rusty Russell
3c3788a241 pytest: fix test_xpay_fake_channeld flake
Also, resulting log was huge, so suppress log level (will probably speed test)

```
2025-09-02T06:01:39.4881086Z >       l1.rpc.plugin_start(os.path.join(os.getcwd(), 'plugins/cln-askrene'))
2025-09-02T06:01:39.4881090Z 
2025-09-02T06:01:39.4881174Z tests/test_xpay.py:279: 
...
2025-09-02T06:01:39.4883193Z >           self.sock.connect(str(self.path))
2025-09-02T06:01:39.4883340Z E           ConnectionRefusedError: [Errno 111] Connection refused
...
2025-09-02T06:01:41.7767610Z lightningd-1 2025-09-02T06:01:27.235Z **BROKEN** plugin-cln-xpay: askrene-age failed with {"code":-4, "message":"Plugin terminated before replying to RPC call."}
2025-09-02T06:01:41.7768127Z lightningd-1 2025-09-02T06:01:27.305Z INFO    plugin-cln-xpay: Killing plugin: exited during normal operation
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-09-30 11:37:31 +09:30
Rusty Russell
40a44d9aab build: fix build with no sqlite3 support.
Reported-by: whitslack
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8473
Changelog-Fixed: build: we can now build without sqlite3 support.
2025-09-29 13:11:44 +09:30
Claudio Raimondi
13c5db51a9 Refactor Dockerfile, Add TODOs
all these changelogs only apply to the Docker image.

Changelog-Added: added verification of GPG keys for the bitcoin and litecoin tarballs.
Changelog-Fixed: fixed compilation on all target architectures; each had their own bugs (poetry, missing packages...).
Changelog-Fixed: fixed cargo cross compilation. it was mistakenly using QEMU before.
Changelog-Fixed: fixed CPU compatibility bug described in issue 8456
Changelog-Changed: improve build time by 8.8x
Changelog-Changed: improve image size by 2.07x

more detailed changelog can be found on the PR: https://github.com/ElementsProject/lightning/pull/8429
2025-09-28 20:56:07 +05:30
Chandra Pratap
a77aa50ae1 plugins/libplugin-pay: Add a check for NaN values
Changelog-Fixed: Due to the imprecision of `htlc_max`'s type (`fp16_t`),
`capacity_bias()` can return `NaN` in some cases. This leads to a
runtime error when compiled with UBSan. Add a check against it.
2025-09-22 10:52:59 +09:30
Chandra Pratap
5365b44e1b plugins/test: Add a test to trigger the bug
Add a test in `plugins/test/run-route-calc.c` that reproduces the
runtime error when the fix is not applied.
2025-09-22 10:52:59 +09:30
ShahanaFarooqui
9f75c99266 docker: Dockerfile fixes after poetry to uv migration
Cargo utilizes `git ls-remote` to resolve git dependencies specified by commit hashes. GitHub only advertises commits that are reachable from branches, tags, or PR references. The `bip353-plugin` was referencing an orphaned commit in the `bitcoin-payment-instructions` dependency that was unreachable through any advertised reference. This can be resolved by installing the tarball release v0.5.0.

Changelog-None.
2025-09-09 10:46:23 +09:30
Nishant Bansal
74489b52b3 plugins/bcli: use -rpcwait to simplify waiting for bitcoind to warm up
Replaced custom wait logic with the -rpcwait flag in bitcoin-cli to handle waiting for bitcoind to warm up. This simplifies the code and ensures that errors unrelated to warmup are passed up directly without additional checks.
Changelog-None

Signed-off-by: Nishant Bansal <nishant.bansal.282003@gmail.com>
2025-09-04 13:33:32 +09:30
Rusty Russell
3e4c922114 bkpr: don't leak event in fillin_apy_acct_details.
Now accounts are not transient, we can't use them as temporary parents:

```
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper: MEMLEAK: 0x5e3bf926ce28
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:   label=plugins/bkpr/sql.c:109:struct chain_event
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:   alloc:
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     ccan/ccan/tal/tal.c:488 (tal_alloc_)
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/sql.c:109 (chain_events)
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/sql.c:197 (chain_events_from_sql)
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/recorder.c:352 (find_chain_event_by_id)
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/channelsapy.c:120 (fillin_apy_acct_details)
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/channelsapy.c:172 (compute_channel_apys)
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:220 (getblockheight_done)
lightningd-2 2025-08-28T02:36:35.629Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1134 (handle_rpc_reply)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1438 (rpc_read_response_one)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1462 (rpc_conn_read_response)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:60 (next_plan)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:422 (do_plan)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:439 (io_ready)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/poll.c:455 (io_loop)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:2564 (plugin_main)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1547 (main)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ../csu/libc-start.c:360 (__libc_start_main_impl)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:   steal:
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/tal/tal.c:559 (tal_steal_)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/recorder.c:361 (find_chain_event_by_id)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/channelsapy.c:120 (fillin_apy_acct_details)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/channelsapy.c:172 (compute_channel_apys)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:220 (getblockheight_done)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1134 (handle_rpc_reply)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1438 (rpc_read_response_one)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1462 (rpc_conn_read_response)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:60 (next_plan)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:422 (do_plan)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:439 (io_ready)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/poll.c:455 (io_loop)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:2564 (plugin_main)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1547 (main)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     ../csu/libc-start.c:360 (__libc_start_main_impl)
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:   parents:
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/account.c:51:struct account
lightningd-2 2025-08-28T02:36:35.630Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1546:struct bkpr
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-28 13:42:04 +09:30
Rusty Russell
22a5e9e7e6 common: reintroduce "ignored" primary tag.
Prior to 23.05, we used this tag to mark onchain to-self inputs we didn't
wait for (because they were too small).  This fixes migration if that happened
(and we are debating whether we should re-introduce this!).

```
lightningd: FATAL SIGNAL 6 (version v25.09rc2)                      
0x100c8683 send_backtrace                                           
        common/daemon.c:33                                          
0x100c876f crashdump                                                
        common/daemon.c:78                                          
0x7fffb2080493 ???                                                  
        ???:0                                                       
0x7fffb1ab0cac ???                                                  
        __pthread_kill_implementation+0x1bc:0
0x7fffb1a48a5b ???                                                  
        __GI_raise+0x2b:0                                           
0x7fffb1a2a3db ???                                                  
        __GI_abort+0x153:0                                          
0x100935b7 migrate_from_account_db
        wallet/account_migration.c:424
0x10093ff7 db_migrate                                               
        wallet/db.c:1139                                            
0x10096763 db_setup                                                 
        wallet/db.c:1185                                            
0x100a1bcb wallet_new                                               
        wallet/wallet.c:223                                         
0x1004485f main                                                     
        lightningd/lightningd.c:1311
0x7fffb1a2aba3 ???                                                  
        __libc_start_call_main+0x93:0
0x7fffb1a2adeb ???                                                  
        __libc_start_main_alias_1+0x1ab:0
0xffffffffffffffff ???                                              
        ???:0                                                       
lightningd: Died with signal 6                                  
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8484
2025-08-28 13:41:44 +09:30
Rusty Russell
1c537c258e pay: fix uninitialized var in debug output.
@nepet noted that Valgrind complained.  Nobody really cares though?
TL;DR: if channel isn't enabled, estimate isn't set.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: CI only
2025-08-27 12:53:38 +09:30
Rusty Russell
afa7aead24 offers: loosen payment_constraints on invoices' blinded paths.
In practice, we were too strict.  Here's Phoenix paying my node via another node:

```
2025-08-25T13:57:53.311Z DEBUG   02...-chan#216: Failing HTLC because of an invalid payload (TLV 10 pos 103): cltv_expiry 911816 > payment_constraint 911721
```

We add 6 blocks, but this is supposed to be the *max* allowed.  Increase it to 1008, to allow shadow padding.  Here are the CLTV delays across advertized channels in the network: most are far less than this:

Count   Delay
      1 0
      1 4
      3 12
    899 18
    602 20
      1 22
      3 24
      1 25
      9 26
      5 29
     72 30
     54 32
   4352 34
      2 35
      6 36
     10 37
     27 38
      8 39
  15256 40
     94 42
    102 44
     13 45
     20 46
     87 48
    230 50
    100 51
     14 55
    316 60
     29 64
     30 65
     24 68
     82 69
      9 70
    779 72
     13 77
    243 78
      1 79
  26793 80
      1 82
      6 83
      1 84
     18 85
      1 86
      9 87
     16 88
    101 89
      2 90
     11 96
    113 99
   9527 100
     41 112
     34 118
    132 119
    403 120
     24 128
    232 140
    138 142
  14303 144
      2 145
     41 150
      6 160
      3 172
      8 174
     27 180
      4 190
    297 200
     99 210
     34 216
    219 222
     15 240
    105 288
     17 300
      7 336
      1 360
     19 400
     24 420
     26 432
     20 450
      1 480
     12 500
      1 720
      1 850
      1 1000
      1 1002
      1 1144
      1 1192
      5 1201
      1 1444
      1 1795
      1 1900
      1 2016

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Offers on nodes with only private channels are now payable (i.e. no more blinded path errors!).
Fixes: https://github.com/ElementsProject/lightning/issues/7718
2025-08-27 12:09:20 +09:30
Matt Whitlock
d635f19dbf plugins: generate certificates with required extensions
Recent versions of urllib3 fail certificate verification if certificates
lack the Authority Key Identifier or Key Usages extensions:

```
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA cert does not include key usage extension (_ssl.c:1032)
```

Luckily, rcgen offers parameters in its CertificateParams structure to
add these extensions. Let's use them.

Changelog-Fixed: Certificates auto-generated by grpc-plugin, rest-plugin, and wss-proxy-plugin now include the required Authority Key Identifier and Key Usages extensions.
2025-08-26 13:52:15 +09:30
Christian Decker
516861c5ec rust: Bump cln-plugin==0.5
Due to the breaking change in 9dcc264d4c8e244f0cb827c8a81e61f977011486.
2025-08-19 15:47:58 +09:30
Christian Decker
45ba719068 rust: Roll the cln-grpc and cln-rpc versions due to schema changes
Commit ebaa25d9e2fd5582b7fe0e3ec482c1627996e4d7 introduced a couple of
breaking changes to the schema and proto files. The bump ensures
backwards compat for users that have indicated `~0.4` as their version
constraint.

Changelog-Changed: rust: New version of `cln-rpc==0.5` and `cln-grpc==0.5`
2025-08-19 15:47:58 +09:30
Rusty Russell
66741dda79 plugins/bkpr/test/run-recorder: don't hand NULL cmd.
ubsan complains that we declared a function not to take NULL.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
f8a44d911d bkpr: restore run-recorder.
This requires us to turn "sql" calls into calls to a local db, which
means pulling in a lot of infrastructure.  But it's possible.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
194febe873 wallet: generate fixup chainmoves and channelmoves when first starting.
If we don't have an accountdb from bookkeeper:

1. Generate a deposit chain event for every confirmed UTXO.
2. Generate an open chain event for every open, confirmed channel.
3. Generate a push/lease event if necessary.
4. Generate a fixup "journal" entry if balance is different from initial.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
52d69df10f lightningd: migrate events from bookkeeper at startup.
We take over the --bookkeeper-dir and --bookkeeper-db options, and
then if we can find the bookkeeper db we extract the records to
initialize our chain_moves and channel_moves tables.

Of course, bookkeeper now needs to not register those options.

When bookkeeper gets invoked the first time, it will reconstruct
everything from listchannelmoves and listcoinmoves.  It cannot
preserve manually-added descriptions, so we put those in the datastore
for it ready to go.

Note that the order of onchain_fee changes slightly from the original.
But this is fine.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
41ac9db7ea bookkeeper: remove database.
Changelog-Changed: Plugins: `bookkeeper` now uses the lightningd database, not "accounts.db".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
8748755834 bkpr: remove missing event handling.
There will be no more missing events (and at initialization time, we will do
that as a migration).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
b70f4f6184 bkpr: convert all the local db sql queries into calls to sql plugin.
With some help (and hinderance!) from ChatGPT: the field names
differ slightly from our internal db.

The particilar wrinkle is that we have to restrict all queries to
limit them to entries we've seen already.  Our code expects this (we
used to only enter it into the db when we processed it), and it would
otherwise be confusing if a sql query returned inconsistent results
because an event occurred while bookkeeper was processing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
2a191479a3 bkpr: take, don't steal in new_channel_event.
Cleaner (I'm about to hand it a sha256 on the stack).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
45e860ad58 bkpr: helpers to query sql plugin for chainmoves and channelmoves.
We're going to be using this instead of our internal db.

I also made json_out_obj() take the str arg, as it didn't and I
expected it to.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
70d19e852e bkpr: use list commands instead of subscribing to notifications.
This is reliable, meaning we should never get replayed events.

We have to reference count to make sure all commands are complete,
before we return.  In particular, annotating with descriptions can
involve several calls to list commands.  We need to give them the
results *after* this is all complete.

test_bookkeeping_descriptions() relied on log messages from
notifications, which now only happen when a command is called.  This
changes the test a bit.

Since we no longer subscribe to the balance_snapshot event, we
need to create the wallet account at initialization, as callers
expect it to exist.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
f50ceb4ce6 bookkeeper: call refresh before processing any commands.
Rearrange all the JSON interfaces to call refresh_moves() (async)
before doing anything.

This does nothing for now, but it will be useful once we transition
from notifications to using the list commands.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
9cc08bf06d plugins/bkpr/test/run-recorder: remove.
It's a great test, but it's very hard to simulate now we are going to be
going from the internal db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
bb6787df06 libplugin: allow sync interface to be used at all times.
After much thought and mis-steps, I chose a simple solution: open another fd
for sync comms.  It's almost impossible to know what state the async one is in.

jsonrpc_request_sync() is enhanced to return a valid tal object, as the current
behaviour of returning a pointer to inside an array was surprising.

Changelog-Changed: libplugin: you can now call the synchronous API functions at any time (not just in the init callback).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
1f7905259b bkpr: forward utxo_deposit / utxo_spend notifications to new injectutxodeposit / injectutxospend calls.
And thus we absorb them as normal when they come back as "foreign" entries.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
980241fc36 bookkeeper: don't set origin account to "null" if that's specified as transfer_from.
Before bkpr_listaccountevents() gave entries with origin like:

	{'account': "nifty's secret stash",
         'blockheight': 111,
         'credit_msat': 180000000,
         'currency': 'bcrt',
         'debit_msat': 0,
         'origin': 'null',
         'outpoint': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:0',
         'tag': 'deposit',
         'timestamp': 1679955976,
         'type': 'chain'},

Changelog-Changed: Plugins: "utxo_deposit" is allows to have missing `transfer_from`, and null is not considered an account name.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
78f88c1c0c common: use "foreign" in extra_tags to indicate a chain movement is injected.
This allows the bookkeeper plugin to know it's not actually a channel account.

Remove the "ignored" tag from the schema too: we removed it previously.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
254b7ed666 lightningd: new internal-use commands to allow utxo spend / deposit injection.
For the moment, we'll continue to use bookkeeper to monitor the
notifications to insert these (we don't have the internal infrastructure
for that, and actually these commands are probably better than using
notifications).

We hoist param_outpoint() into common code, since there are already
two uses.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
c5e359c7ee bkpr: add in-mem & datastore storage for external blockheights.
We won't be able to "UPDATE chain_events", so keep a separate record
of these blockheights, and lookup that when the blockheight is 0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
e28443ed40 bkpr: pass bkpr struct into all functions in recorder.c.
We want to access it in stmt2chain_event, so plumb it through.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30
Rusty Russell
0afc4c0346 bookkeeper: use the in-memory rebalances.
Remove the rebalance field from channel_event, and use the
find_rebalance(bkpr, ev->db_id) to look it up instead.

chain_event's also had a `rebalance` field, but it was only ever set
(to false), never read.

Note: list_rebalances() was only used by tests, not a public API.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2025-08-19 13:37:50 +09:30