Commit ebaa25d9e2fd5582b7fe0e3ec482c1627996e4d7 introduced a couple of
breaking changes to the schema and proto files. The bump ensures
backwards compat for users that have indicated `~0.4` as their version
constraint.
Changelog-Changed: rust: New version of `cln-rpc==0.5` and `cln-grpc==0.5`
This requires us to turn "sql" calls into calls to a local db, which
means pulling in a lot of infrastructure. But it's possible.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't have an accountdb from bookkeeper:
1. Generate a deposit chain event for every confirmed UTXO.
2. Generate an open chain event for every open, confirmed channel.
3. Generate a push/lease event if necessary.
4. Generate a fixup "journal" entry if balance is different from initial.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We take over the --bookkeeper-dir and --bookkeeper-db options, and
then if we can find the bookkeeper db we extract the records to
initialize our chain_moves and channel_moves tables.
Of course, bookkeeper now needs to not register those options.
When bookkeeper gets invoked the first time, it will reconstruct
everything from listchannelmoves and listcoinmoves. It cannot
preserve manually-added descriptions, so we put those in the datastore
for it ready to go.
Note that the order of onchain_fee changes slightly from the original.
But this is fine.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There will be no more missing events (and at initialization time, we will do
that as a migration).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With some help (and hinderance!) from ChatGPT: the field names
differ slightly from our internal db.
The particilar wrinkle is that we have to restrict all queries to
limit them to entries we've seen already. Our code expects this (we
used to only enter it into the db when we processed it), and it would
otherwise be confusing if a sql query returned inconsistent results
because an event occurred while bookkeeper was processing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to be using this instead of our internal db.
I also made json_out_obj() take the str arg, as it didn't and I
expected it to.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is reliable, meaning we should never get replayed events.
We have to reference count to make sure all commands are complete,
before we return. In particular, annotating with descriptions can
involve several calls to list commands. We need to give them the
results *after* this is all complete.
test_bookkeeping_descriptions() relied on log messages from
notifications, which now only happen when a command is called. This
changes the test a bit.
Since we no longer subscribe to the balance_snapshot event, we
need to create the wallet account at initialization, as callers
expect it to exist.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rearrange all the JSON interfaces to call refresh_moves() (async)
before doing anything.
This does nothing for now, but it will be useful once we transition
from notifications to using the list commands.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a great test, but it's very hard to simulate now we are going to be
going from the internal db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
After much thought and mis-steps, I chose a simple solution: open another fd
for sync comms. It's almost impossible to know what state the async one is in.
jsonrpc_request_sync() is enhanced to return a valid tal object, as the current
behaviour of returning a pointer to inside an array was surprising.
Changelog-Changed: libplugin: you can now call the synchronous API functions at any time (not just in the init callback).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before bkpr_listaccountevents() gave entries with origin like:
{'account': "nifty's secret stash",
'blockheight': 111,
'credit_msat': 180000000,
'currency': 'bcrt',
'debit_msat': 0,
'origin': 'null',
'outpoint': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:0',
'tag': 'deposit',
'timestamp': 1679955976,
'type': 'chain'},
Changelog-Changed: Plugins: "utxo_deposit" is allows to have missing `transfer_from`, and null is not considered an account name.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows the bookkeeper plugin to know it's not actually a channel account.
Remove the "ignored" tag from the schema too: we removed it previously.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For the moment, we'll continue to use bookkeeper to monitor the
notifications to insert these (we don't have the internal infrastructure
for that, and actually these commands are probably better than using
notifications).
We hoist param_outpoint() into common code, since there are already
two uses.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We won't be able to "UPDATE chain_events", so keep a separate record
of these blockheights, and lookup that when the blockheight is 0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Remove the rebalance field from channel_event, and use the
find_rebalance(bkpr, ev->db_id) to look it up instead.
chain_event's also had a `rebalance` field, but it was only ever set
(to false), never read.
Note: list_rebalances() was only used by tests, not a public API.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The new access APIs are more symmetrical:
1. edit_utxo_description -> add_utxo_description
2. add_payment_hash_desc -> add_payment_hash_description
And to read it, instead of accessing ->ev_desc (now removed) we use
chain_event_description() & channel_event_description(), threading bkpr though
as needed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is not quite as efficient, perhaps, but in practice there are only
a handful of onchain fee records per account.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's account_get_chain_fees() without the ORDER BY clause, *and* it's
only used on the run-recorder.c test.
I also got ChatGPT to compare them, in case I missed something subtle.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that the test where we remove the database causes the bookkeeper
plugin to assert, since we have removed part (but not all!) of its data
by removing the datastore.
Once the transition to the datastore is complete, this can be restored.
Note that we destroy the request before receiving a response, which causes
a message in the trace span which was confusing our test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we don't have our own db, we're going to need to keep this information
in memory (and the datastore). As a first step, simply cache it in memory
and still write through to the db.
This introduces some changes:
1. The account structures are not temporary, but in the hash table (so don't steal them).
2. test_forward_pad_fees_and_cltv assumed ordering, which was a latent bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In practice, it's always either find_account() or find_or_create_account().
This means account_add can be made internal: we rename it to
account_db_add.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will let us cache them gradually in account.c, and eventually
remove the database table.
We also change find_close_account() to find_close_account_name(),
and maybe_mark_account_onchain() into account_onchain_closeheight()
and account_update_closeheight();
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that these migrations were inserted for v0.12, so only someone
upgrading directly from before that (2022-08-23) would be affected.
This avoids having to fix the migrations as we make changes.
We are going to mangle the db to allow testing, but then the final
step will be to migrate it to the core.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Simply wait if there's one going already. This is a minor
optimization, but critical for the case where we do partial refreshes
asynchonously (rather than deleting everything and reloading). This
is currently only coinmoves and chainmoves, but the duplicated effort
is a waste everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a unique integer, and very useful for querying changes. Unlike
our generated rowid, it's *stable* across queries.
We still need an explicit rowid column for list commands which don't
(currently) have this.
Here's the documentation diff:
@@ -85,69 +85,69 @@
TABLES
------
-Note that the first column of every table is a unique integer called `rowid`: this is used for related tables to refer to specific rows in their parent. sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key.
+Note that tables which have a `created_index` field use that as the primary key (and `rowid` is an alias to this), otherwise an explicit `rowid` integer primary key is generated, whose value changes on each refresh. This field is used for related tables to refer to specific rows in their parent. (sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key).
The following tables are currently supported:
- `bkpr_accountevents` (see lightning-bkpr-listaccountevents(7))
@@ -119,14 +119,14 @@
- `payment_id` (type `hex`, sqltype `BLOB`)
- `chainmoves` indexed by `account_id` (see lightning-listchainmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
- `timestamp` (type `u64`, sqltype `INTEGER`)
- `primary_tag` (type `string`, sqltype `TEXT`)
- related table `chainmoves_extra_tags`
- - `row` (reference to `chainmoves.rowid`, sqltype `INTEGER`)
+ - `row` (reference to `chainmoves.created_index`, sqltype `INTEGER`)
- `arrindex` (index within array, sqltype `INTEGER`)
- `extra_tags` (type `string`, sqltype `TEXT`)
- `peer_id` (type `pubkey`, sqltype `BLOB`)
@@ -139,7 +139,7 @@
- `blockheight` (type `u32`, sqltype `INTEGER`)
- `channelmoves` indexed by `account_id` (see lightning-listchannelmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
@@ -204,7 +204,7 @@
- `last_stable_connection` (type `u64`, sqltype `INTEGER`)
- `forwards` indexed by `in_channel` and `in_htlc_id` (see lightning-listforwards(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `in_channel` (type `short_channel_id`, sqltype `TEXT`)
- `in_htlc_id` (type `u64`, sqltype `INTEGER`)
- `in_msat` (type `msat`, sqltype `INTEGER`)
@@ -222,7 +222,7 @@
- `htlcs` indexed by `short_channel_id` and `id` (see lightning-listhtlcs(7))
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `id` (type `u64`, sqltype `INTEGER`)
- `expiry` (type `u32`, sqltype `INTEGER`)
@@ -242,7 +242,7 @@
- `bolt12` (type `string`, sqltype `TEXT`)
- `local_offer_id` (type `hash`, sqltype `BLOB`)
- `invreq_payer_note` (type `string`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `pay_index` (type `u64`, sqltype `INTEGER`)
- `amount_received_msat` (type `msat`, sqltype `INTEGER`)
@@ -408,7 +408,7 @@
- `features` (type `hex`, sqltype `BLOB`)
- `sendpays` indexed by `payment_hash` (see lightning-listsendpays(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `id` (type `u64`, sqltype `INTEGER`)
- `groupid` (type `u64`, sqltype `INTEGER`)
- `partid` (type `u64`, sqltype `INTEGER`)
Changelog-Changed: Plugins: `sql` tables `forwards`, `htlcs`, `invoices`, `sendpays` all use `created_index` as their primary key (and `rowid` is now an alias to this).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't yet do the other list commands, as they are not append-only:
we would need to check deletes and updates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And note the other commands in See Also section.
Note that this means handling the "outpoint" type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `sql` plugin now supports `chainmoves` and `channelmoves` tables.
fetchinvoice is still good for detailed diagnostics and handling
recurring invoices and alternate currencies, but this covers the
"throw some sats" case well.
Changelog-Added: JSON-RPC: `xpay` can now pay a simple offer directly, rather than requiring fetchinvoice first.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't remove them from the graph if they were disconnected, leading to us trying to send an onion message to it:
```
➜ ~ lightning-cli fetchinvoice lno1pg7y7s69g98zq5rp09hh2arnypnx7u3qvf3nzut68pmkcet2xdm8s7ngw5ux2am4wqekwmtkddkkk7nyv45ryefexum82anr8phpp6qrse80qf0aara4slvcjxrvu6j2rp5ftmjy4yntlsmsutpkvkt6878sy4xtpdc0j42xp72an4cl0unvzm7ckx2e6ltlzlgeekh0hhe0mgs2qgpetp66ufc9pwln9gmyv0gk3ndpqvvtynef8adzm3lxv0astkxunjcqx0fsq8kmx5cz574ft45vcweaf3tffp90fjdfl9gkdh4s6xfxaz7srtmsju9gnr3xerjhjqw4xtsmp55f4jjsxxrch703kx020l6wn2ttcd24w5h8f2lz723lk2kpa8ftkernz7h2qqkd3zecz2jmhucwh555xf0np4w45zm8tlka4ktw6d3ne4l9u678y37d24xjcy3el53faulcu5tzzquxrxvcyrvzgk7tyjeapt8wu858m2msgdznhqxl8fps04lef9dc9c 1sat
{
"code": -1,
"message": "onion msg: unknown next peer 024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605"
}
```
Fixes: https://github.com/ElementsProject/lightning/issues/8225
Changelog-Fixed: JSON-RPC: `fetchinvoice` is now more reliable.
Modern style for notifications is to put everything inside an object
of same name as the method.
For now this means duplication for backward compatibility. ChatGPT
helped me do that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
