Note that these migrations were inserted for v0.12, so only someone
upgrading directly from before that (2022-08-23) would be affected.
This avoids having to fix the migrations as we make changes.
We are going to mangle the db to allow testing, but then the final
step will be to migrate it to the core.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Simply wait if there's one going already. This is a minor
optimization, but critical for the case where we do partial refreshes
asynchonously (rather than deleting everything and reloading). This
is currently only coinmoves and chainmoves, but the duplicated effort
is a waste everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we do this, we get a database error (now we try to refresh
intelligently, is this is currently only chainmoves / channelmoves).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a unique integer, and very useful for querying changes. Unlike
our generated rowid, it's *stable* across queries.
We still need an explicit rowid column for list commands which don't
(currently) have this.
Here's the documentation diff:
@@ -85,69 +85,69 @@
TABLES
------
-Note that the first column of every table is a unique integer called `rowid`: this is used for related tables to refer to specific rows in their parent. sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key.
+Note that tables which have a `created_index` field use that as the primary key (and `rowid` is an alias to this), otherwise an explicit `rowid` integer primary key is generated, whose value changes on each refresh. This field is used for related tables to refer to specific rows in their parent. (sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key).
The following tables are currently supported:
- `bkpr_accountevents` (see lightning-bkpr-listaccountevents(7))
@@ -119,14 +119,14 @@
- `payment_id` (type `hex`, sqltype `BLOB`)
- `chainmoves` indexed by `account_id` (see lightning-listchainmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
- `timestamp` (type `u64`, sqltype `INTEGER`)
- `primary_tag` (type `string`, sqltype `TEXT`)
- related table `chainmoves_extra_tags`
- - `row` (reference to `chainmoves.rowid`, sqltype `INTEGER`)
+ - `row` (reference to `chainmoves.created_index`, sqltype `INTEGER`)
- `arrindex` (index within array, sqltype `INTEGER`)
- `extra_tags` (type `string`, sqltype `TEXT`)
- `peer_id` (type `pubkey`, sqltype `BLOB`)
@@ -139,7 +139,7 @@
- `blockheight` (type `u32`, sqltype `INTEGER`)
- `channelmoves` indexed by `account_id` (see lightning-listchannelmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
@@ -204,7 +204,7 @@
- `last_stable_connection` (type `u64`, sqltype `INTEGER`)
- `forwards` indexed by `in_channel` and `in_htlc_id` (see lightning-listforwards(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `in_channel` (type `short_channel_id`, sqltype `TEXT`)
- `in_htlc_id` (type `u64`, sqltype `INTEGER`)
- `in_msat` (type `msat`, sqltype `INTEGER`)
@@ -222,7 +222,7 @@
- `htlcs` indexed by `short_channel_id` and `id` (see lightning-listhtlcs(7))
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `id` (type `u64`, sqltype `INTEGER`)
- `expiry` (type `u32`, sqltype `INTEGER`)
@@ -242,7 +242,7 @@
- `bolt12` (type `string`, sqltype `TEXT`)
- `local_offer_id` (type `hash`, sqltype `BLOB`)
- `invreq_payer_note` (type `string`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `pay_index` (type `u64`, sqltype `INTEGER`)
- `amount_received_msat` (type `msat`, sqltype `INTEGER`)
@@ -408,7 +408,7 @@
- `features` (type `hex`, sqltype `BLOB`)
- `sendpays` indexed by `payment_hash` (see lightning-listsendpays(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `id` (type `u64`, sqltype `INTEGER`)
- `groupid` (type `u64`, sqltype `INTEGER`)
- `partid` (type `u64`, sqltype `INTEGER`)
Changelog-Changed: Plugins: `sql` tables `forwards`, `htlcs`, `invoices`, `sendpays` all use `created_index` as their primary key (and `rowid` is now an alias to this).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't yet do the other list commands, as they are not append-only:
we would need to check deletes and updates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And note the other commands in See Also section.
Note that this means handling the "outpoint" type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `sql` plugin now supports `chainmoves` and `channelmoves` tables.
Only makes sense to wait on creation, since they neither are deleted
nor updated.
We also enhance the list commands to take the standard index options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `wait`: new subsystems `chainmoves` and `channelmoves`.
This happens if l1 doesn't get a signature, so it doesn't consider
the fulfill complete, but l2 does (and thus credits l1).
This is a trivial test, but will matter should we later correctly
account for channelmoves when onchain: in that case it will actually
hard to tell, in general, what HTLC(s) were fulfilled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that the channel account on l1 doesn't account for the
onchain-fulfilled HTLC:
```
> assert sum(channel1[channel_id]) == -msats_sent_to_2
E assert -50000000000 == -50100000000
E + where -50000000000 = sum([0, -50000000000])
```
Lisa points out that this is accounted for in the chain moves, instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is where all the previous work pays off: we can access the coinmoves
in the db.
Changelog-Added: JSON-RPC: `listchainmoves` and `listchannelmoves` commands to access the audit log of coin movements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Iterating through every peer and channel every time can be very slow
for large nodes, when calling wallet_coinmoves_extract for listcoinmoves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is mainly for the coming list commands, but it's just more
logical to have the common fields at the top.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We change notify_chain_mvt to wallet_save_chain_mvt, and
notify_channel_mvt to wallet_save_channel_mvt, which save to the db
and call the notifier themselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that bookkeeper de-duplicates chain_moves: we need to too! So we add
an index to make this efficient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/4873
In particular, we used to get upset when a peer accepts our channel,
if it was too small! We should do reasonable checks first.
We no longer try to send requests to delay for 2017 blocks though,
so remove that test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: trying to create a channel below our own min-capacity-sat will now fail before asking the peer, not with an error blaming the peer when they accept!
fetchinvoice is still good for detailed diagnostics and handling
recurring invoices and alternate currencies, but this covers the
"throw some sats" case well.
Changelog-Added: JSON-RPC: `xpay` can now pay a simple offer directly, rather than requiring fetchinvoice first.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't remove them from the graph if they were disconnected, leading to us trying to send an onion message to it:
```
➜ ~ lightning-cli fetchinvoice lno1pg7y7s69g98zq5rp09hh2arnypnx7u3qvf3nzut68pmkcet2xdm8s7ngw5ux2am4wqekwmtkddkkk7nyv45ryefexum82anr8phpp6qrse80qf0aara4slvcjxrvu6j2rp5ftmjy4yntlsmsutpkvkt6878sy4xtpdc0j42xp72an4cl0unvzm7ckx2e6ltlzlgeekh0hhe0mgs2qgpetp66ufc9pwln9gmyv0gk3ndpqvvtynef8adzm3lxv0astkxunjcqx0fsq8kmx5cz574ft45vcweaf3tffp90fjdfl9gkdh4s6xfxaz7srtmsju9gnr3xerjhjqw4xtsmp55f4jjsxxrch703kx020l6wn2ttcd24w5h8f2lz723lk2kpa8ftkernz7h2qqkd3zecz2jmhucwh555xf0np4w45zm8tlka4ktw6d3ne4l9u678y37d24xjcy3el53faulcu5tzzquxrxvcyrvzgk7tyjeapt8wu858m2msgdznhqxl8fps04lef9dc9c 1sat
{
"code": -1,
"message": "onion msg: unknown next peer 024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605"
}
```
Fixes: https://github.com/ElementsProject/lightning/issues/8225
Changelog-Fixed: JSON-RPC: `fetchinvoice` is now more reliable.
This was overloading the reckless-rpc plugin input when outputting
json all in one shot. The verbosity was mostly dependency resolution
which wasn't all that helpful so call uv pip install as normal.
Changelog-None: bug introduced this release.
The character classification functions in <ctype.h> are designed to
classify characters returned by <stdio.h> getchar() and friends, which
return characters as signed integers in the range 0 to 255 or EOF. The
behavior of the ctype functions is undefined if they are passed a value
outside of that range, which may happen if they are passed a char-typed
value and the system's char type is signed.
<ccan/str/str.h> defines some inline utility functions that perform the
necessary cast to coerce a char-typed argument into the allowed value
range. Call these wrappers instead of the bare ctype functions when
classifying char-typed characters.
Changelog-None
Modern style for notifications is to put everything inside an object
of same name as the method.
For now this means duplication for backward compatibility. ChatGPT
helped me do that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that we need a workaround for deprecated APIs where "channel_state_changed" output "null" which violated the schema.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
All the core notifications changed over to wrapping the notification
fields in an object with the name of the notification, but notifications
from plugins were missed.
Changelog-Added: Plugins: `channel_hint_update`, `pay_failure` and `pay_success` notifications now have objects of the same name containing the expected fields.
Changelog-Deprecated: Plugins: `channel_hint_update`, `pay_failure` and `pay_success` notification fields outside the same-named object.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than forcing them to wrap their parameters in a "payload"
sub-object, copy in params directly. We include the "origin" field
one level up, if they care.
The next patch restores compatibility for the one place we currently use
them, which is the pay plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: pyln-client: plugin custom notifications origins and payload (use parameters directly)
For older lightningd, we copy field into the raw dict, for newer we recreate the old
"payload" member.
We do fix up the custom_notification test which set params to a string instead of a dict:
that's just weird!
We also change the hacky parsing to proper dict extraction.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: pyln-client: plugin notifications parameters now exposed directly, not wrapped in `params` object.
We always prefer to omit fields rather than use 'null' (or unknown!).
Note that before this, the schema was broken, so we have to put a special
exemption in for that case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We haven't seen the "excessive queue length" backtrace since we fixed gossipd,
so it's safe to drop excess messages without worrying about losing gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Particularly important since we're going to update the format: this makes sure we don't break them!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
