Nobody has hit this yet, but we're about to with our tests.
The size of the db is going to be whatever the total size of the tables are; bigger nodes,
bigger db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes a big difference for large tables. Consider 1.6M channelmoves,
which took 82 seconds to populate, now takes 17 seconds:
Before:
plugin-sql: Time to call listchannelmoves: 10.380341485 seconds
plugin-sql: Time to refresh channelmoves: 82.311287310 seconds
After:
plugin-sql: Time to call listchannelmoves: 9.962815480 seconds
plugin-sql: Time to refresh channelmoves: 15.711549299 seconds
plugin-sql: Time to refresh + create indices for channelmoves: 17.100151235 seconds
tests/test_coinmoves.py::test_generate_coinmoves (50,000):
Time (from start to end of l2 node): 27 seconds
Worst latency: 16.0 seconds
Changelog-Changed: Plugins: `sql` initial load for tables is much faster (e.g 82 to 17 seconds for very large channelmoves table).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Basically, `devtools/reduce-includes.sh */*.c`.
Build time from make clean (RUST=0) (includes building external libs):
Before:
real 0m38.944000-40.416000(40.1131+/-0.4)s
user 3m6.790000-17.159000(15.0571+/-2.8)s
sys 0m35.304000-37.336000(36.8942+/-0.57)s
After:
real 0m37.872000-39.974000(39.5466+/-0.59)s
user 3m1.211000-14.968000(12.4556+/-3.9)s
sys 0m35.008000-36.830000(36.4143+/-0.5)s
Build time after touch config.vars (RUST=0):
Before:
real 0m19.831000-21.862000(21.5528+/-0.58)s
user 2m15.361000-30.731000(28.4798+/-4.4)s
sys 0m21.056000-22.339000(22.0346+/-0.35)s
After:
real 0m18.384000-21.307000(20.8605+/-0.92)s
user 2m5.585000-26.843000(23.6017+/-6.7)s
sys 0m19.650000-22.003000(21.4943+/-0.69)s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This significantly speeds up the query which bookkeeper often does:
"SELECT created_index"
" FROM channelmoves"
" WHERE payment_hash = X'%s'"
" AND credit_msat = %"PRIu64
" AND created_index <= %"PRIu64,
On large databases this scan is expensive, and a payment_hash index
cuts it down a great deal. It does take longer to load the channelmoves
in the first place though (about 3x).
Before:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
169505 /tmp/bkpr-progress
196010 /tmp/bkpr-progress
219370 /tmp/bkpr-progress
235671 /tmp/bkpr-progress
244242 /tmp/bkpr-progress
255362 /tmp/bkpr-progress
265636 /tmp/bkpr-progress
276966 /tmp/bkpr-progress
284451 /tmp/bkpr-progress
288836 /tmp/bkpr-progress
296578 /tmp/bkpr-progress
304571 /tmp/bkpr-progress
After:
$ while sleep 10; do wc -l /tmp/bkpr-progress; done
161421 /tmp/bkpr-progress
238273 /tmp/bkpr-progress
281185 /tmp/bkpr-progress
305787 /tmp/bkpr-progress
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: plugins: the sql plugin now keeps an index on `channelmoves` by `payment_hash`.
Simply wait if there's one going already. This is a minor
optimization, but critical for the case where we do partial refreshes
asynchonously (rather than deleting everything and reloading). This
is currently only coinmoves and chainmoves, but the duplicated effort
is a waste everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a unique integer, and very useful for querying changes. Unlike
our generated rowid, it's *stable* across queries.
We still need an explicit rowid column for list commands which don't
(currently) have this.
Here's the documentation diff:
@@ -85,69 +85,69 @@
TABLES
------
-Note that the first column of every table is a unique integer called `rowid`: this is used for related tables to refer to specific rows in their parent. sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key.
+Note that tables which have a `created_index` field use that as the primary key (and `rowid` is an alias to this), otherwise an explicit `rowid` integer primary key is generated, whose value changes on each refresh. This field is used for related tables to refer to specific rows in their parent. (sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key).
The following tables are currently supported:
- `bkpr_accountevents` (see lightning-bkpr-listaccountevents(7))
@@ -119,14 +119,14 @@
- `payment_id` (type `hex`, sqltype `BLOB`)
- `chainmoves` indexed by `account_id` (see lightning-listchainmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
- `timestamp` (type `u64`, sqltype `INTEGER`)
- `primary_tag` (type `string`, sqltype `TEXT`)
- related table `chainmoves_extra_tags`
- - `row` (reference to `chainmoves.rowid`, sqltype `INTEGER`)
+ - `row` (reference to `chainmoves.created_index`, sqltype `INTEGER`)
- `arrindex` (index within array, sqltype `INTEGER`)
- `extra_tags` (type `string`, sqltype `TEXT`)
- `peer_id` (type `pubkey`, sqltype `BLOB`)
@@ -139,7 +139,7 @@
- `blockheight` (type `u32`, sqltype `INTEGER`)
- `channelmoves` indexed by `account_id` (see lightning-listchannelmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
@@ -204,7 +204,7 @@
- `last_stable_connection` (type `u64`, sqltype `INTEGER`)
- `forwards` indexed by `in_channel` and `in_htlc_id` (see lightning-listforwards(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `in_channel` (type `short_channel_id`, sqltype `TEXT`)
- `in_htlc_id` (type `u64`, sqltype `INTEGER`)
- `in_msat` (type `msat`, sqltype `INTEGER`)
@@ -222,7 +222,7 @@
- `htlcs` indexed by `short_channel_id` and `id` (see lightning-listhtlcs(7))
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `id` (type `u64`, sqltype `INTEGER`)
- `expiry` (type `u32`, sqltype `INTEGER`)
@@ -242,7 +242,7 @@
- `bolt12` (type `string`, sqltype `TEXT`)
- `local_offer_id` (type `hash`, sqltype `BLOB`)
- `invreq_payer_note` (type `string`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `pay_index` (type `u64`, sqltype `INTEGER`)
- `amount_received_msat` (type `msat`, sqltype `INTEGER`)
@@ -408,7 +408,7 @@
- `features` (type `hex`, sqltype `BLOB`)
- `sendpays` indexed by `payment_hash` (see lightning-listsendpays(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `id` (type `u64`, sqltype `INTEGER`)
- `groupid` (type `u64`, sqltype `INTEGER`)
- `partid` (type `u64`, sqltype `INTEGER`)
Changelog-Changed: Plugins: `sql` tables `forwards`, `htlcs`, `invoices`, `sendpays` all use `created_index` as their primary key (and `rowid` is now an alias to this).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't yet do the other list commands, as they are not append-only:
we would need to check deletes and updates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And note the other commands in See Also section.
Note that this means handling the "outpoint" type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `sql` plugin now supports `chainmoves` and `channelmoves` tables.
Changelog-Added: Plugins: `sql` also supports functions `json_object(key1, value1, ...)` to construct JSON objects and `json_group_array(value)` to aggregate rows into JSON array.
Security Considerations
- No new SQL injection risks: Functions only process explicitly provided column values (no arbitrary string parsing).
- Explicit column requirements: Wildcards (*) are not supported, all fields must be named (e.g., json_object('peer_id', id)).
- Permission-bound data access: Functions adhere to the same table/row permissions as the underlying query.
Performance Impact
- Optimized native execution: Leverages SQLite’s built-in JSON1 extension (when available) for efficiency.
- Moderate CPU overhead: Complex nesting may impact performance on large datasets but still faster than application-layer JSON conversion.
Now you can grep for 'sqlite3 version' and see where we would like
to update.
Debian 11 (Bullseye) and Ubuntu 20.04 (Focal) ship with SQLite 3.31.1.
RHEL 9 ships with 3.34.1. Fedora 38+ uses SQLite 3.40+.
Unfortunately, RHEL8 ships with 3.26.0, and is still on maintenance Support
(security fixes, no new features): runs until May 31, 2029.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before vs after:
- `forwards` indexed by `in_channel and in_htlc_id` (see lightning-listforwards(7))
- `forwards` indexed by `in_channel` and `in_htlc_id` (see lightning-listforwards(7))
And:
- `htlcs` indexed by `short_channel_id and id` (see lightning-listhtlcs(7))
- `htlcs` indexed by `short_channel_id` and `id` (see lightning-listhtlcs(7))
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise, deprecating a field causes SELECT * to fail:
```
> l1.rpc.sql(f"SELECT * FROM peerchannels;")
...
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: sql, payload: ('SELECT * FROM peerchannels;',), error: {'code': -1, 'message': 'query failed with access to peerchannels.max_total_htlc_in_msat is prohibited (Deprecated column table peerchannels.max_total_htlc_in_msat)'}
```
So if they use a wildcard, allow access: though "SELECT *" is fraught,
"COUNT(*)" is perfectly legit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without knowing what method was called, we can't have useful general logging
methods, so go through the pain of adding "const char *method" everywhere,
and add:
1. ignore_and_complete - we're done when jsonrpc returned
2. log_broken_and_complete - we're done, but emit BROKEN log.
3. plugin_broken_cb - if this happens, fail the plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we used to allow cmd to be NULL, we had to hand the plugin
everywhere. We no longer do.
1. Various jsonrpc_ functions no longer need the plugin arg.
2. send_outreq no longer needs a plugin arg.
3. The init function takes a command, not a plugin.
4. Remove command_deprecated_in_nocmd_ok.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids globals (and means memleak traverses the variables!): we
only change over the test plugin though, to avoid unnecessary churn.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. sql's dev-sqlfilename should be registered as a dev option.
2. bcli's timeout is an integer, not a string.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a u64, we should pass by copy. This is a big sweeping change,
but mainly mechanical (change one, compile, fix breakage, repeat).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This has the benefit of being shorter, as well as more reliable (you
will get a link error if we can't print it, not a runtime one!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have various functions to convert to a string, rename them all so we can
count on fmt_X being the formatter for struct X, and make them all return
`char *`.
Sometimes they existed but were private, sometimes they had a
different name. Most take a pointer, but simple types pass by copy:
short_channel_id, amount_msat and amount_sat.
The following public functions changed:
1. psbt_to_b64 -> fmt_wally_psbt.
2. pubkey_to_hexstr -> fmt_pubkey.
3. short_channel_id_to_str -> fmt_short_channel_id (scid by copy now!)
4. fmt_signature -> fmt_secp256k1_ecdsa_signature
5. fmt_amount_sat/fmt_amount_msat pass copy not pointer, return non-const char *.
6. node_id_to_hexstr -> fmt_node_id
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In this case the cmd is `sql` but the field we're talking about is from
a different command, so we need a new libplugin API.
Note: there are still no deprecations in any tables used by `sql`, so this
is a bit moot for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we allow deprecation to be set per-connection, we need to
generalize this approach. Instead of filtering out deprecated
fields at schema loading, we need to load them, then refuse
to serve deprecated fields on a per-request basis.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`struct column` has a dynamically allocated member, which is neater if
it is a tal object itself (we allocated the member off the array, instead).
We're about to add two new members, so clean this up first.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I added a plugin arg and was surprised that compile didn't break.
This is because typesafe_cb et al are conditional casts: if the type
isn't as expected it has no effect, but we're passing plugin_option() through
varargs, so everything is accepted!
Add a noop inline to check type, and fix up the two cases where we
used `const char *` instead of `char *`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't handle the case of an array inside a subobject. But that
happens when we add the next commit!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this patch, we only ever loaded the "nodes" table once, then
didn't see updates.
How this ever got past CI is a mystery; perhaps valgrind was so slow that
the updated node_announcement hit the gossmap before we even asked sql
on l3 about the nodes table?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `sql` nodes table now gets refreshed when gossip changes.
I noticed that our subtables were not being cleaned, despite being "ON
DELETE CASCADE". This is because foreign keys were not enabled, but
then I got foreign key errors: rowid cannot be a foreign key anyway!
So create a real "rowid" column. We want "ON DELETE CASCADE" for
nodes and channels (and other tables in future) where we update
partially.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, we generate the schema part from the plugin itself.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
We now add tables to the strmap as we allocate them, since we don't
want to call "finish_td" when we're merely invoked for the
documentation, and don't need a database.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For now, we ignore every deprecated field, but put in the logic so
that future deprecations will work as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This requires us to rename "index" fields, rename fields if we have a
sub-object, and create sub-tables if we have an array, and handle the
fact that some listX commands don't contain array X (listsendpays
contains "payments").
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than two arrays "columns" (for names) and "fieldtypes" (for
types), use a struct. This makes additions easier for successive
patches.
Also pull process_json_obj() out of the loop in list_done().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
