This schema change updates newaddr to remove bip86 which was previously added, since don't want to make unnecessary schema changes this is being removed.
The generated files for the exposesecret schema change are also being added
Incorporate a time: this covers the restart case as well. And make it time_mono(),
which doesn't get overridden when we override normal wall time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a trick from bcli: we ask bitcoind for the block, and it hands
us a 2MB hex blob (which we read in multiple parts). Our parser wades
through it all, but a quick search for '}' makes it much faster.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For old channels, this can take a while, and it stops everything. But
we are only doing this to save space; it's not a *functional* necessity.
A quick and dirty test with 50,000 htlcs shows the htlc deletion took
450msec. I tried adding an index, and changing it to set hstate to
HTLC_STATE_INVALID instead of deleting entries, but it still took about 350ms.
Whereas the "COUNT(*)" only took 1.7msec, so it's worth keeping.
Reported-by: @michael1011
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: lightningd: we defer deletion of old htlcs on channel close, to avoid pausing for a long time (we clean them on startup)
Fixes: https://github.com/ElementsProject/lightning/issues/7962
Note that the test where we remove the database causes the bookkeeper
plugin to assert, since we have removed part (but not all!) of its data
by removing the datastore.
Once the transition to the datastore is complete, this can be restored.
Note that we destroy the request before receiving a response, which causes
a message in the trace span which was confusing our test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that we need a workaround for deprecated APIs where "channel_state_changed" output "null" which violated the schema.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As per BOLT recommendation https://github.com/lightning/bolts/pull/1232, this means
we will insist on this being available.
For CLN, we added this in 0.12.0 (2022-08-23), though there were fixes as late as 24.02. Either way that's well outside our support window.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Closes: https://github.com/ElementsProject/lightning/issues/8152
Changelog-Changed: Protocol: We now insist that peers support `option_channel_type` (in CLN since 0.12.0 in late 2022, similar for other implementations).
By policy, our stub hsmd accepts everything: openingd is supposed to
sort this out (or use VLS for a real HSM implementation!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Config: the node no longer crashes if you set `watchtime-blocks` to 0 (which is fine for testing: don't do this on mainnet!).
Now we've make it only on existing channels, and not have to call
listdatastore every time, that means we can safely turn it on by
default.
Changelog-Added: Protocol: we now offer peer storage to any peers who create a channel.
Changelog-Deprecated: Config: `--experimental-peer-storage` (it's now the default).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
signmessagewithkey: allows to sign a message with a key associated with
one bitcoin address in our wallet.
Changelog-Added: add a new rpc command signmessagewithkey to sign input messages with keys from our wallet.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Make sure we are sending bogus channel reestablish after recovering from
emergency.recover file and peer storage backup.
Key Changes:
- Add wait_for_log() with appropriate debug statements
We need to make sure the close tx is generated where we expect, so the following code
doesn't wait foreverL
```
2025-05-08T04:31:31.1071194Z > assert l2.rpc.wait('htlcs', 'deleted', 3)['deleted'] == 5
2025-05-08T04:31:31.1071518Z
2025-05-08T04:31:31.1071645Z tests/test_misc.py:3398:
...
2025-05-08T04:31:31.1079841Z E Failed: Timeout >1800.0s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need one byte for the number of witness elements. Some callers added it themselves,
but it's always needed. So document and fix the callers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We previously treated it as a P2WPKH, which is wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: wallet: fees are much closer to target feerate when doing txprepare/fundchannel.
To actually evaluate spend cost, we need to know whether it's taproot or not.
Using an enum (rather than making callers examine the script) means we can
ensure all cases are handled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were supposed to put the sqlite db in a different directory, but
the test was wrong! So occasionally we would crash with:
```
Failed to commit DB transaction: Failed to commit a transaction: disk I/O error
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It makes the schema simpler, and indeed, expressable by GRPC.
Changelog-Added: JSON-RPC: `wait` now has separate `invoices`, `forwards` and `sendpays` objects for each subsystem.
Changelog-Deprecated: JSON-RPC: `wait` reply `details` object: use subsytem specific object instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Testing parenting handling revealed several issues:
1. By calling "trace_span_start" when CLN_TRACEPARENT is set produces a bogus
entry, for which the span_id is overwritten so we never end it.
2. We don't need to close the remote parent when we close the first child: in
fact, this causes the remaining traces to be detached from the parent!
3. Suspension should return current to the parent, not to NULL.
Now the traces balance as we expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We play with directory permissions, but sqlites needs that, and sometimes (due to a timer, perhaps?)
it gets really upset about it:
```
lightningd-1 2025-03-16T23:29:58.506Z INFO lightningd: Server started with public key 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518, alias JUNIORBEAM-f91eb11-modded (color #0266e4) and lightningd f91eb11-modded
lightningd-1 2025-03-16T23:29:58.617Z DEBUG lightningd: Adding block 101: 55af171ade598f8c4f9a494eaaffe6b9f5ea64c7b0f3b273694148adf7ad7385
lightningd-1 2025-03-16T23:29:58.752Z TRACE lightningd: Calling rpc_command hook of plugin cln-xpay
lightningd-1 2025-03-16T23:29:58.764Z TRACE lightningd: Plugin cln-xpay returned from rpc_command hook call
lightningd-1 2025-03-16T23:29:58.784Z TRACE lightningd: Calling rpc_command hook of plugin cln-xpay
lightningd-1 2025-03-16T23:29:58.840Z TRACE lightningd: Plugin cln-xpay returned from rpc_command hook call
lightningd-1 2025-03-16T23:29:58.854Z TRACE lightningd: Calling rpc_command hook of plugin cln-xpay
lightningd-1 2025-03-16T23:29:58.865Z DEBUG gossipd: REPLY WIRE_GOSSIPD_NEW_BLOCKHEIGHT_REPLY with 0 fds
lightningd-1 2025-03-16T23:29:58.901Z **BROKEN** lightningd: Error executing statement: db/exec.c:108: UPDATE vars SET intval = intval + 1 WHERE name = 'data_version' AND intval = ?: attempt to write a readonly database
lightningd-1 2025-03-16T23:29:58.916Z **BROKEN** lightningd: Error executing statement: db/exec.c:108: UPDATE vars SET intval = intval + 1 WHERE name = 'data_version' AND intval = ?: attempt to write a readonly database
lightningd-1 2025-03-16T23:29:58.941Z **BROKEN** lightningd: FATAL SIGNAL 6 (version f91eb11-modded)
{'github_repository': 'ElementsProject/lightning', 'github_sha': 'f91eb118388dc1455c67d16079168fd0267ba248', 'github_ref': 'refs/pull/8112/merge', 'github_ref_name': 'HEAD', 'github_run_id': 13888173501, 'github_head_ref': 'flake-memleak', 'github_run_number': 12573, 'github_base_ref': 'master', 'github_run_attempt': '1', 'testname': 'test_setconfig_access', 'start_time': 1742167762, 'end_time': 1742167800, 'outcome': 'fail'}
----------------------------- Captured stderr call -----------------------------
Error executing statement: db/exec.c:108: UPDATE vars SET intval = intval + 1 WHERE name = 'data_version' AND intval = ?: attempt to write a readonly database
lightningd: FATAL SIGNAL 6 (version f91eb11-modded)
Log dumped in /tmp/lightning-crash.log.20250316232958
Lost connection to the RPC socket.Lost connection to the RPC socket.
=========================== short test summary info ============================
FAILED tests/test_misc.py::test_setconfig_access - AssertionError: Regex pattern did not match.
Regex: 'Cannot write to config file /tmp/ltests-22a5dz1j/test_setconfig_access_1/lightning-1/regtest/config'
Input: "RPC call failed: method: check, payload: {'command_to_check': 'setconfig', 'config': 'min-capacity-sat', 'val': 1000000}, error: Connection to RPC server lost."
ERROR tests/test_misc.py::test_setconfig_access - ValueError:
Node errors:
- lightningd-1: had BROKEN messages
- lightningd-1: Node exited with return code -6
```
So we move the db file for sqlite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unfortunately a spec typo means the data fields are missing (PR pending),
so we still patch those in.
The message "your_peer_storage" got renamed to "peer_storage_retrieval",
and the option "want_peer_backup_storage" was removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `experimental-peer-storage` now only advertizes feature 43, not 41.
The old blindedpay fields would have an entry per hop, but that was changed
to a single per-path entry. The devtools hadn't caught up.
Similarly, it didn't like descriptionless offer fields in invreqs and invoices.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is neater than appending to some random file: we only do that once
if there's no "include" line to include a ".setconfig" file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setconfig` now has a `transient` flag which means it won't rewrite your config file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are about to start logging warnings, so use modern versions for tests
which aren't explicitly about testing obsolete ones.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I can't reproduce this, but CI did (with Elements):
```
[gw3] linux -- Python 3.8.18 /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.8/bin/python
node_factory = <pyln.testing.utils.NodeFactory object at 0x7fd0e20f57f0>
bitcoind = <pyln.testing.utils.ElementsD object at 0x7fd0e307dbe0>
executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x7fd0e307da30>
@pytest.mark.openchannel('v1')
@pytest.mark.openchannel('v2')
def test_lightningd_still_loading(node_factory, bitcoind, executor):
"""Test that we recognize we haven't got all blocks from bitcoind"""
mock_release = Event()
# This is slow enough that we're going to notice.
def mock_getblock(r):
conf_file = os.path.join(bitcoind.bitcoin_dir, 'bitcoin.conf')
brpc = RawProxy(btc_conf_file=conf_file)
if r['params'][0] == slow_blockid:
mock_release.wait(TIMEOUT)
return {
"result": brpc._call(r['method'], *r['params']),
"error": None,
"id": r['id']
}
# Start it, establish channel, get extra funds.
l1, l2, l3 = node_factory.get_nodes(3, opts=[{'may_reconnect': True,
'wait_for_bitcoind_sync': False},
{'may_reconnect': True,
'wait_for_bitcoind_sync': False},
{}])
node_factory.join_nodes([l1, l2])
# Balance l1<->l2 channel
l1.pay(l2, 10**9 // 2)
l1.stop()
# Now make sure l2 is behind.
bitcoind.generate_block(2)
# Make sure l2/l3 are synced
sync_blockheight(bitcoind, [l2, l3])
# Make it slow grabbing the final block.
slow_blockid = bitcoind.rpc.getblockhash(bitcoind.rpc.getblockcount())
l1.daemon.rpcproxy.mock_rpc('getblock', mock_getblock)
l1.start(wait_for_bitcoind_sync=False)
# It will warn about being out-of-sync.
assert 'warning_bitcoind_sync' not in l1.rpc.getinfo()
assert 'warning_lightningd_sync' in l1.rpc.getinfo()
# Make sure it's connected to l2 (otherwise we get TEMPORARY_CHANNEL_FAILURE)
wait_for(lambda: only_one(l1.rpc.listpeers(l2.info['id'])['peers'])['connected'])
# Payments will succced.
l1.pay(l2, 1000)
> assert l1.daemon.is_in_log(r"Sending HTLC while still syncing with bitcoin network \(104 vs 105\)")
E AssertionError: assert None
E + where None = <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0>>('Sending HTLC while still syncing with bitcoin network \\(104 vs 105\\)')
E + where <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0>> = <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0>.is_in_log
E + where <pyln.testing.utils.LightningD object at 0x7fd0e20f9fa0> = <fixtures.LightningNode object at 0x7fd0e20f59d0>.daemon
```
What was in logs was:
```
lightningd-1 2024-11-18T05:33:50.634Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Sending HTLC while still syncing with bitcoin network (103 vs 105)
```
Implying that l1 was an extra block behind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As the first user of a persistent layer, this tripped tests which
assumed the datastore would be empty!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
