This speeds logging slightly, but most of the time is actually
spent in fflush() (which we need).
Result (10 runs, eatmydata):
FAILED tests/test_connection.py::test_bench - assert 24.086638-25.347475(24.6901+/-0.4) == 0
This is an 8% performance improvement (over the entire series), which is not bad.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to iterate through all plugins, making
our "do we even have to construct this notification" optimization
much more efficient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If nobody is subscribed, have notify_start return NULL and the caller
can skip serialization. This is particularly useful for the "log"
notification which can get called a lot.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our CORK logic was wrong, and it's better to use Nagle anyway:
we disable Nagle just before sending timing-critical messages.
Time for 100 (failed) payments:
Before:
148.8573575
After:
10.7356977
Note this revealed a timing problem in test_reject_invalid_payload: we would
miss the cause of the sendonion failure, and waitsendpay would be called
*after* it had failed, so simply returns "Payment failure reason unknown".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: Removed 200ms latency from sending commit/revoke messages.
We need to make sure the lease starts a the block we expect:
```
# Note that l3 has the whole lease delay (minus blocks already mined)
_, _, l3blocks = l3.wait_for_onchaind_tx('OUR_DELAYED_RETURN_TO_WALLET',
'OUR_UNILATERAL/DELAYED_OUTPUT_TO_US')
> assert l3blocks == 4032 - 6 - 2 - 1
E assert 4022 == (((4032 - 6) - 2) - 1)
tests/test_closing.py:985: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can fail to connect because l3 connects to l2 at the same time l2 connects to l3:
```
# The plugin in l2 fixes up the connection, so this works!
> l1.rpc.fetchinvoice(offer['bolt12'])
tests/test_connection.py:4792:
...
elif "error" in resp:
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: fetchinvoice, payload: {'offer': 'lno1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcgqgqjczs7w3jhxazldahxjmmwd4jhxumpvaj47en0wfmkzunytanxz6tvzcssxhftzxfdlwsnfcgw2sy8t5mxa0ytcdfat2nkdwqvpy9nnsa9mzza'}, error: {'code': 1005, 'message': 'Timeout waiting for response'}
```
From logs:
```
lightningd-2 2025-05-07T21:20:54.367Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: peer_out WIRE_INIT
lightningd-2 2025-05-07T21:20:54.367Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: Connect IN
``
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Presumably we want to wait for htlcs to be all resolved here. With the speedup,
this failed under Postgres:
```
> wait_for(lambda: only_one(l2.rpc.listpeerchannels(l3.info['id'])['channels'])['htlcs'] != [])
tests/test_closing.py:4191:
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Once we speed things up, this can receive COMMITMENT_SIGNED, and hence succeed. Let's
drop on receive, not send!
```
2025-05-06T06:07:51.5997333Z # Peers try RBF, break on initial COMMITMENT_SIGNED
2025-05-06T06:07:51.5997855Z bump = l1.rpc.openchannel_bump(chan_id, chan_amount, initpsbt['psbt'])
2025-05-06T06:07:51.5998214Z > with pytest.raises(RpcError):
2025-05-06T06:07:51.5998542Z E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Once we speed things up, we can *receive* commitment_signed before we disconnect
due to sending ours: in this case, we *will* have a scratch tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
DNS seeds have been down/offline for a while, and this code (which
blocks!) has been a source of trouble. We should probably use a
canned set of "known nodes" if we want to bootstrap.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we no longer use DNS seeds for peer lookup fallbacks.
Fixes: https://github.com/ElementsProject/lightning/issues/7913
From grubles' logs:
```
2025-01-06T15:30:31.449Z DEBUG lightningd: attempting connection to 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923 for additional gossip
2025-01-06T15:30:31.449Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Adding 0 addresses to important peer
2025-01-06T15:30:31.449Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:31.449Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.037Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.037Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.428Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.428Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:32.680Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:32.681Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.468Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.469Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.471Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.471Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:33.935Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:33.935Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:34.125Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:34.125Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.496Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.497Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.623Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.623Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.751Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.751Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
2025-01-06T15:30:35.892Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Failed connected out: Unable to connect, no address known for peer
2025-01-06T15:30:35.892Z DEBUG 035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-connectd: Will try reconnect in 300 seconds
```
We promised to wait 300 seconds!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The channel bias feature is not being used yet by any plugin, so this
hopefully doesn't break any working code.
When askrene-bias-channel is called the bias quantity is added on top of
any previous biased already present on that channel instead of
overwriting it.
Changelog-Changed: askrene-bias-channel: bias call add up.
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Reported-by: daywalker90
Changelog-Deprecated: JSON-RPC: channel_state_changed notification field `old_state` value "unknown" (it will be omitted, instead)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
fundwallet generates a block: if l2 sees one block and not the other, it can
consider the time on the first fundchannel to be one block earlier than expected:
```
2025-05-05T21:05:33.6260600Z E TimeoutError: Unable to find "[re.compile('UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Forgetting channel: It has been 51 blocks without the funding transaction ')]" in logs.
...
2025-05-05T21:05:33.7179461Z lightningd-2 2025-05-05T20:44:26.141Z UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Forgetting channel: It has been 52 blocks without the funding transaction 5a18d113f53df8b205ab679924babde4068f2d1876d34edc43701bf92b8ff13f getting deeply confirmed. We are fundee and can forget channel without loss of funds.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is inspired by a patch from @whitslack, which overlapped with this series.
Most importantly, there was only one call to bitcoin_tx_simple_input_weight(),
and it is better to be explicit with that one.
This also changes our funder calculation to assume our own input is taproot,
which it is likely to be given we've defaulted to taproot for outputs for
change addresses since 23.08.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
One in 256 times, we will grind a signature to 70 bytes (or shorter). This breaks
our feerate tests. Unfortunately the grinding is deterministic, so there doesn't
seem to be a way to avoid it. So we add a log message, and then we skip the
feerate test if it happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is legal! And we actually do this in tests, but we didn't check the psbt
was spendable (the next patch does, indirectly, by testing feerate):
```
# Discard prep4 and get all funds again
l1.rpc.txdiscard(prep5['txid'])
# You can have one which is all, but not two.
prep5 = l1.rpc.txprepare([{addr: Millisatoshi(amount * 3 * 1000)},
{addr: 'all'}])
# Feerate should be ~ as we asked for
> assert normal_feerate_perkw - 1 < feerate_from_psbt(bitcoind, l1, prep5['psbt']) < normal_feerate_perkw + 1
E AssertionError: assert (7500 - 1) < -1091803.9574935874
```
Changelog-Fixed: JSON-RPC: `txprepare` with `all` as well as one or more non-all amount fields now produces a valid PSBT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we've fixed various underlying weight mis-estimation, we can do a few final tweaks and
test that anchors work as intended.
1. We assumed a p2wpkh output, but modern change is p2tr.
2. We handed `2` instead of `1` to bitcoin_tx_core_weight (which doesn't matter, but was weird).
3. Make change calculation clearer. I'm not sure the previous one was wrong, but it was harder
to understand.
4. Fix the test and make it clearly test that we are aiming for (and achieving) the right feerate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: anchors' fees are now much closer to the feerate targets.
We need one byte for the number of witness elements. Some callers added it themselves,
but it's always needed. So document and fix the callers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't add the weight of the two sigs! The BOLT defines that to be a worst-case 73 byte sig,
but that turns out to be an overestimate (and this is not required for consensus) so we assume
everyone grinds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use this for anchors, in which case we have a minimum value for
change. If we don't take this into account, we end up with a lower
feerate once we actually create the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We previously treated it as a P2WPKH, which is wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: wallet: fees are much closer to target feerate when doing txprepare/fundchannel.
Due to a bug elsewhere I actually triggered this path, and it
broadcast the tx anyway, *then* closed the channel. We should abandon
the channel if we can, instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To actually evaluate spend cost, we need to know whether it's taproot or not.
Using an enum (rather than making callers examine the script) means we can
ensure all cases are handled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The renaming makes it clear that it's HSM specific.
And it has no pointers, so we can have an array instead of an array of pointers.
I tested this hadn't accidentally changed the wire format by disabling
version checks and using an old hsmd with the altered daemons and
running the test suite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is such a simple struct that we can actually define it in csv.
This prevents us from accidentally breaking the ABI in future.
I tested this hadn't accidentally changed the wire format by disabling
version checks and using an old hsmd with the altered daemons and
running the test suite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I'm about to update our utxo type, but Christian spotted that this is
part of the ABI for the hsm. So make that a private "hsm_utxo" type,
to insulate it from changes.
In particular, the HSM versions only contain the fields that the
hsm cares about, and the wire format is consistent (even though that
*did* include some of those fields, they are now dummies).
In the long term, this should be removed from the ABI: once we
no longer have "close_info" utxos, this information should already be
in the PSBT.
I tested this hadn't accidentally changed the wire format by disabling
version checks and using an old hsmd with the altered daemons and
running the test suite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not just the key index.
Also, remove FIXME: wallet_can_spend is no longer slow with lots of inputs!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're about to fix the feerate calculations in various places, and one
side effect is that we end up trying to add an empty anchor if none is
necessary (and failing, but we log a nasty message about it).
So don't do that, and fix the test which expected it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We actually need to mine another block, so that lightningd considers both
channels candidates for forgetting (and forgets the older one).
Reported-by: daywalker90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We say the offer amount arg can be either an `msat_or_any` or a `currency` string, but technically the order of evaluation is undefined, so it can check the other way, and it should not crash:
```
> offer = l3.rpc.offer(1000, 'test_pay_blindedpath_nodeaddr')
tests/test_pay.py:5692:
...
checker = <TypeChecker types={'array', 'bip340sig', 'boolean', 'currency', 'feerate', 'hash', 'hex', 'integer', 'msat', 'msat_or..._all', 'secret', 'short_channel_id', 'short_channel_id_dir', 'signature', 'string', 'txid', 'u16', 'u32', 'u64', 'u8'}>
instance = 1000
def is_currency(checker, instance):
"""currency including currency code"""
pattern = re.compile(r'^\d+(\.\d+)?[A-Z][A-Z][A-Z]$')
> if pattern.match(instance):
E TypeError: expected string or bytes-like object
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
