This is reliable, meaning we should never get replayed events.
We have to reference count to make sure all commands are complete,
before we return. In particular, annotating with descriptions can
involve several calls to list commands. We need to give them the
results *after* this is all complete.
test_bookkeeping_descriptions() relied on log messages from
notifications, which now only happen when a command is called. This
changes the test a bit.
Since we no longer subscribe to the balance_snapshot event, we
need to create the wallet account at initialization, as callers
expect it to exist.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we don't have our own db, we're going to need to keep this information
in memory (and the datastore). As a first step, simply cache it in memory
and still write through to the db.
This introduces some changes:
1. The account structures are not temporary, but in the hash table (so don't steal them).
2. test_forward_pad_fees_and_cltv assumed ordering, which was a latent bug.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We didn't remove them from the graph if they were disconnected, leading to us trying to send an onion message to it:
```
➜ ~ lightning-cli fetchinvoice lno1pg7y7s69g98zq5rp09hh2arnypnx7u3qvf3nzut68pmkcet2xdm8s7ngw5ux2am4wqekwmtkddkkk7nyv45ryefexum82anr8phpp6qrse80qf0aara4slvcjxrvu6j2rp5ftmjy4yntlsmsutpkvkt6878sy4xtpdc0j42xp72an4cl0unvzm7ckx2e6ltlzlgeekh0hhe0mgs2qgpetp66ufc9pwln9gmyv0gk3ndpqvvtynef8adzm3lxv0astkxunjcqx0fsq8kmx5cz574ft45vcweaf3tffp90fjdfl9gkdh4s6xfxaz7srtmsju9gnr3xerjhjqw4xtsmp55f4jjsxxrch703kx020l6wn2ttcd24w5h8f2lz723lk2kpa8ftkernz7h2qqkd3zecz2jmhucwh555xf0np4w45zm8tlka4ktw6d3ne4l9u678y37d24xjcy3el53faulcu5tzzquxrxvcyrvzgk7tyjeapt8wu858m2msgdznhqxl8fps04lef9dc9c 1sat
{
"code": -1,
"message": "onion msg: unknown next peer 024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605"
}
```
Fixes: https://github.com/ElementsProject/lightning/issues/8225
Changelog-Fixed: JSON-RPC: `fetchinvoice` is now more reliable.
Make the payment secret field ('s') mandatory for BOLT11 payment requests,
implementing the requirement specified in BOLT11 spec PR #1242
(https://github.com/lightning/bolts/pull/1242).
This security enhancement prevents payment probing attacks by requiring
all invoices to include payment secrets. Changes include:
1. Adding validation in bolt11_decode_nosig() to reject invoices without
the 's' field
2. Adding payment secrets to all test vectors
3. Updating expected encoded values in test cases to include payment secrets
4. Adding a specific test case that verifies proper rejection of invoices
missing the payment secret field
Changelog-Changed: Made payment secret ('s' field) mandatory in BOLT11 payment requests for improved security.
Sangbida and I traced back through ancient history: when the pay plugin was introduced
in 0.9.0 (2019!) it already used the amount_msat parameter (then called `msatoshi`),
so this case effectively "never happens".
But we added a test for it just in case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to wait for exiting startup mode *before* closing channel, so it doesn't ignore
the disconnection of l3, otherwise we don't get a disabled update, and timeout waiting
at line 330:
wait_for(lambda: not l1.is_channel_active(chanid2))
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: sendonion: a new paramter total_amount_msat to make MPP payments with sendpay and sendonion compatible.
[ Reordered to put new parameter at the end --RR ]
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Our CORK logic was wrong, and it's better to use Nagle anyway:
we disable Nagle just before sending timing-critical messages.
Time for 100 (failed) payments:
Before:
148.8573575
After:
10.7356977
Note this revealed a timing problem in test_reject_invalid_payload: we would
miss the cause of the sendonion failure, and waitsendpay would be called
*after* it had failed, so simply returns "Payment failure reason unknown".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: Removed 200ms latency from sending commit/revoke messages.
It makes the schema simpler, and indeed, expressable by GRPC.
Changelog-Added: JSON-RPC: `wait` now has separate `invoices`, `forwards` and `sendpays` objects for each subsystem.
Changelog-Deprecated: JSON-RPC: `wait` reply `details` object: use subsytem specific object instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We reconnect from l3->l2, but l2 is also trying to reconnect and
we can race:
```
# Now try when l3 uses scid for entry point of blinded path.
l3.stop()
l3.daemon.opts['dev-invoice-bpath-scid'] = None
l3.start()
> l3.rpc.connect(l2.info['id'], 'localhost', l2.port)
tests/test_pay.py:5667:
...
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: connect, payload: {'id': '022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 'host': 'localhost', 'port': 33557}, error: {'code': 402, 'message': 'disconnected during connection'}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By submitting them all at once, rather than serially, we should *definitely* hit the
ratelimit. We can also reduce the timeout (from 60 seconds) to speed the test up.
```
@pytest.mark.slow_test
def test_onionmessage_ratelimit(node_factory):
l1, l2 = node_factory.line_graph(2, fundchannel=False,
opts={'allow_warning': True})
offer = l2.rpc.call('offer', {'amount': '2msat',
'description': 'simple test'})
# Hopefully we can do this fast enough to reach ratelimit!
> with pytest.raises(RpcError, match="Timeout waiting for response"):
E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
tests/test_pay.py:5825: Failed
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Really, any peer without a live channel is a bad prospect.
This requires us to wire the "enabled" flag through listincoming:
fortunately that's an internal, undocumented interface, so we don't
have a schema change.
Changelog-Fixed: Offline peers no longer selected for blinded paths..
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/8127
Following 6e4ff6a, nodes now check for their public address and
generate a blinded path for invoices if the address is not advertized.
This breaks several of the tests where the blinded path is expected
to have the entrypoint be the node itself.
Changelog-None
Suggested-by: Matt Corallo
Fixes: https://github.com/ElementsProject/lightning/issues/7806
Changelog-Changed: Offers: we will use a blinded path if we have no advertized address (so payers wouldn't be able to connect directly).
If they give us the invstring, we can at least set who signed the invoice. Of course,
it might not be a real node_id (with blinded paths).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This appears in listsendpays / listpays, and is useful information (if we know!).
This doesn't fix old payments, but means that xpay can use this for new payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In fact, there are several places where we try to decode old invoices,
and they should all work. The only place we should enforce expiration is
when we're going to pay.
This also revealed that xpay wasn't checking bolt11 expiries!
Reported-by: hMsats
Fixes: https://github.com/ElementsProject/lightning/issues/7869
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `decode` refused to decode expired bolt12 invoices.
Let lightningd feed us hints to try first, but we can extract the
addresses from node_announcement messages ourselves.
(Lightningd used to ask gossipd on our behalf: this is far simpler!)
One side effect of this is that we don't hand back address hints given to us
by lightningd: it would use these again for reconnecting. This is breaks
test_sendpay_grouping, so we disable it temporarily.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As we can see from the previous test, l3 tells us why it rejected the payment:
```
lightningd-3 2024-11-19T03:56:27.151Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Failing HTLC because of an invalid payload (TLV 10 pos 104): cltv_expiry 136 > payment_constraint 130
```
It turns out, we were not updating the block height in the plugin!
Without this, when we create a (non-dummy) blinded path we set a
too-low CLTV restriction, and it doesn't work after a few blocks!
Note we were actually triggering this error in the xpay tests!
Reported-by: Vincenzo Palazzo
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Offers: Receiving bolt12 payments where we have no public channels would fail a few blocks after startup.
Don't reply with update_fail_malformed_htlc, even though WIRE_INVALID_ONION_BLINDING
has BADONION set. Fail it with a normal error message.
This fixes a known FIXME.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: entry to blinded paths return more useful errors (e.g if it's the final node, you get a real error, otherwise you get invalid_onion_blinding).
Sometimes l1 ratelimits before l2, and l2 receives the warning message, not l1:
```
> assert l1.daemon.is_in_log('WARNING: Ratelimited onion_message: exceeded one per 250msec')
E AssertionError: assert None
E + where None = <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7f13435f45b0>>('WARNING: Ratelimited onion_message: exceeded one per 250msec')
E + where <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7f13435f45b0>> = <pyln.testing.utils.LightningD object at 0x7f13435f45b0>.is_in_log
E + where <pyln.testing.utils.LightningD object at 0x7f13435f45b0> = <fixtures.LightningNode object at 0x7f13435cbb80>.daemon
...
lightningd-1 2024-11-19T00:45:43.721Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_in WIRE_ONION_MESSAGE
lightningd-1 2024-11-19T00:45:43.721Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_out WIRE_WARNING
lightningd-2 2024-11-19T00:45:43.722Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_out WIRE_ONION_MESSAGE
lightningd-2 2024-11-19T00:45:43.722Z DEBUG connectd: REPLY WIRE_CONNECTD_INJECT_ONIONMSG_REPLY with 0 fds
lightningd-2 2024-11-19T00:45:43.722Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: peer_in WIRE_WARNING
lightningd-2 2024-11-19T00:45:43.722Z INFO 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: Received WIRE_WARNING: WARNING: Ratelimited onion_message: exceeded one per 250msec
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It only works on BOLT11, and has long been replaced by the more
generic "decode".
Removing it will stop the confusion!
(Note: documentation claims it was introduced in 23.08, but that was
wrong, as it's been in CLN since the beginning).
[ Fixup from: niftynei <niftynei@gmail.com> ]
Fixes: https://github.com/ElementsProject/lightning/issues/6419
Changelog-Deprecated: JSON-RPC: `decodepay`: use `decode`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fails when l3 doesn't know address for l1, to connect to it:
```
2024-11-16T04:45:42.2243366Z lightningd-3 2024-11-16T04:35:10.582Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: peer_in WIRE_ONION_MESSAGE
2024-11-16T04:45:42.2244342Z lightningd-3 2024-11-16T04:35:10.582Z DEBUG lightningd: Got onionmsg reply_path
2024-11-16T04:45:42.2245398Z lightningd-3 2024-11-16T04:35:10.582Z DEBUG plugin-offers: Note: disallowing deprecated onion_message_recv.blinding
2024-11-16T04:45:42.2246408Z lightningd-3 2024-11-16T04:35:10.586Z UNUSUAL plugin-offers: No incoming channel for 5msat, so no blinded path
2024-11-16T04:45:42.2247289Z lightningd-3 2024-11-16T04:35:10.605Z DEBUG hsmd: Client: Received message 25 from client
2024-11-16T04:45:42.2248372Z lightningd-3 2024-11-16T04:35:10.606Z DEBUG plugin-offers: connecting directly to 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518
2024-11-16T04:45:42.2249451Z lightningd-3 2024-11-16T04:35:10.606Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds
2024-11-16T04:45:42.2250743Z lightningd-3 2024-11-16T04:35:10.607Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: Failed connected out: Unable to connect, no address known for peer
```
This is because the test which was supposed to wait for addresses is
wrong: it passes when l3 knows nothing! (`all([])` == `True`)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now pay learns, it sometimes learns not to try again:
```
> assert(len(l1.rpc.listpays()['pays']) == 2)
E AssertionError: assert 1 == 2
E + where 1 = len([{'amount_sent_msat': 0, 'bolt11': 'lnbcrt1pnjj7mysp5tfx8n6nyx7ehszgqn7gqm2r6n079p22u2yddtg797ka3pa9557tspp5f89z6genjqrl3knymvav9ajwcxrm5w7arxux06rrhjux88derjyqdq8v3jhxccxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqsqqqqqqqqpqqqqqzsqqc9qxpqysgqcuyr7qlyctf9w96fqg4wetqt7t5v938dagmv0r777n902utjufujzjxl3289r97yngft966zly3ehxfp469dh3lq0hkv6r684snvunqppuyvsl', 'created_at': 1730771812, 'destination': '035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d', ...}])
tests/test_pay.py:5147: AssertionError
```
We fix this by creating a fresh channel, so it will try.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is like `sendonion` but unwraps the onion as the first hop,
avoiding nasty special cases for blinded paths which start with this
node, and also self-pay.
Tests split into multiple ones after Christian's review.
Changelog-Added: JSON-RPC: `injectpaymentonion` for initiating an HTLC like a peer would do.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
