"When should we reveal preimages onchain?"
This commit tries to simplify the thinking by making the observation:
- we can reveal preimages (actually in any context) if they are already public
- a preimage is public if any other lightning node knows it besides us
- if we learn the preimage from another LN node, it is public
- if we send update_fulfill_htlc, it becomes public
- if we see a preimage onchain, it is public
- in lnsweep._maybe_reveal_preimage_for_htlc:
- partial mpp check is not relevant if preimage is already public
- let's just always do KeepWatchingTXO, for sanity/safety
Co-authored-by: ThomasV <thomasv@electrum.org>
Assert `WalletDBUpgrader.data` is a regular in-memory dict and not
some StoredDict, so if an exception would happen during a wallet
db upgrade the partial changes don't get commited to disk.
Adds the invoice features to the `PaymentInfo` class so we can check if
the sender respects our requested features (e.g. if they tried to send
mpp if we requested no mpp).
Store the channel id instead of the scid in ReceivedMPPHtlc.
The scid can be None, in theory even for multiple channels at the same
time. Using the channel_id which is always available and unique seems
less error prone at the cost of temporarily higher storage requirements
in the db for the duration of the pending htlcs.
Alternatively we could use the local scid alias however using the
channel_id seems less complex and leaves less room for ambiguity.
Allows storing two different payment info of the same payment hash by
including the direction into the db key.
We create and store PaymentInfo for sending attempts and for requests (receiving),
if we try to pay ourself (e.g. through a channel rebalance) the checks
in `save_payment_info` would prevent this and throw an exception.
By storing the PaymentInfos of outgoing and incoming payments separately in
the db this collision is avoided and it makes it easier to reason about
which PaymentInfo belongs where.
refactor `htlc_switch` to new architecture to make it more robust
against partial settlement of htlc sets and increase maintainability.
Htlcs are now processed in two steps, first the htlcs are collected into
sets from the channels, and potentially failed on their own already.
Then a second loop iterates over the htlc sets and finalizes only on
whole sets.
# Conflicts:
# electrum/lnpeer.py
Implement logic to claim a reverse swap funding output to any given
address. This allows to do onchain payments to external recipients
through a submarine swap.
StoredDict to json_db.
convert_key, convert_value are used to convert json objects
to python classes, do not call them in StoredDict.__setitem__
This makes StoredDict agnostic about the type of database we use.
Enforce that the information used to create a bolt11 invoice using
`get_bolt11_invoice()` is similar to the related instance of PaymentInfo
by requiring a PaymentInfo as argument for `get_bolt11_invoice()`.
This way the invoice cannot differ from the created PaymentInfo.
This allows to use the information in PaymentInfo for validation of
incoming htlcs more reliably.
To cover all required information for the creation of a b11 invoice the
PaymentInfo class has to be extended with a expiry and
min_final_cltv_expiry. This requires a db upgrade.
Adds a new configvar `WALLET_PARTIAL_WRITES` to enable/disable partial writes for the walletDB.
This is a further restriction on top of the existing restrictions,
e.g. wallet files still need to have file encryption disabled for partial writes.
It defaults to off, so even for unencrypted wallets we disable partial writes for now.
This is used as a stopgap measure until we fix the issues found with the partial writes impl
(see https://github.com/spesmilo/electrum/issues/10000).
store all plugin data by plugin name in a root dictionary `plugin_data`
inside the wallet db so that plugin data can get deleted again.
Prunes the data of plugins from the wallet db on wallet stop if the
plugin is not installed anymore.
```
$ export ELECTRUM_LINTERS=E9,E101,E129,E273,E274,E703,E71,E722,F5,F6,F7,F8,W191,W29,B
$ export ELECTRUM_LINTERS_IGNORE=B007,B009,B010,B019,B036,F541,F841
$ flake8 . --count --select="$ELECTRUM_LINTERS" --ignore="$ELECTRUM_LINTERS_IGNORE" --show-source --statistics --exclude "*_pb2.py,electrum/_vendor/"
./electrum/commands.py:98:1: F811 redefinition of unused 'format_satoshis' from line 48
def format_satoshis(x):
^
./electrum/commands.py:437:9: F811 redefinition of unused 'Mnemonic' from line 62
from .mnemonic import Mnemonic
^
./electrum/gui/qt/wizard/wallet.py:37:5: F811 redefinition of unused 'Daemon' from line 14
from electrum.daemon import Daemon
^
./electrum/lntransport.py:14:1: F811 redefinition of unused 'Optional' from line 12
from typing import NamedTuple, List, Tuple, Mapping, Optional, TYPE_CHECKING, Union, Dict, Set, Sequence
^
./electrum/lntransport.py:14:1: F811 redefinition of unused 'TYPE_CHECKING' from line 12
from typing import NamedTuple, List, Tuple, Mapping, Optional, TYPE_CHECKING, Union, Dict, Set, Sequence
^
./electrum/plugin.py:966:13: F811 redefinition of unused 'hid' from line 593
import hid
^
./electrum/plugin.py:1040:13: F811 redefinition of unused 'hid' from line 593
import hid
^
./electrum/util.py:44:1: F811 redefinition of unused 'json' from line 26
import json
^
./electrum/util.py:46:1: F811 redefinition of unused 'NamedTuple' from line 29
from typing import NamedTuple, Optional
^
./electrum/util.py:46:1: F811 redefinition of unused 'Optional' from line 29
from typing import NamedTuple, Optional
^
./electrum/util.py:1456:56: F811 redefinition of unused 'traceback' from line 34
async def __aexit__(self, exc_type, exc_value, traceback):
^
./electrum/wallet_db.py:536:9: F811 redefinition of unused 'LOCAL' from line 46
LOCAL = 1
^
./electrum/wallet_db.py:537:9: F811 redefinition of unused 'REMOTE' from line 46
REMOTE = -1
^
./tests/test_bitcoin.py:28:1: F811 redefinition of unused 'bitcoin' from line 9
from electrum import crypto, constants, bitcoin
^
./tests/test_txbatcher.py:11:1: F811 redefinition of unused 'Transaction' from line 7
from electrum.transaction import Transaction, PartialTxInput, PartialTxOutput, TxOutpoint
^
./tests/test_wallet_vertical.py:20:1: F811 redefinition of unused 'Transaction' from line 10
from electrum.transaction import Transaction, PartialTxOutput, tx_from_any, Sighash
^
16 F811 redefinition of unused 'format_satoshis' from line 48
16
```
Ideally, given an on-chain backup, after the remote force-closes, we should be able to spend our anchor output,
to CPFP the remote commitment tx (assuming the channel used OPTION_ANCHORS).
To spend the anchor output, we need to be able to sign with the local funding_privkey.
Previously we derived the funding_key from the channel_seed (which comes from os.urandom).
Prior to anchors, there was no use case for signing with the funding_key given a channel backup.
Now with anchors, we should make its derivation deterministic somehow, in a way so that it can
be derived given just an on-chain backup.
- one way would be to put some more data into the existing OP_RETURN
- uses block space
- the OP_RETURNs can be disabled via "use_recoverable_channels"
- only the initiator can use OP_RETURNs (so what if channel is in incoming dir?)
- instead, new scheme for our funding_key:
- we derive the funding_privkey from the lnworker root secret (derived from our bip32 seed)
- for outgoing channels:
- lnworker_root_secret + remote_node_id + funding_tx_nlocktime
- for incoming channels:
- lnworker_root_secret + remote_node_id + remote_funding_pubkey
- a check is added to avoid reusing the same key between channels:
not letting to user open more than one channel with the same peer in a single block
- only the first 16 bytes of the remote_node_id are used, as the onchain backup OP_RETURNs only contain that
- as the funding_privkey cannot be derived from the channel_seed anymore, it is included in the
imported channel backups, which in turn need a new version defined
- a wallet db upgrade is used to update already stored imported cbs
- alternatively we could keep the imported cbs as-is, so no new version, no new funding_privkey field, as it is clearly somewhat redundant given on-chain backups can reconstruct it
- however adding the field seems easier
- otherwise the existing code would try to derive the funding_privkey from the channel_seed
- also note: atm there is no field in the imported backups to distinguish anchor channels vs static-remotekey channels
Instead of some functions operating with hex strings,
and others using bytes, this consolidates most things to use bytes.
This mainly focuses on bitcoin.py and transaction.py,
and then adapts the API usages in other files.
Notably,
- scripts,
- pubkeys,
- signatures
should be bytes in almost all places now.
The transaction dict can also contain PSBTs (in addition to complete raw hex txs).
This is the case if the user has saved a partial (e.g. unsigned) tx as local into the history.
fixes https://github.com/spesmilo/electrum/issues/8913
In the db, the 'seed_type' field could be present both at the top-level and inside keystores.
Note:
- both fields had usages
- the top-level field was added in 2.8 re "initial segwit support" (3a64ec0f2e)
- there was no db upgrade for it, so older files are missing it
- if present, valid values can be electrum types but also
other types supported by the wizard, e.g. "bip39"
- the keystore-level field was added in 4.1 (7b7bba2299)
- there was a db upgrade when it was introduced, so old files also have it
- if present, valid values can only be electrum types
- there is not much value in the top-level one having a non-electrum value,
and those values were never used by other parts of the code
- note that when creating a standard wallet from a bip39 seed, the seed is discarded.
Only the derived xprv and the derivation path are kept. If we changed this and also kept the seed,
e.g. to display it to the user, then it would make sense to save the seed type (e.g. "bip39").
However storing that seed_type would make more sense at the keystore level (not top-level).
We delete the top-level 'seed_type' field.
```
{
"keystore": {
"seed_type": "segwit",
...
},
"seed_type": "segwit",
...
}
```
This is a regression from 7ca89f56ee, which introduced StoredList.
The newly added test was failing without the change.
```
Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/gui/qt/main_window.py", line 1786, in new_contact_dialog
self.set_contact(line2.text(), line1.text())
File "/home/user/wspace/electrum/electrum/gui/qt/main_window.py", line 1435, in set_contact
self.contacts[address] = ('address', label)
File "/home/user/wspace/electrum/electrum/contacts.py", line 75, in __setitem__
self.save()
File "/home/user/wspace/electrum/electrum/contacts.py", line 62, in save
self.db.put('contacts', dict(self))
File "/home/user/wspace/electrum/electrum/json_db.py", line 42, in wrapper
return func(self, *args, **kwargs)
File "/home/user/wspace/electrum/electrum/json_db.py", line 318, in put
self.data[key] = copy.deepcopy(value)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/lib/python3.10/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle '_thread.RLock' object
```
The following exceptions should be expected:
FileNotFoundError: given wallet path does not exist
StorageReadWriteError: given file is not readable/writable or containing folder is not writable
InvalidPassword: wallet requires a password but no password or an invalid password was given
WalletFileException: any internal wallet data issue. specific subclasses can be caught separately:
- WalletRequiresSplit: wallet needs splitting (split_data passed in Exception)
- WalletRequiresUpgrade: wallet needs upgrade, and no upgrade=True was passed to load_wallet
- WalletUnfinished: wallet file contains an action and needs additional information to finalize. (WalletDB passed in exception)
Removed qml/qewalletdb.py
This patch also fixes load_wallet calls in electrum/scripts and adds a qml workaround for dialogs opening and closing so
fast that the dialog opened==true property change is missed (which we need to manage the dialog/page stack)
- @amount_msat.validator prevents the creation of invoices with e.g. too large amounts
- however the qml gui is mutating invoices by directly setting the `amount_msat` field,
and it looks like attrs validators only run during init.
We can use `on_setattr` (introduced in attrs==20.1.0).
- a wallet db upgrade is added to rm existing insane invoices
- btw the qml gui was already doing its own input validation on the textedit
(see qeconfig.btcAmountRegex). however that only limits the input to not have more
chars than what is needed to represent 21M BTC (e.g. you can still enter 99M BTC,
which the invoice logic does not tolerate later on - but is normally caught).
fixes https://github.com/spesmilo/electrum/issues/8582
follow-up https://github.com/spesmilo/electrum/pull/8536
This replaces 69336befee, which was insufficient.
#8536 added a new field into the struct, which older versions do not ignore but raise:
opening a wallet file with new code updated the struct to include it,
after which old code could no longer open the wallet file.
i.e. #8536 was an invisible wallet upgrade, breaking compat.
This commit simply formalises the wallet upgrade: old code will now show
an understandable error when trying to open new files.
decorators (instead of overloading JsonDB._convert_dict and
_convert_value)
- stored_in for elements of a StoreDict
- stored_as for singletons
- extra register methods are defined for key conversions
This commit was adapted from the jsonpatch branch
- case 1: in version 4.4.1, 4.4.2, the qml GUI wizard allowed creating multisig wallets with an old_mpk as cosigner.
- case 2: in version 4.4.0, 4.4.1, 4.4.2, the qml GUI wizard allowed creating multisig wallets with mixed xpub/Ypub/Zpub.
The corresponding missing input validation was a bug in the wizard, it was unintended behaviour. Validation was added in d2cf21fc2b. Note however that there might be users who created such wallet files.
Re case 1 wallet files: there is no version of Electrum that allows spending from such a wallet. Coins received at addresses are not burned, however it is technically challenging to spend them. (unless the multisig can spend without needing the old_mpk cosigner in the quorum).
Re case 2 wallet files: it is possible to create a corresponding spending wallet for such a multisig, however it is a bit tricky. The script type for the addresses in such a heterogeneous xpub wallet is based on the xpub_type of the first keystore. So e.g. given a wallet file [Yprv1, Zpub2] it will have sh(wsh()) scripts, and the cosigner should create a wallet file [Ypub1, Zprv2] (same order).
Technically case 2 wallet files could be "fixed" automatically by converting the xpub types as part of a wallet_db upgrade. However if the wallet files also contain seeds, those cannot be converted ("standard" vs "segwit" electrum seed).
Case 1 wallet files are not possible to "fix" automatically as the cosigner using the old_mpk is not bip32 based.
It is unclear if there are *any* users out there affected by this. I suspect for case 1 it is very likely there are none (not many people have pre-2.0 electrum seeds which were never supported as part of a multisig who would also now try to create a multisig using them); for case 2 however there might be.
This commit breaks both case 1 and case 2 wallets: these wallet files can no longer be opened in new Electrum, an error message is shown and the crash reporter opens. If any potential users opt to send crash reports, at least we will know they exist and can help them recover.
f321x