Commit Graph

33 Commits

Author SHA1 Message Date
bitromortac e5458bb534 wallet_db: migrate to upfront shutdown scripts 2021-01-11 11:37:14 +01:00
Malcolm Smith 67ae678137 storage/db: use faster JSON encoder settings when wallet is encrypted
The standard json module has an optimized C encoder, but that doesn't
currently support indentation. So if you request indentation, it falls
back on the slower Python encoder.

Readability doesn't matter for encrypted wallets, so this disables
indentation when the wallet is encrypted.

-----

based on https://github.com/Electron-Cash/Electron-Cash/commit/b2399b6a3e3e39ddd82ffc432bb1ca07f2aab454

For a large encrypted wallet, compare:
before change:
JsonDB.dump 1.3153 sec
zlib.compress 1.281 sec
ECPubkey.encrypt_message 0.1744 sec

after change:
JsonDB.dump 0.5059 sec
zlib.compress 1.3120 sec
ECPubkey.encrypt_message 0.1630 sec

Co-authored-by: SomberNight <somber.night@protonmail.com>
2021-01-06 21:14:56 +01:00
ThomasV 89ecc85c64 convert_version_24: loop over list instead of dict items (see #6863 and #6349) 2020-12-18 14:30:47 +01:00
SomberNight 3a7c00634e wallet_db: impl convert_version_33: put 'height' field into invoices
The 'height' field was added in https://github.com/spesmilo/electrum/commit/cdfaaa260942b807f809c2c0414fb242a03e945a
At the time we thought we could just add it with a default value without a db upgrade;
however the issue is that if old code tries to open a new db, it will fail (due to unexpected new field).
Hence it is better to do an explicit conversion where old code *knows* it cannot open the new db.

E | gui.qt.ElectrumGui |
Traceback (most recent call last):
  File "...\electrum\electrum\gui\qt\__init__.py", line 257, in start_new_window
    wallet = self.daemon.load_wallet(path, None)
  File "...\electrum\electrum\daemon.py", line 488, in load_wallet
    db = WalletDB(storage.read(), manual_upgrades=manual_upgrades)
  File "...\electrum\electrum\wallet_db.py", line 72, in __init__
    self.load_data(raw)
  File "...\electrum\electrum\wallet_db.py", line 103, in load_data
    self._after_upgrade_tasks()
  File "...\electrum\electrum\wallet_db.py", line 189, in _after_upgrade_tasks
    self._load_transactions()
  File "...\electrum\electrum\util.py", line 408, in <lambda>
    return lambda *args, **kw_args: do_profile(args, kw_args)
  File "...\electrum\electrum\util.py", line 404, in do_profile
    o = func(*args, **kw_args)
  File "...\electrum\electrum\wallet_db.py", line 1139, in _load_transactions
    self.data = StoredDict(self.data, self, [])
  File "...\electrum\electrum\json_db.py", line 79, in __init__
    self.__setitem__(k, v)
  File "...\electrum\electrum\json_db.py", line 44, in wrapper
    return func(self, *args, **kwargs)
  File "...\electrum\electrum\json_db.py", line 105, in __setitem__
    v = self.db._convert_dict(self.path, key, v)
  File "...\electrum\electrum\wallet_db.py", line 1182, in _convert_dict
    v = dict((k, Invoice.from_json(x)) for k, x in v.items())
  File "...\electrum\electrum\wallet_db.py", line 1182, in <genexpr>
    v = dict((k, Invoice.from_json(x)) for k, x in v.items())
  File "...\electrum\electrum\invoices.py", line 108, in from_json
    return OnchainInvoice(**x)
TypeError: __init__() got an unexpected keyword argument 'height'
2020-12-17 15:17:08 +01:00
SomberNight da6080421e wallet_db: WalletDB.get_txo_addr now returns dict instead of list 2020-10-18 20:37:25 +02:00
SomberNight 4b6c86ecbe wallet: make labels private, and access to need lock
e.g. labels plugin iterated over wallet.labels on asyncio thread while user could trigger an edit from Qt thread
2020-10-13 18:57:55 +02:00
SomberNight a7199696d3 json_db: exempt keystore from StoredDict conversion
The keystore logic would need to be significantly changed to nicely
interoperate with StoredDict/json_db logic.
(just see KeyStore.__init__() and KeyStore.dump())
For now we exempt the keystore from the recursive StoredDict conversion, as
it is a smaller change that is also easier to review for correctness.

fixes #6066
fixes #6401

also reverts 2d3c2eeea9 (which was an even hackier workaround for #6066)
2020-09-04 16:11:01 +02:00
SomberNight 307403a02c invoices: rm old corrupted non-bip70 invoices
fixes #6345
2020-07-08 00:57:23 +02:00
SomberNight 9cd79ec2e5 WalletDB: raise different exc if cannot parse given file
closes #6292
2020-06-27 16:03:03 +02:00
SomberNight dee5d52948 invoices: make sure that OnchainInvoice .exp and .time are not None
related: #6284
2020-06-27 02:27:50 +02:00
SomberNight d5f368c584 LN invoices: support msat precision
fixes #6250
2020-06-22 22:48:13 +02:00
ThomasV 3874f7ec77 swaps: use StoredObject to store data 2020-06-18 14:28:40 +02:00
SomberNight 43892dd61a invoices: fix #6233 2020-06-14 03:39:35 +02:00
ThomasV 6058829870 Use attr.s classes for invoices and requests:
- storage upgrade
 - fixes #6192
 - add can_pay_invoice, can_receive_invoice to lnworker
2020-06-01 21:02:45 +02:00
ThomasV 8f41aeb783 Replace wallet backup with channel backups
- channels can be backed up individually
 - backups are added to lnwatcher
 - AbstractChannel ancestor class
2020-04-10 14:45:23 +02:00
ThomasV f3995350e8 localconfig: rename seed to channel_seed 2020-04-06 16:53:48 +02:00
ThomasV 0ea21c59d2 Save channel seed in localconfig 2020-04-04 13:28:19 +02:00
SomberNight 7ac1cace7a wallet_db.clear_history: now clears prevouts_by_scripthash too
(which is the logical thing to do, as it too will be rebuilt as part of
the history, and the parts of it that might not be present after the
rebuild is exactly what a call to "clear_history" is supposed to get rid of)
2020-03-27 02:28:43 +01:00
SomberNight 01207316aa storage upgrade: move "htlc_minimum_msat" to base channel config 2020-03-26 09:05:04 +01:00
SomberNight ea0981ebeb lnutil.UpdateAddHtlc: use attrs instead of old-style namedtuple 2020-03-17 20:31:11 +01:00
ThomasV 444610452e wallet_db: encapsulate type conversions with attr.s converter 2020-03-17 11:04:49 +01:00
SomberNight bf4b2a15a6 wallet_db: fix _convert_version_25 (bip70 expiration) 2020-03-05 17:15:09 +01:00
SomberNight 02fcc6f570 wallet_db.get_transaction: tolerate if tx_hash is None
Traceback (most recent call last):
  File "...\electrum\electrum\gui\qt\main_window.py", line 1503, in do_pay
    self.do_pay_invoice(invoice)
  File "...\electrum\electrum\gui\qt\main_window.py", line 1516, in do_pay_invoice
    self.pay_onchain_dialog(self.get_coins(), outputs)
  File "...\electrum\electrum\gui\qt\main_window.py", line 1570, in pay_onchain_dialog
    self.preview_tx_dialog(make_tx=make_tx,
  File "...\electrum\electrum\gui\qt\main_window.py", line 1574, in preview_tx_dialog
    d = PreviewTxDialog(make_tx=make_tx, external_keypairs=external_keypairs,
  File "...\electrum\electrum\gui\qt\transaction_dialog.py", line 654, in __init__
    self.update()
  File "...\electrum\electrum\gui\qt\transaction_dialog.py", line 392, in update
    tx_details = self.wallet.get_tx_info(self.tx)
  File "...\electrum\electrum\wallet.py", line 486, in get_tx_info
    tx_we_already_have_in_db = self.db.get_transaction(tx_hash)
  File "...\electrum\electrum\json_db.py", line 44, in wrapper
    return func(self, *args, **kwargs)
  File "...\electrum\electrum\wallet_db.py", line 822, in get_transaction
    assert isinstance(tx_hash, str)
AssertionError

Traceback (most recent call last):
  File "...\electrum\electrum\gui\qt\confirm_tx_dialog.py", line 65, in timer_actions
    self.update()
  File "...\electrum\electrum\gui\qt\transaction_dialog.py", line 392, in update
    tx_details = self.wallet.get_tx_info(self.tx)
  File "...\electrum\electrum\wallet.py", line 486, in get_tx_info
    tx_we_already_have_in_db = self.db.get_transaction(tx_hash)
  File "...\electrum\electrum\json_db.py", line 44, in wrapper
    return func(self, *args, **kwargs)
  File "...\electrum\electrum\wallet_db.py", line 822, in get_transaction
    if tx_hash is None:
AssertionError
2020-02-28 20:23:50 +01:00
SomberNight 88658f9c2c WalletDB: add type hints, and also corresponding asserts for sanity 2020-02-24 18:26:49 +01:00
ThomasV 20d8da7e24 db upgrade: store channel tx height and timestamps in 'channels' 2020-02-24 10:07:55 +01:00
ThomasV 317d405432 follow-up previous commit 2020-02-19 11:39:52 +01:00
ThomasV cc6e461d3e storage upgrade for invoices and requests. fixes #5959 2020-02-19 11:26:03 +01:00
SomberNight 111ef9ebb1 follow-up fixes to storage-db separation
e1ce3aace7
2020-02-13 20:00:12 +01:00
ThomasV beee880dba fix data_loss_protect (missing return, json conversion) 2020-02-12 14:19:31 +01:00
ThomasV e1ce3aace7 Separate db from storage
- storage is content-agnostic
 - db and storage are passed to wallet contructor
2020-02-10 17:45:23 +01:00
ThomasV dbceed2647 Restructure wallet storage:
- Perform json deserializations in wallet_db
 - use StoredDict class that keeps tracks of its modifications
2020-02-04 13:35:58 +01:00
ThomasV b08947a506 storage upgrade: convert lists to dict (txi, txo, revocation_store channels) 2020-02-04 12:11:18 +01:00
ThomasV 149cd9598a Separate JsonDB and WalletDB 2020-02-03 12:36:07 +01:00