Commit Graph

59 Commits

Author SHA1 Message Date
ThomasV b96cc82333 Make storage a field of db
This comes from the jsonpatch_new branch.
I rather have in master now, because it touches a lot of filese.
2023-08-18 08:08:31 +02:00
SomberNight 7012d998f1 json_db: add back profiler to _write
follow-up 411098f293
2023-08-15 17:04:49 +00:00
ThomasV 97768a13b9 follow-up previous commit 2023-06-24 13:00:15 +02:00
ThomasV b8b36c7c30 follow-up prev: fix flake8 test 2023-06-24 12:56:55 +02:00
ThomasV 411098f293 move methods from wallet_db to json_db
the goal of this commit is to call JsonDB.__init__ with data,
not an empty dict
2023-06-24 12:45:07 +02:00
ThomasV 759eaf1cf5 json_db: register extra types outside of constructor 2023-06-23 12:16:14 +02:00
ThomasV 606c51bc4e follow-up previous commit 2023-06-18 13:49:56 +02:00
ThomasV 295734fc53 storage: encapsulate type conversions of stored objects using
decorators (instead of overloading JsonDB._convert_dict and
 _convert_value)
 - stored_in for elements of a StoreDict
 - stored_as for singletons
 - extra register methods are defined for key conversions

This commit was adapted from the jsonpatch branch
2023-06-18 13:08:57 +02:00
SomberNight 312f2641e7 don't use bare except
use "except Exception", or if really needed explicitly "except BaseException"
2023-04-24 12:58:01 +00:00
ThomasV 015648c4e5 Move get_dict method from wallet_db to json_db.
Define overloaded methods: _convert_dict _convert_values
2022-11-03 12:41:49 +01:00
bitromortac 6915e3cb10 lnpeer+wallet: use channel type for channel open
* channel_type is put into storage, serialized as int and
  deserialized as ChannelType
* check for static_remotekey is done via channel type
2022-01-20 16:47:48 +01:00
ThomasV eadd1bebb2 Do not convert StoredDict keys to str.
Instead, convert json keys to int when storage is read.
2021-09-20 14:49:59 +02:00
Malcolm Smith 67ae678137 storage/db: use faster JSON encoder settings when wallet is encrypted
The standard json module has an optimized C encoder, but that doesn't
currently support indentation. So if you request indentation, it falls
back on the slower Python encoder.

Readability doesn't matter for encrypted wallets, so this disables
indentation when the wallet is encrypted.

-----

based on https://github.com/Electron-Cash/Electron-Cash/commit/b2399b6a3e3e39ddd82ffc432bb1ca07f2aab454

For a large encrypted wallet, compare:
before change:
JsonDB.dump 1.3153 sec
zlib.compress 1.281 sec
ECPubkey.encrypt_message 0.1744 sec

after change:
JsonDB.dump 0.5059 sec
zlib.compress 1.3120 sec
ECPubkey.encrypt_message 0.1630 sec

Co-authored-by: SomberNight <somber.night@protonmail.com>
2021-01-06 21:14:56 +01:00
SomberNight a7199696d3 json_db: exempt keystore from StoredDict conversion
The keystore logic would need to be significantly changed to nicely
interoperate with StoredDict/json_db logic.
(just see KeyStore.__init__() and KeyStore.dump())
For now we exempt the keystore from the recursive StoredDict conversion, as
it is a smaller change that is also easier to review for correctness.

fixes #6066
fixes #6401

also reverts 2d3c2eeea9 (which was an even hackier workaround for #6066)
2020-09-04 16:11:01 +02:00
SomberNight d5f368c584 LN invoices: support msat precision
fixes #6250
2020-06-22 22:48:13 +02:00
SomberNight 9a70b79eea follow-up prev: try to handle json db int key madness :/ 2020-03-17 20:32:27 +01:00
ThomasV e8ee4250d9 Do not save new channels before they are added to lnworker 2020-02-24 09:44:35 +01:00
ThomasV dbceed2647 Restructure wallet storage:
- Perform json deserializations in wallet_db
 - use StoredDict class that keeps tracks of its modifications
2020-02-04 13:35:58 +01:00
ThomasV 7507942b7a (minor) json_db: add file header, fix formatting 2020-02-04 12:43:04 +01:00
ThomasV 63963323be storage: take the DB lock when writing to disk. 2020-02-03 17:08:34 +01:00
ThomasV 149cd9598a Separate JsonDB and WalletDB 2020-02-03 12:36:07 +01:00
ThomasV aa51df0a1a Use attr.s for Feeupdates and Outpoints
Storage upgrade to version 23
2020-02-01 16:45:19 +01:00
SomberNight 18c6451518 json_db: only deserialize transactions on-demand 2020-01-19 05:49:12 +01:00
SomberNight d2f132738a wallet: only select mature coins by default
this is a regression from #5721

Removed the `TxInput.is_coinbase` method as I think it is a confusing API,
instead we now have `TxInput.is_coinbase_input` and `TxInput.is_coinbase_output`.

related #5872
2020-01-02 00:43:49 +01:00
SomberNight 72491bdf18 synchronizer: request tx from server if we only have partial local tx
Note that there is a slight distinction between
`not tx.is_complete()` and `isinstance(tx, PartialTransaction)`,
which is that technically you can have a PSBT that is already complete
but was not yet converted to a standard bitcoin tx.
2019-12-16 21:15:20 +01:00
SomberNight 01fc048484 CLI: properly auto-upgrade storage when needed even if storage-encrypted
previously commands would error if user had an encrypted storage that needed upgrading
2019-12-15 20:12:51 +01:00
ThomasV 2c6a1f55fb Merge pull request #5825 from SomberNight/201912_local_tx_can_be_partial
wallet: allow saving partial tx as local (if it has a txid)
2019-12-15 16:40:46 +01:00
SomberNight 869a728317 wallet: use abstract base classes 2019-12-10 19:34:44 +01:00
SomberNight 30dcab0877 wallet: allow saving partial txns as local (but require txid) 2019-12-08 04:32:44 +01:00
SomberNight 8dbbc21aff wallet: better (outgoing) invoice "paid" detection
- no more passing around "invoice" in GUIs, invoice "paid" detection is now handled by wallet logic
- a tx can now pay for multiple invoices
- an invoice can now be paid by multiple txs (through partial payments)
- new data structure in storage: prevouts_by_scripthash
  - type: scripthash -> set of (outpoint, value)
  - also, storage upgrade to build this for existing wallets
2019-11-29 15:06:16 +01:00
ThomasV 61dfcba092 Refactor channel states:
- persisted states are saved
 - state transitions are checked
 - transient states are stored in channel.peer_state
 - new channel states: 'PREOPENING', 'FUNDED' and 'REDEEMED'
 - upgrade storage to version 21
2019-11-22 20:14:54 +01:00
SomberNight e6c841d05f psbt: put fake xpubs into globals. keystores handle xfp/der_prefix missing 2019-11-04 22:24:59 +01:00
SomberNight bafe8a2fff integrate PSBT support natively. WIP 2019-11-04 22:24:36 +01:00
ThomasV 7b828359c6 simplify get_tx_fee 2019-09-12 12:26:49 +02:00
SomberNight 482605edbb wallet: organise get_tx_fee. store calculated fees. storage version 19. 2019-09-12 08:59:27 +02:00
SomberNight b138fff9a5 wallet: txi/txo small clean-up 2019-09-12 04:07:17 +02:00
SomberNight 8c91deb621 storage: better error msg on unsupported seed version 2019-08-09 18:11:50 +02:00
SomberNight c6a54f05f5 wallet: some performance optimisations for get_receiving_addresses
jsondb takes a copy of the whole self.receiving_addresses
good for avoiding race conditions but horrible for performance...
this significantly speeds up at least
- synchronize_sequence, and
- is_beyond_limit (used by Qt AddressList)
2019-06-28 20:20:24 +02:00
SomberNight e3c26d7c7a json_db: fix remove_spent_outpoint
method should make sure prevout_n is str...
also wrote failing test
2019-06-15 03:51:11 +02:00
SomberNight 6bdc6f559c storage: fix bug in convert_version_17
closes #5400
2019-06-06 19:51:37 +02:00
SomberNight 53d189fc7a storage: fix some madness about get_data_ref() and put() interacting badly
previously load_transactions() had to be called before upgrade();
now we reverse this order.

to reproduce/illustrate issue, before this commit:

try running convert_version_17 and convert_version_18
(e.g. see testcase test_upgrade_from_client_2_9_3_old_seeded_with_realistic_history)
and then in qt console:
>> wallet.storage.db.get_data_ref('spent_outpoints') == wallet.storage.db.spent_outpoints
False
>> wallet.storage.db.get_data_ref('verified_tx3') == wallet.storage.db.verified_tx
False
2019-06-06 19:49:06 +02:00
SomberNight fd58a0cb20 json_db: enforce order of 'load_transactions' and 'upgrade'
fixes #5331
2019-05-13 19:21:26 +02:00
SomberNight 3385a94753 logging: basics 2019-05-02 15:19:03 +02:00
ThomasV 56ced1dfd5 let DB handle addresses 2019-03-05 08:20:34 +01:00
SomberNight 514d0ae958 wallet: towards restoring previous performance 2 2019-03-04 18:16:48 +01:00
SomberNight b134f04fef storage: fix convert_version_18 2019-03-01 19:54:09 +01:00
SomberNight 9ecb504739 storage: fix convert_version_17 and add new test case
follow-up 121b8048b0
2019-03-01 19:46:23 +01:00
SomberNight 121b8048b0 json_db: store Transaction objects in memory, not raw hex
to avoid deserializing the same tx multiple times
2019-03-01 17:59:22 +01:00
SomberNight 2ad73050b3 wallet: towards restoring previous performance 2019-03-01 17:59:22 +01:00
ThomasV 2fd4cdcaa9 json_db: add missing lock 2019-02-28 16:11:19 +01:00