Adds a new configvar `WALLET_PARTIAL_WRITES` to enable/disable partial writes for the walletDB.
This is a further restriction on top of the existing restrictions,
e.g. wallet files still need to have file encryption disabled for partial writes.
It defaults to off, so even for unencrypted wallets we disable partial writes for now.
This is used as a stopgap measure until we fix the issues found with the partial writes impl
(see https://github.com/spesmilo/electrum/issues/10000).
Notably verifymessage and decrypt(message) were silently ignoring trailing garbage
or inserted non-base64 characters present in signatures/ciphertext.
(both the CLI commands and in the GUI)
I think it is much cleaner and preferable to treat such signatures/ciphertext as invalid.
In fact I find it surprising that base64.b64decode(validate=False) is the default.
Perhaps we should create a helper function for it that set validate=True and use that.
The return value of f.write and f.seek cannot be compared when using open() in text mode:
```
>>> import os
>>> s = "aá"
>>>
>>> with open("a1", "w", encoding='utf-8') as f:
... a = f.write(s)
... pos = f.seek(0, os.SEEK_END)
... print(a, pos)
...
2 3
>>>
>>> with open("a2", "wb") as f:
... a = f.write(s.encode('utf-8'))
... pos = f.seek(0, os.SEEK_END)
... print(a, pos)
...
3 3
```
Was getting errors on Windows, probably due to `\r\n` vs `\n`?
```
20231010T121334.522573Z | ERROR | util.CallbackManager | cb errored. event='adb_set_up_to_date'. exc=AssertionError((2471475, 2522998))
Traceback (most recent call last):
File "...\electrum\electrum\wallet.py", line 497, in on_event_adb_set_up_to_date
self.save_db()
File "...\electrum\electrum\wallet.py", line 403, in save_db
self.db.write()
File "...\electrum\electrum\json_db.py", line 48, in wrapper
return func(self, *args, **kwargs)
File "...\electrum\electrum\json_db.py", line 389, in write
self._append_pending_changes()
File "...\electrum\electrum\json_db.py", line 48, in wrapper
return func(self, *args, **kwargs)
File "...\electrum\electrum\json_db.py", line 400, in _append_pending_changes
self.storage.append(s)
File "...\electrum\electrum\storage.py", line 110, in append
assert pos == self.pos, (self.pos, pos)
AssertionError: (2471475, 2522998)
```
- partial writes are append only.
- StoredDict objects will append partial writes to the wallet
file when items are added, replaced, removed.
- Lists in the wallet file that have not been registered
as StoredObject are converted to StoredList, which
overloads append() and remove(). Those methods too will
append partial writes to the wallet file.
- Unlike the old jsonpatch branch, this branch does not support
file encryption. Encrypted files always fully rewritten, even
if the change before encryption is a partial write.
Note in particular that check_password_for_directory was not safe to use while the daemon had wallets loaded,
as the same file would have two corresponding Wallet() instances in memory. This was specifically handled in
the kivy GUI, on the caller side, by stopping-before and reloading-after the wallets; but it was dirty to
have the caller handle this.
Re total runtime of WalletDB.write() and file size on disk,
for a large encrypted wallet, compare:
before (zlib level=6):
file size 16_670 KB
JsonDB.dump 0.5099 sec
zlib.compress 1.3280 sec
ECPubkey.encrypt_message 0.1720 sec
after change (zlib level=1):
file size 17_527 KB
JsonDB.dump 0.5344 sec
zlib.compress 0.5320 sec
ECPubkey.encrypt_message 0.1837 sec
The standard json module has an optimized C encoder, but that doesn't
currently support indentation. So if you request indentation, it falls
back on the slower Python encoder.
Readability doesn't matter for encrypted wallets, so this disables
indentation when the wallet is encrypted.
-----
based on https://github.com/Electron-Cash/Electron-Cash/commit/b2399b6a3e3e39ddd82ffc432bb1ca07f2aab454
For a large encrypted wallet, compare:
before change:
JsonDB.dump 1.3153 sec
zlib.compress 1.281 sec
ECPubkey.encrypt_message 0.1744 sec
after change:
JsonDB.dump 0.5059 sec
zlib.compress 1.3120 sec
ECPubkey.encrypt_message 0.1630 sec
Co-authored-by: SomberNight <somber.night@protonmail.com>
previously load_transactions() had to be called before upgrade();
now we reverse this order.
to reproduce/illustrate issue, before this commit:
try running convert_version_17 and convert_version_18
(e.g. see testcase test_upgrade_from_client_2_9_3_old_seeded_with_realistic_history)
and then in qt console:
>> wallet.storage.db.get_data_ref('spent_outpoints') == wallet.storage.db.spent_outpoints
False
>> wallet.storage.db.get_data_ref('verified_tx3') == wallet.storage.db.verified_tx
False
SomberNight