util.callback_mgr.callbacks was not getting properly cleared between tests.
Every time an Abstract_Wallet or an LNWorker (or many other subclasses of EventListener) is instantiated,
self.register_callbacks() is called in __init__, which puts callbacks into util.callback_mgr.callbacks.
These are only cleaned up if we explicitly call Abstract_Wallet.stop() or LNWorker.stop() later, which we usually do not do in the tests.
As a result, when running multiple unit tests in a row, lots of objects created in a given testcase are never GC-ed and leak into subsequent tests. This is not only a memory leak, but wastes compute too: when events are triggered and cbs get called, these old objects also have their cbs called.
After running all (~1061) unit tests, I observe util.callback_mgr.callbacks had 30 events with a total of 3156 callbacks stored.
On my laptop, running all unit tests previously took ~115 sec, and now it takes ~73 sec.
Replace calls to deprecated asyncio.iscoroutinefunction with calls to
inspect.iscoroutinefunction to prevent the following deprecation
warnings from showing up if running with Python 3.14:
```
/home/user/code/electrum-fork/electrum/util.py:1225: DeprecationWarning: 'asyncio.iscoroutinefunction' is deprecated and slated for removal in Python 3.16; use inspect.iscoroutinefunction() instead
assert asyncio.iscoroutinefunction(func), 'func needs to be a coroutine'
/home/user/code/electrum-fork/electrum/util.py:507: DeprecationWarning: 'asyncio.iscoroutinefunction' is deprecated and slated for removal in Python 3.16; use inspect.iscoroutinefunction() instead
if asyncio.iscoroutinefunction(func):
/home/user/code/electrum-fork/electrum/util.py:1246: DeprecationWarning: 'asyncio.iscoroutinefunction' is deprecated and slated for removal in Python 3.16; use inspect.iscoroutinefunction() instead
assert asyncio.iscoroutinefunction(func), 'func needs to be a coroutine'
/home/user/code/electrum-fork/electrum/lnpeer.py:272: DeprecationWarning: 'asyncio.iscoroutinefunction' is deprecated and slated for removal in Python 3.16; use inspect.iscoroutinefunction() instead
assert asyncio.iscoroutinefunction(func), 'func needs to be a coroutine'
/home/user/code/electrum-fork/electrum/util.py:1225: DeprecationWarning: 'asyncio.iscoroutinefunction' is deprecated and slated for removal in Python 3.16; use inspect.iscoroutinefunction() instead
assert asyncio.iscoroutinefunction(func), 'func needs to be a coroutine'
/home/user/code/electrum-fork/electrum/util.py:507: DeprecationWarning: 'asyncio.iscoroutinefunction' is deprecated and slated for removal in Python 3.16; use inspect.iscoroutinefunction() instead
if asyncio.iscoroutinefunction(func):
```
```
28.99 | I | util.DebugMem | Start memscan
29.10 | E | plugin.Plugins |
Traceback (most recent call last):
File "...\electrum\util.py", line 405, in run_jobs
job.run()
File "...\electrum\util.py", line 376, in run
self.mem_stats()
File "...\electrum\util.py", line 368, in mem_stats
if isinstance(obj, class_):
File "...\Python310\lib\abc.py", line 119, in __instancecheck__
return _abc_instancecheck(cls, instance)
File "...\electrum\simple_config.py", line 609, in __getattribute__
raise AttributeError()
AttributeError
```
closes https://github.com/spesmilo/electrum/issues/10119
also:
- wallet.get_onchain_history was broken with from_height/to_height args
- "show_fees" param is and was non-existent. fees are always added to output
- MyEncoder(json.JSONEncoder) changed a bit:
- I am pretty sure cutting the last 3 chars was intended to cut off the seconds
- however that was making incorrect assumptions about what datetime.isoformat() returns
- which depends on whether microsecond precision is available or whether an explicit timezone is set
- this now makes it clear that we want minutes-resolution, but still leaves the timezone-ambiguity
Consider e.g.:
```
class AddressSynchronizer(Logger, EventListener):
[... snip ...]
@event_listener
@with_lock
def on_event_blockchain_updated(self, *args):
self._get_balance_cache = {} # invalidate cache
self.db.put('stored_height', self.get_local_height())
```
was raising:
```
func.__qualname__='with_lock.<locals>.func_wrapper'
Traceback (most recent call last):
File "...\electrum\run_electrum", line 105, in <module>
from electrum.logging import get_logger, configure_logging # import logging submodule first
File "...\electrum\electrum\__init__.py", line 19, in <module>
from .wallet import Wallet
File "...\electrum\electrum\wallet.py", line 70, in <module>
from .address_synchronizer import (
File "...\electrum\electrum\address_synchronizer.py", line 75, in <module>
class AddressSynchronizer(Logger, EventListener):
File "...\electrum\electrum\address_synchronizer.py", line 205, in AddressSynchronizer
def on_event_blockchain_updated(self, *args):
File "...\electrum\electrum\util.py", line 2005, in event_listener
classname, method_name = func.__qualname__.split('.')
ValueError: too many values to unpack (expected 2)
```
When launching an electrum daemon and an electrum cli command quickly after each other,
in a test using a random fresh datadir, they both try to create the same folder hierarchy
and race on the "if not exist: mkdir" commands.
Adds validation and deduplication of the relay urls entered in the QT
settings dialog. This is supposed to prevent malformed or duplicated
relay entries.
Also resets the relays to the default value if no (valid) url
is entered. This prevents the user from getting stuck without relays
(otherwise the user would have to research for relay urls manually if
they don't know any).
We added some code in 0b3a283586
to explicitly hold strong refs for all tasks/futures. At the time I was uncertain if that also solves
GC issues with asyncio.run_coroutine_threadsafe.
ref https://github.com/spesmilo/electrum/pull/9608#issuecomment-2703681663
Looks like it does. run_coroutine_threadsafe *is* going through the custom task factory.
See the unit test.
The somewhat confusing thing is that we need a few event loop iterations for the task factory to run,
due to how run_coroutine_threadsafe is implemented. And also, the task that we will hold as strong ref
in the global set is not the concurrent.futures.Future that run_coroutine_threadsafe returns.
So this commit simply "fixes" the unit test so that it showcases this, and removes related, older, plumbing
from util.py that we now know is no longer needed because of this.
```
$ export ELECTRUM_LINTERS=E9,E101,E129,E273,E274,E703,E71,E722,F5,F6,F7,F8,W191,W29,B
$ export ELECTRUM_LINTERS_IGNORE=B007,B009,B010,B019,B036,F541,F841
$ flake8 . --count --select="$ELECTRUM_LINTERS" --ignore="$ELECTRUM_LINTERS_IGNORE" --show-source --statistics --exclude "*_pb2.py,electrum/_vendor/"
./electrum/commands.py:98:1: F811 redefinition of unused 'format_satoshis' from line 48
def format_satoshis(x):
^
./electrum/commands.py:437:9: F811 redefinition of unused 'Mnemonic' from line 62
from .mnemonic import Mnemonic
^
./electrum/gui/qt/wizard/wallet.py:37:5: F811 redefinition of unused 'Daemon' from line 14
from electrum.daemon import Daemon
^
./electrum/lntransport.py:14:1: F811 redefinition of unused 'Optional' from line 12
from typing import NamedTuple, List, Tuple, Mapping, Optional, TYPE_CHECKING, Union, Dict, Set, Sequence
^
./electrum/lntransport.py:14:1: F811 redefinition of unused 'TYPE_CHECKING' from line 12
from typing import NamedTuple, List, Tuple, Mapping, Optional, TYPE_CHECKING, Union, Dict, Set, Sequence
^
./electrum/plugin.py:966:13: F811 redefinition of unused 'hid' from line 593
import hid
^
./electrum/plugin.py:1040:13: F811 redefinition of unused 'hid' from line 593
import hid
^
./electrum/util.py:44:1: F811 redefinition of unused 'json' from line 26
import json
^
./electrum/util.py:46:1: F811 redefinition of unused 'NamedTuple' from line 29
from typing import NamedTuple, Optional
^
./electrum/util.py:46:1: F811 redefinition of unused 'Optional' from line 29
from typing import NamedTuple, Optional
^
./electrum/util.py:1456:56: F811 redefinition of unused 'traceback' from line 34
async def __aexit__(self, exc_type, exc_value, traceback):
^
./electrum/wallet_db.py:536:9: F811 redefinition of unused 'LOCAL' from line 46
LOCAL = 1
^
./electrum/wallet_db.py:537:9: F811 redefinition of unused 'REMOTE' from line 46
REMOTE = -1
^
./tests/test_bitcoin.py:28:1: F811 redefinition of unused 'bitcoin' from line 9
from electrum import crypto, constants, bitcoin
^
./tests/test_txbatcher.py:11:1: F811 redefinition of unused 'Transaction' from line 7
from electrum.transaction import Transaction, PartialTxInput, PartialTxOutput, TxOutpoint
^
./tests/test_wallet_vertical.py:20:1: F811 redefinition of unused 'Transaction' from line 10
from electrum.transaction import Transaction, PartialTxOutput, tx_from_any, Sighash
^
16 F811 redefinition of unused 'format_satoshis' from line 48
16
```
- on my PC, with Tor Browser running (socks proxy on port 9150), detect_tor_socks_proxy took ~4.01 seconds
- this was because we probed port 9050, 2 sec timeout, 9051, 2 sec timeout, 9150, ~few ms to succeed
- instead we now probe all ports concurrently
allow enable/disable proxy without nuking proxy mode, host and port (explicit enable_proxy config setting),
move tor probe from frontend to backend code, add probe buttons for Qt and QML
- reduce number of methods
- use nametuples instead of dicts
- only two types: OnchainHistoryItem and LightningHistoryItem
- channel open/closes are groups
- move capital gains into separate RPC
* sets the weight of htlc transactions to zero, thereby putting a zero
fee for the htlc transactions
* add inputs to htlc-tx for fee bumping
* switches feature flags
* disable anchor test vectors, which are now partially invalid
The messages are sometimes logged and sometimes shown to the user,
- for logging we might not want to truncate or have higher limits,
- but when shown to the user, we definitely want to truncate the error text.
It is simplest to just do the truncation here, at the lowest level.
Note that we usually prepend the error text with a header e.g. "[DO NOT TRUST THIS MESSAGE]"
and if the error text is too long, this header at the beginning might get "lost" in some way.
Hence we should truncate the error text.
Previously, generally, in case of any error, commands would raise a generic "Exception()" and the CLI/RPC would convert that and return it as `str(e)`.
With this change, we now distinguish "user-facing exceptions" (e.g. "Password required" or "wallet not loaded") and "internal errors" (e.g. bugs).
- for "user-facing exceptions", the behaviour is unchanged
- for "internal errors", we now pass around the traceback (e.g. from daemon server to rpc client) and show it to the user (previously, assuming there was a daemon running, the user could only retrieve the exception from the log of that daemon). These errors use a new jsonrpc error code int (code 2).
As the logic only changes for "internal errors", I deem this change not to be compatibility-breaking.
----------
Examples follow.
Consider the following two commands:
```
@command('')
async def errorgood(self):
from electrum.util import UserFacingException
raise UserFacingException("heyheyhey")
@command('')
async def errorbad(self):
raise Exception("heyheyhey")
```
----------
(before change)
CLI with daemon:
```
$ ./run_electrum --testnet daemon -d
starting daemon (PID 9221)
$ ./run_electrum --testnet errorgood
heyheyhey
$ ./run_electrum --testnet errorbad
heyheyhey
$ ./run_electrum --testnet stop
Daemon stopped
```
CLI without daemon:
```
$ ./run_electrum --testnet -o errorgood
heyheyhey
$ ./run_electrum --testnet -o errorbad
heyheyhey
```
RPC:
```
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorgood","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 1, "message": "heyheyhey"}}
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorbad","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 1, "message": "heyheyhey"}}
```
----------
(after change)
CLI with daemon:
```
$ ./run_electrum --testnet daemon -d
starting daemon (PID 9254)
$ ./run_electrum --testnet errorgood
heyheyhey
$ ./run_electrum --testnet errorbad
(inside daemon): Traceback (most recent call last):
File "/home/user/wspace/electrum/electrum/daemon.py", line 254, in handle
response['result'] = await f(*params)
File "/home/user/wspace/electrum/electrum/daemon.py", line 361, in run_cmdline
result = await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 163, in func_wrapper
return await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 217, in errorbad
raise Exception("heyheyhey")
Exception: heyheyhey
internal error while executing RPC
$ ./run_electrum --testnet stop
Daemon stopped
```
CLI without daemon:
```
$ ./run_electrum --testnet -o errorgood
heyheyhey
$ ./run_electrum --testnet -o errorbad
0.78 | E | __main__ | error running command (without daemon)
Traceback (most recent call last):
File "/home/user/wspace/electrum/./run_electrum", line 534, in handle_cmd
result = fut.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/user/wspace/electrum/./run_electrum", line 255, in run_offline_command
result = await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 163, in func_wrapper
return await func(*args, **kwargs)
File "/home/user/wspace/electrum/electrum/commands.py", line 217, in errorbad
raise Exception("heyheyhey")
Exception: heyheyhey
```
RPC:
```
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorgood","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 1, "message": "heyheyhey"}}
$ curl --data-binary '{"id":"curltext","jsonrpc":"2.0","method":"errorbad","params":[]}' http://user:pass@127.0.0.1:7777
{"id": "curltext", "jsonrpc": "2.0", "error": {"code": 2, "message": "internal error while executing RPC", "data": {"exception": "Exception('heyheyhey')", "traceback": "Traceback (most recent call last):\n File \"/home/user/wspace/electrum/electrum/daemon.py\", line 254, in handle\n response['result'] = await f(*params)\n File \"/home/user/wspace/electrum/electrum/commands.py\", line 163, in func_wrapper\n return await func(*args, **kwargs)\n File \"/home/user/wspace/electrum/electrum/commands.py\", line 217, in errorbad\n raise Exception(\"heyheyhey\")\nException: heyheyhey\n"}}}
```
was getting log spam when running the pycharm debugger in certain cases:
```
3.29 | E | concurrent.futures | exception calling callback for <Future at 0x1d4987bcc70 state=cancelled>
Traceback (most recent call last):
File "...\Python310\lib\concurrent\futures\_base.py", line 342, in _invoke_callbacks
callback(self)
File "...\electrum\electrum\util.py", line 1785, in on_done
if exc := fut_.exception():
File "...\Python310\lib\concurrent\futures\_base.py", line 485, in exception
raise CancelledError()
concurrent.futures._base.CancelledError
```
- rename globals
- also rm hardcoded strings from qml
- use consistent unit names in qml
(previously mixed sat/vB and sat/byte (latter coming from core lib))
