davide/plm-electrumx
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: snapshot of spesmilo/electrumx 2.0.0

Browse Source 
Unmodified source tree of upstream commit 280cb339aa158e099eb936033eecd15719770651
("Merge pull request #389 from SomberNight/202607_release_2_0",
https://github.com/spesmilo/electrumx, release 2.0.0, 3 July 2026).

All Palladium-specific changes are in the commits that follow.
This commit is contained in:
davide committed 2026-10-22 15:03:56 +02:00
commit 384d907c65
271 files changed
+27905
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/ACKNOWLEDGEMENTS
+22
View File
@@ -0,0 +1,22 @@
Thanks to Thomas Voegtlin for creating the Electrum software and
infrastructure and for maintaining it so diligently. Electrum is
probably the best desktop Bitcoin wallet solution for most users. My
faith in it is such that I use Electrum software to store most of my
Bitcoins.
Whilst the vast majority of the code here is my own original work and
includes some new ideas, it is very clear that the general structure
and concept are those of Electrum. Some parts of the code and ideas
of Electrum, some of which it itself took from other projects such as
Abe and pywallet, remain. Thanks to the authors of all the software
this is derived from.
I'd like to thank bauerj, hsmiths and JWU42 of #electrum for their
help and support in testing the versions of ElectrumX prior to 1.0.
Thanks to Daniel Bernstein for daemontools and other software, and to
Matthew Dillon for DragonFlyBSD. They are both deeply inspirational
people.
And of course, thanks to Satoshi for the wonderful creation that is
Bitcoin.
docs/HOWTO.rst
+461
View File
@@ -0,0 +1,461 @@
.. _HOWTO:
=====
HOWTO
=====
Prerequisites
=============
**ElectrumX** should run on any flavour of unix. I have run it
successfully on MacOS and DragonFlyBSD. It won't run out-of-the-box
on Windows, but the changes required to make it do so should be
small - pull requests are welcome.
================ ========================
Package Notes
================ ========================
Python3 ElectrumX uses asyncio. Python version >= 3.10 is
**required**.
`aiohttp`_ Python library for asynchronous HTTP. Version >=
2.0 required.
DB Engine A database engine package is required; two are
supported (see `Database Engine`_ below).
================ ========================
Some coins need an additional package, typically for their block hash
functions. For example, `dash_hash`_ is required for DASH. Scrypt coins
require a Python interpreter compiled and/or linked with OpenSSL 1.1.0
or higher.
You **must** be running a non-pruning bitcoin daemon with::
txindex=1
set in its configuration file. If you have an existing installation
of bitcoind and have not previously set this you will need to reindex
the blockchain with::
bitcoind -reindex
which can take some time.
While not a requirement for running ElectrumX, it is intended to be
run with supervisor software such as Daniel Bernstein's
`daemontools`_, Gerrit Pape's `runit`_ package or :command:`systemd`.
These make administration of secure unix servers very easy, and I
strongly recommend you install one of these and familiarise yourself
with them. The instructions below and sample run scripts assume
``daemontools``; adapting to ``runit`` should be trivial for someone
used to either.
When building the database from the genesis block, ElectrumX has to
flush large quantities of data to disk and its DB. You will have a
better experience if the database directory is on an SSD than on an
HDD. Currently to around height 611,600 of the Bitcoin blockchain the
final size of the leveldb database, and other ElectrumX file metadata
comes to just over 46.9GB (43.7 GiB). LevelDB needs a bit more for
brief periods, and the block chain is only getting longer, so I would
recommend having at least 70-80GB of free space before starting.
Database Engine
===============
You can choose from LevelDB and RocksDB to store transaction
information on disk. On an SSD, RocksDB performs better than LevelDB.
We tried to support LMDB, but its history write performance
was much worse.
You will need to install one of:
+ `plyvel <https://plyvel.readthedocs.io/en/latest/installation.html>`_ for LevelDB.
``apt install libleveldb-dev build-essential``
``pip3 install plyvel`` , or use the ``[leveldb]`` extra install option to ElectrumX.
+ `rocksdb-ng <https://pypi.org/project/rocksdb-ng>`_ for RocksDB
(Should work with RocksDB versions 8-10)
``apt install librocksdb-dev build-essential pkg-config``
``pip3 install rocksdb-ng`` , or use the ``[rocksdb]`` extra install option to ElectrumX.
+ `python-rocksdb <https://pypi.python.org/pypi/python-rocksdb>`_ for an unmaintained version
that doesn't work with recent releases of RocksDB (only <7.0)
Running
=======
Install the prerequisites above.
Check out the code from Github::
git clone https://github.com/spesmilo/electrumx.git
cd electrumx
You can install with::
pip3 install .
There are many extra Python dependencies available to fit the needs of your
system or coins. For example, to install the RocksDB dependencies and a faster
JSON (de)serialization library::
pip3 install ".[rocksdb,orjson]"
see pyproject.toml's ``project.optional-dependencies`` for a complete list.
You can also run the code from the source tree or a copy of it.
You should create a standard user account to run the server under;
your own is probably adequate unless paranoid. The paranoid might
also want to create another user account for the daemontools logging
process. The sample scripts and these instructions assume it is all
under one account which I have called ``electrumx``.
Next create a directory where the database will be stored and make it
writeable by the ``electrumx`` account. I recommend this directory
live on an SSD::
mkdir /path/to/db_directory
chown electrumx /path/to/db_directory
Process limits
--------------
The ElectrumX process needs a large open file limit. On Linux systems,
the default (:command:`ulimit -n`) (soft) open file limit is usually 1,024.
ElectrumX tries to raise this automatically during startup to the hard limit,
which is usually 100,000+. If that fails (which would get logged),
you might have to manually increase the limit.
During sync it should not need more than about 1,024 open files. When
serving it will use approximately 256 for LevelDB plus the number of
incoming connections. It is not unusual to have 1,000 to 2,000
connections being served, so I suggest you set your open files limit
to at least 2,500.
Note that setting the limit in your shell does *NOT* affect ElectrumX
unless you are invoking ElectrumX directly from your shell. If you
are using :command:`systemd`, you need to set it in the
:file:`.service` file (see `contrib/systemd/electrumx.service`_).
Using daemontools
-----------------
Next create a daemontools service directory; this only holds symlinks
(see daemontools documentation). The :command:`svscan` program will
ensure the servers in the directory are running by launching a
:command:`supervise` supervisor for the server and another for its
logging process. You can run :command:`svscan` under the *electrumx*
account if that is the only one involved (server and logger) otherwise
it will need to run as root so that the user can be switched to
electrumx.
Assuming this directory is called :file:`service`, you would do one
of::
mkdir /service # If running svscan as root
mkdir ~/service # As electrumx if running svscan as that a/c
Next create a directory to hold the scripts that the
:command:`supervise` process spawned by :command:`svscan` will run -
this directory must be readable by the :command:`svscan` process.
Suppose this directory is called :file:`scripts`, you might do::
mkdir -p ~/scripts/electrumx
Then copy the all sample scripts from the ElectrumX source tree there::
cp -R /path/to/repo/electrumx/contrib/daemontools ~/scripts/electrumx
This copies 3 things: the top level server run script, a :file:`log/`
directory with the logger :command:`run` script, an :file:`env/`
directory.
You need to configure the :ref:`environment variables <environment>`
under :file:`env/` to your setup. ElectrumX server currently takes no
command line arguments; all of its configuration is taken from its
environment which is set up according to :file:`env/` directory (see
:manpage:`envdir` man page). Finally you need to change the
:command:`log/run` script to use the directory where you want the logs
to be written by multilog. The directory need not exist as
:command:`multilog` will create it, but its parent directory must
exist.
Now start the :command:`svscan` process. This will not do much as the
service directory is still empty::
svscan ~/service & disown
svscan is now waiting for services to be added to the directory::
cd ~/service
ln -s ~/scripts/electrumx electrumx
Creating the symlink will kick off the server process almost immediately.
You can see its logs with::
tail -F /path/to/log/dir/current | tai64nlocal
Using systemd
-------------
This repository contains a sample systemd unit file that you can use
to setup ElectrumX with systemd. Simply copy it to
:file:`/etc/systemd/system`::
cp contrib/systemd/electrumx.service /etc/systemd/system/
The sample unit file assumes that the repository is located at
:file:`/home/electrumx/electrumx`. If that differs on your system, you
need to change the unit file accordingly.
You need to set a few :ref:`environment variables <environment>` in
:file:`/etc/electrumx.conf`.
Now you can start ElectrumX using :command:`systemctl`::
systemctl start electrumx
You can use :command:`journalctl` to check the log output::
journalctl -u electrumx -f
Once configured you may want to start ElectrumX at boot::
systemctl enable electrumx
.. Warning:: systemd is aggressive in forcibly shutting down
processes. Depending on your hardware, ElectrumX can need several
minutes to flush cached data to disk during initial sync. You
should set TimeoutStopSec to *at least* 10 mins in your
:file:`.service` file.
Installing on Raspberry Pi 3
----------------------------
To install on the Raspberry Pi 3 you will need to update to the
``stretch`` distribution. See the full procedure in
`contrib/raspberrypi3/install_electrumx.sh`_.
See also `contrib/raspberrypi3/run_electrumx.sh`_ for an easy way to
configure and launch electrumx.
Sync Progress
=============
Time taken to index the blockchain depends on your hardware of course.
As Python is single-threaded most of the time only 1 core is kept
busy. ElectrumX uses Python's :mod:`asyncio` to prefill a cache of
future blocks asynchronously to keep the CPU busy processing the chain
without pausing.
Consequently there will probably be only a minor boost in performance
if the daemon is on the same host. It may even be beneficial to have
the daemon on a *separate* machine so the machine doing the indexing
has its caches and disk I/O tuned to that task only.
The :envvar:`CACHE_MB` environment variable controls the total cache
size ElectrumX uses; see :ref:`here <CACHE>` for caveats.
Here is my experience with the codebase of early 2017 (the current
codebase is faster), to given heights and rough wall-time. The period
from heights 363,000 to 378,000 is the most sluggish::
Machine A Machine B
181,000 25m 00s 5m 30s
283,500 1h 00m
321,800 1h 40m
357,000 12h 32m 2h 41m
386,000 21h 56m 4h 25m
414,200 1d 12h 29m 6h 30m
447,168 2d 13h 20m 9h 47m
*Machine A*: a low-spec 2011 1.6GHz AMD E-350 dual-core fanless CPU,
8GB RAM and a DragonFlyBSD UFS filesystem on an SSD. It requests
blocks over the LAN from a bitcoind on machine B. :envvar:`DB_CACHE`
the default of 1,200. LevelDB.
*Machine B*: a late 2012 iMac running Sierra 10.12.2, 2.9GHz quad-core
Intel i5 CPU with an HDD and 24GB RAM. Running bitcoind on the same
machine. :envvar:`DB_CACHE` set to 1,800. LevelDB.
For chains other than bitcoin-mainnet synchronization should be much
faster.
.. note:: ElectrumX will not serve normal client connections until it
has fully synchronized and caught up with your daemon.
However LocalRPC connections are served at all times.
Terminating ElectrumX
=====================
The preferred way to terminate the server process is to send it the
``stop`` RPC command::
electrumx_rpc stop
or alternatively on Unix the ``INT`` or ``TERM`` signals. For a
daemontools supervised process this can be done by bringing it down
like so::
svc -d ~/service/electrumx
ElectrumX will note receipt of the signals in the logs, and ensure the
block chain index is flushed to disk before terminating. You should
be patient as flushing data to disk can take many minutes.
ElectrumX uses the transaction functionality, with fsync enabled, of
the databases. I have written it with the intent that, to the extent
the atomicity guarantees are upheld by the DB software, the operating
system, and the hardware, the database should not get corrupted even
if the ElectrumX process if forcibly killed or there is loss of power.
The worst case should be having to restart indexing from the most
recent UTXO flush.
Once the process has terminated, you can start it up again with::
svc -u ~/service/electrumx
You can see the status of a running service with::
svstat ~/service/electrumx
:command:`svscan` can of course handle multiple services
simultaneously from the same service directory, such as a testnet or
altcoin server. See the man pages of these various commands for more
information.
Understanding the Logs
======================
You can see the logs usefully like so::
tail -F /path/to/log/dir/current | tai64nlocal
Here is typical log output on startup::
INFO:BlockProcessor:switching current directory to /crucial/server-good
INFO:BlockProcessor:using leveldb for DB backend
INFO:BlockProcessor:created new database
INFO:BlockProcessor:creating metadata diretcory
INFO:BlockProcessor:software version: ElectrumX 0.10.2
INFO:BlockProcessor:DB version: 5
INFO:BlockProcessor:coin: Bitcoin
INFO:BlockProcessor:network: mainnet
INFO:BlockProcessor:height: -1
INFO:BlockProcessor:tip: 0000000000000000000000000000000000000000000000000000000000000000
INFO:BlockProcessor:tx count: 0
INFO:BlockProcessor:sync time so far: 0d 00h 00m 00s
INFO:BlockProcessor:reorg limit is 200 blocks
INFO:Daemon:daemon at 192.168.0.2:8332/
INFO:BlockProcessor:flushing DB cache at 1,200 MB
INFO:Controller:RPC server listening on localhost:8000
INFO:Prefetcher:catching up to daemon height 447,187...
INFO:Prefetcher:verified genesis block with hash 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
INFO:BlockProcessor:our height: 9 daemon: 447,187 UTXOs 0MB hist 0MB
INFO:BlockProcessor:our height: 52,509 daemon: 447,187 UTXOs 9MB hist 14MB
INFO:BlockProcessor:our height: 85,009 daemon: 447,187 UTXOs 12MB hist 31MB
INFO:BlockProcessor:our height: 102,384 daemon: 447,187 UTXOs 15MB hist 47MB
[...]
INFO:BlockProcessor:our height: 133,375 daemon: 447,187 UTXOs 80MB hist 222MB
INFO:BlockProcessor:our height: 134,692 daemon: 447,187 UTXOs 96MB hist 250MB
INFO:BlockProcessor:flushed to FS in 0.7s
INFO:BlockProcessor:flushed history in 16.3s for 1,124,512 addrs
INFO:BlockProcessor:flush #1 took 18.7s. Height 134,692 txs: 941,963
INFO:BlockProcessor:tx/sec since genesis: 2,399, since last flush: 2,400
INFO:BlockProcessor:sync time: 0d 00h 06m 32s ETA: 1d 13h 03m 42s
Under normal operation these cache stats repeat once or twice a
minute. UTXO flushes can take several minutes and look like this::
INFO:BlockProcessor:our height: 378,745 daemon: 447,332 UTXOs 1,013MB hist 184MB
INFO:BlockProcessor:our height: 378,787 daemon: 447,332 UTXOs 1,014MB hist 194MB
INFO:BlockProcessor:flushed to FS in 0.3s
INFO:BlockProcessor:flushed history in 13.4s for 934,933 addrs
INFO:BlockProcessor:flushed 6,403 blocks with 5,879,440 txs, 2,920,524 UTXO adds, 3,646,572 spends in 93.1s, committing...
INFO:BlockProcessor:flush #120 took 226.4s. Height 378,787 txs: 87,695,588
INFO:BlockProcessor:tx/sec since genesis: 1,280, since last flush: 359
INFO:BlockProcessor:sync t ime: 0d 19h 01m 06s ETA: 3d 21h 17m 52s
INFO:BlockProcessor:our height: 378,812 daemon: 447,334 UTXOs 10MB hist 10MB
The ETA shown is just a rough guide and in the short term can be quite
volatile. It tends to be a little optimistic at first; once you get
to height 280,000 is should be fairly accurate.
.. _SSL certificates:
Creating a self-signed SSL certificate
======================================
These instructions are based on those of the ``electrum-server``
documentation.
To run an SSL server you need to generate a self-signed certificate
using openssl. Alternatively you could not set :envvar:`SSL_PORT` in
the environment and not serve over SSL, but this is not recommended.
Use the sample code below to create a self-signed cert with a
recommended validity of 5 years. You may supply any information for
your sign request to identify your server. They are not currently
checked by the client except for the validity date. When asked for a
challenge password just leave it empty and press enter::
$ openssl genrsa -out server.key 2048
$ openssl req -new -key server.key -out server.csr
...
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:California
Common Name (eg, YOUR name) []: electrum-server.tld
...
A challenge password []:
...
$ openssl x509 -req -days 1825 -in server.csr -signkey server.key -out server.crt
The :file:`server.crt` file goes in :envvar:`SSL_CERTFILE` and
:file:`server.key` in :envvar:`SSL_KEYFILE` in the server process's
environment.
Starting with Electrum 1.9, the client will learn and locally cache
the SSL certificate for your server upon the first request to prevent
man-in-the middle attacks for all further connections.
If your certificate is lost or expires on the server side, you will
need to run your server with a different server name and a new
certificate. Therefore it's a good idea to make an offline backup
copy of your certificate and key in case you need to restore them.
Running on a privileged port
============================
You may choose to run electrumx on a different port than 50001
/ 50002. If you choose a privileged port ( < 1024 ) it makes sense to
make use of a iptables NAT rule.
An example, which will forward Port 110 to the internal port 50002 follows::
iptables -t nat -A PREROUTING -p tcp --dport 110 -j DNAT --to-destination 127.0.0.1:50002
You can then set the port as follows and advertise the service externally on the privileged port::
REPORT_SSL_PORT=110
.. _`contrib/systemd/electrumx.service`: https://github.com/spesmilo/electrumx/blob/master/contrib/systemd/electrumx.service
.. _`daemontools`: http://cr.yp.to/daemontools.html
.. _`runit`: http://smarden.org/runit/index.html
.. _`aiohttp`: https://pypi.python.org/pypi/aiohttp
.. _`dash_hash`: https://pypi.python.org/pypi/dash_hash
.. _`contrib/raspberrypi3/install_electrumx.sh`: https://github.com/spesmilo/electrumx/blob/master/contrib/raspberrypi3/install_electrumx.sh
.. _`contrib/raspberrypi3/run_electrumx.sh`: https://github.com/spesmilo/electrumx/blob/master/contrib/raspberrypi3/run_electrumx.sh
docs/Makefile
+20
View File
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS = -W
SPHINXBUILD = sphinx-build
SPHINXPROJ = ElectrumX
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/PERFORMANCE-NOTES
+45
View File
@@ -0,0 +1,45 @@
Just some notes on performance with Python 3.5. We are taking these into
account in the code.
- 60% faster to create lists with [] list comprehensions than tuples
or lists with tuple(), list(). Of those list is 10% faster than
tuple.
- however when not initializing from a generator, a fixed-length tuple
is at least 80% faster than a list.
- an implicit default argument is ~5% faster than passing the default
explicitly
- using a local variable x rather than self.x in loops and list
comprehensions is over 50% faster
- struct.pack, struct.unpack are over 60% faster than int.to_bytes and
int.from_bytes. They are faster little endian (presumably because
it matches the host) than big endian regardless of length. Furthermore,
using stored packing and unpacking methods from Struct classes is faster
than using the flexible-format struct.[un]pack equivalents.
After storing the Struct('<Q').unpack_from function as unpack_uint64_from,
later calls to unpack_uint64_from(b, 0) are about 30% faster than calls to
unpack_from('<Q', b, 0).
- single-item list and tuple unpacking. Suppose b = (1, )
a, = b is a about 0.4% faster than (a,) = b
and about 45% faster than a = b[0]
- multiple assignment is faster using tuples only for 3 or more items
- retrieving a previously stored length of a bytes object can be over 200%
faster than a new call to len(b)
- The cpython and pypy stdlibs use a Python-based JSON serializer/deserializer,
which is stable and easy to use and install, but somewhat slower than
system-compiled variants. ElectrumX now looks for known popular
C/Rust-based JSON libraries for Python. If it finds one,
it's used instead of the stdlib module for improved performance.
To install a known-compatible system-compiled JSON lib, install ElectrumX
with the "orjson" extras parameter.
docs/architecture.rst
+83
View File
@@ -0,0 +1,83 @@
Architecture
============
.. image:: https://docs.google.com/drawings/d/1Su_DR2c8__-4phm12hAzV65fL2tNm_1IhKr4XivkW6Q/pub?w=720&h=540
:target: https://docs.google.com/drawings/d/1Su_DR2c8__-4phm12hAzV65fL2tNm_1IhKr4XivkW6Q/pub?w=960&h=720
Env
---
Holds configuration taken from the environment, with appropriate
defaulting appropriately. Generally passed to the constructor of
other components which take their settings from it.
Controller
----------
The central part of the server process initialising and coordinating
all the others. Manages resource usage.
LocalRPC
--------
Handles local JSON RPC connections querying ElectrumX server state.
Started when the ElectrumX process starts.
ElectrumX
---------
Handles JSON Electrum client connections over TCP or SSL. One
instance per client session. Should be the only component concerned
with the details of the Electrum wire protocol.
Not started until the Block Processor has caught up with bitcoind.
Daemon
------
Encapsulates the RPC wire protocol with bitcoind for the whole server.
Transparently handles temporary bitcoind connection errors, and fails
over if necessary.
Notifies the Mempool when the list of mempool transaction hashes is
updated.
Block Processor
---------------
Responsible for managing block chain state (UTXO set, history,
transaction and undo information) and for handling block chain
reorganisations.
When caught up, processes new blocks as they are found, and flushes
the updates to the Database immediately.
When syncing uses caches for in-memory state updates since the prior
flush. Occasionally flushes state to the storage layer when caches
get large.
Prefetcher
----------
Cooperates with the Block Processor to asynchronously prefetch blocks
from bitcoind. Once it has caught up it additionally asks the Daemon
to refresh its view of bitcoind's mempool transaction hashes. Serves
blocks to the Block Processor via a queue.
Mempool
-------
Handles all the details of maintaining a representation of bitcoind's
mempool state. Obtains the list of current mempool transaction hashes
from the Daemon when notified by the Prefetcher.
Notifies the Controller that addresses have been touched when the
mempool refreshes (or implicitly when a new block is found).
Database
--------
The underlying data store, made up of the DB backend (such as
`leveldb`) and the host filesystem.
docs/authors.rst
+20
View File
@@ -0,0 +1,20 @@
.. _Authors:
Authors
=======
* Neil Booth
Original creator, past maintainer.
* Electrum developers
Current maintainers.
* Johann Bauer
Backend DB abstraction.
* John Jegutanis
Alt-chain integrations.
docs/changelog.rst
+431
View File
@@ -0,0 +1,431 @@
===========
ChangeLog
===========
Version 2.0.0 (03 July 2026)
==============================
TL;DR:
- new DB schema, so you need to rebuild from scratch. No more "DB::flush_count overflow".
- Bitcoin Core >=31 is now required, with `txospenderindex=1`
- added support for Electrum Protocol 1.7
- running with free-threaded CPython is now (experimentally) supported, and is much faster
-----
* Breaking changes to on-disk database. Server operators need to manually delete old DB,
no migration path. New DB will be rebuilt while rescanning again from genesis.
- in return, no more manual "history compaction" and unexpected downtimes every year:
no more "DB::flush_count overflow" (`spesmilo/electrumx#88`_)
- the size of the new DB is comparable to the old one, however:
* We now require Bitcoin Core to have :code:`txospenderindex=1` (added in Bitcoin Core 31)
in addition to :code:`txindex=1`. This is needed to serve the :code:`blockchain.outpoint.subscribe` RPC
added in Electrum Protocol 1.7.
For reference, on Bitcoin mainnet around height=950k,
- the ElectrumX db uses around 122 GiB (using LevelDB, roughly same for both of e-x 1.x and 2.0),
- note: using RocksDB 9, it is a bit smaller, around 106 GiB
- :code:`.bitcoin/blocks/` uses 788 GiB,
- :code:`.bitcoin/chainstate/` uses 12 GiB,
- :code:`.bitcoin/indexes/txindex/` uses 66 GiB,
- :code:`.bitcoin/indexes/txospenderindex/` uses 88 GiB (new!)
- This also means that we now require Bitcoin Core 31.0 or newer (for :code:`COIN=Bitcoin`).
* env: the previously optional envvar :code:`DB_ENGINE`, is now mandatory.
- choose either :code:`leveldb` or :code:`rocksdb`
- in ElectrumX 1.x versions, the default was leveldb.
You need to install the appropriate dependencies for your engine, see the :code:`[leveldb]` and
:code:`[rocksdb]` pip extras and the "Database Engine" section of the HOWTO.
Previously RocksDB was difficult to use as the python bindings for it have been unmaintained for years.
By the super-slop-powers of LLMs, we revived the python bindings as :code:`rocksdb-ng` and made it compatible with modern
cpython, cython, and rocksdb. The changes are reviewable and not really slop. (see `spesmilo/electrumx#347`_)
rocksdb-ng (starting with version 2.3) is also compatible with `free-threaded python`_, see "performance" section.
LevelDB was written with HDDs in mind. RocksDB is more modern
and on an SSD takes around ~25% less time than LevelDB to sync from genesis.
Given that this release contains breaking DB changes and a resync is needed anyway, maybe try out RocksDB :)
* performance:
- the BlockProcessor is now massively multi-threaded, both inside a single block and across a span of blocks.
This can mainly only be taken advantage of if the GIL (Global Interpreter Lock) is disabled, that is,
when using a `free-threaded python`_ interpreter: e.g. cpython :code:`3.14t`, compiled with :code:`--disable-gil`.
On a decent modern machine with a multi-core CPU, this has been measured to cut initial sync time
from genesis to a bit less than half. OTOH, much more RAM is required for the parallel processing
(example: approx 2 GB when syncing with GIL, 8 GB when syncing without GIL).
- processing a single block at the tip, when already caught up, is expected to take around half the time,
regardless of the GIL. Still, without the GIL, it is even faster.
- for more details, see `spesmilo/electrumx#369`_
- the (block)Prefetcher and the Mempool code are now partially event-based (but still also polling):
e.g. a mempool update is triggered as soon as processing a new block begins (and part of the work
is done concurrently)
- json (de)serialization can now optionally use the Rust-based
:code:`orjson` library (via the :code:`orjson` pip extra), replacing the
previous :code:`ujson` and :code:`rapidjson` extras.
* Dockerfiles: the existing Dockerfile was updated and cleaned up a bit, and it serves
as a simple minimal example to show how to run ElectrumX. There is now also a second Dockerfile
that sets up a free-threaded python for a beefy server to utilise the above-mentioned performance gains.
See them in :code:`contrib/`. (also see the new runtime logline "Python GIL enabled" to verify effect)
* protocol:
- new: implement electrum protocol version 1.7 (`spesmilo/electrum-protocol#2`_).
The min supported protocol version remains 1.4, the max is now 1.7.
* LocalRPC: new command: "inspect_session" (`spesmilo/electrumx#361`_).
Version 1.20.0 (03 June 2026)
=============================
Small fixes and general maintenance.
After this release, the master branch will move towards a 2.0 release, which will require
a resync from genesis. So for a while, master might be less stable than usual.
* fix: lib/tx.py: vsize calculation was off by 6-8 vbytes (`spesmilo/electrumx#328`_).
Added tests. The discrepancy was externally observable via the `mempool.get_fee_histogram` RPC.
* fix: don't get stuck during shutdown waiting for SessionManager to stop (`spesmilo/electrumx#339`_)
* fix: protocol: name of arg for broadcast_package RPC: tx_package->raw_txs (`c9b72e7e`_)
* changed: peer discovery: allow short chain splits (`spesmilo/electrumx#351`_)
* changed: rm 'attrs' as a direct dependency, use stdlib `dataclasses` instead (`spesmilo/electrumx#345`_)
* new: daemon: check early "txindex" is enabled for bitcoind, error if missing (`spesmilo/electrumx#349`_)
* new: coins: add Bitcoin mutinynet (`spesmilo/electrumx#352`_)
* maintenance: coins: some clean-up (`spesmilo/electrumx#344`_, `spesmilo/electrumx#348`_, ...)
* maintenance: moved CI from Cirrus CI to Github Actions (`spesmilo/electrumx#340`_)
(We also made some changes to make RocksDB practical and nicer to use, but if you currently use LevelDB,
maybe consider waiting for 2.0 before you resync the DB)
Version 1.19.0 (11 Nov 2025)
=============================
* protocol:
- new: implement electrum protocol version 1.6 (`spesmilo/electrumx#317`_, ...)
(`spesmilo/electrum-protocol#6`_).
The min supported protocol version remains 1.4, the max is now 1.6.
* coins:
- changed: for `COIN=Bitcoin`, there is now a min required bitcoind version:
Bitcoin Core 28.0 (or Knots 28) (`spesmilo/electrumx#316`_).
This is due to protocol 1.6 requiring a bitcoind with working `submitpackage` RPC.
* security:
- fix DOS vector: if MAX_SESSIONS was reached, when using python 3.12+, the server
stopped accepting new incoming sessions until full-restart (`spesmilo/electrumx#312`_)
* session:
- changed: add warmup budget to PaddedRSTransport (`spesmilo/electrumx#323`_):
this is a small relaxation of the traffic analysis countermeasures added in 1.18,
where the first 1024 bytes we send are now exempt from buffering (and hence delays)
- fix ReplyAndDisconnect for PaddedRSTransport: flush message buffer (`spesmilo/electrumx#322`_)
* env:
- rm DROP_CLIENT_UNKNOWN opt-in env var: this behaviour is now mandated by protocol 1.6
and we just always enforce it (~small breaking change, affected clients would have
already been considered misbehaving)
* misc:
- new: try to raise `ulimit -n` open file limit automatically at startup
(`spesmilo/electrumx#326`_)
Version 1.18.0 (14 June 2025)
=============================
* protocol:
- add basic countermeasures against traffic analysis by padding the jsonrpc payload
with whitespaces to have ~uniform-size TCP packets, and by artificially delaying
sending messages a bit. (`spesmilo/electrumx#301`_)
* dependencies:
- bump required aiorpcx to >=0.25.0,<0.26
Version 1.17.0 (16 Apr 2025)
============================
Long time no see!
* coins:
- rm support for legacy coin name "BitcoinSegwit" (`ca59151d`_)
(users should just change their config files to :code:`COIN=Bitcoin`)
- add support for Bitcoin Signet (`spesmilo/electrumx#122`_)
- add support for Bitcoin Testnet4 (`spesmilo/electrumx#273`_)
* dependencies:
- bump required python to >=3.10
- bump required aiorpcx to >=0.23.0,<0.25
- rm "pylru" dep, instead bundle stripped-down "cachetools" (`spesmilo/electrumx#248`_)
* session: log IP address of excessive resusage sessions (`spesmilo/electrumx#75`_)
* robustness:
- session: more robust session handling (`25339328`_, `8198ab99`_)
- daemon: catch RPC_PARSE_ERROR from bitcoind and simply retry (`31489365`_)
* env:
- add MAX_RECV environment variable (`a0d3053f`_)
- increase default for MAX_SEND (`2969ab11`_)
* protocol: increase resolution of :code:`mempool.get_fee_histogram` (`265a5a87`_)
* LocalRPC:
- add --timeout arg (`5f4dd2cd`_)
- add commands to help debug memory leaks (`f5582b29`_)
* build/maintenance:
- add scripts for reproducible build (`0ba87447`_)
- migrate from setup.py to pyproject.toml
- refactor file layout flat->src (`spesmilo/electrumx#298`_)
* some internal refactors in preparation for future changes
Version 1.16.0 (10 Dec 2020)
============================
Note: this is the first release since forking from kyuupichan/electrumx.
kyuupichan has the :code:`electrumx` name on PyPI, so we needed a new name there.
We are using the `e-x <https://pypi.org/project/e-x/>`_ name on PyPI, so you can
install this package via e.g. :code:`pip install e-x`.
* security: a vulnerability has been fixed that allowed a remote attacker to
crash electrumx if peer discovery was enabled (`#22`_)
* fixed some peer-discovery-related bugs (e.g. `#35`_)
* ENV: when using Bitcoin, the COIN ENV var can now be set to :code:`Bitcoin`.
For compatibility, using :code:`BitcoinSegwit` will also keep working.
(`#5`_)
* session resource limits: made more useful in general. connection-time-based
grouping has been removed (`#70`_). Disconnects of over-limit sessions happen
sooner (`4b3f6510`_). Subnet sizes for IP-based grouping can now be
configured (`a61136c5`_).
* protocol: :code:`mempool.get_fee_histogram` now returns fee rates with
0.1 sat/byte resolution instead of 1 sat/byte, and the compact histogram
is calculated differently. (`#67`_)
* performance: bitcoind is now queried less frequently as estimatefee,
relayfee, and server.banner requests are now cached (`#24`_)
* performance: json ser/deser is now abstracted away and the :code:`ujson` and
:code:`rapidjson` exras can be used for somewhat faster block processing.
(`#11`_)
* multiple other small performance improvements
Version 1.15.0 (27 May 2020)
============================
* switch to 5-byte txnums to handle larger blockchains. Upgrade DBs during restart.
* accurate clearing of stale caches
* coin additions / updates: NavCoin + Hush + VersusCoin + Zero (cipig), DashRegtest (colmenero),
Quebecoin (morinpa), Primecoin (Sunny King), multiple (Panagiotis David), RVN (standard-error),
Sumcoin
* other: Jeremy Rand, Jin Eguchi, ddude, Jonathan Cross, Carsen Klock, cipig
Version 1.14.0 (19 Jan 2020)
============================
* require Python 3.7
* support for Bitcoin SV Genesis activation
* DB upgrade to allow for larger transactions. Your DB will automatically upgrade when
starting, the upgrade should take approximately 15 mintues.
* fix server shutdown process
* fix cache race condition (issue `#909`_)
* faster initial sync
* coin additions / updates: Emercoin (yakimka), Feathercoin (wellenreiter01),
Peercoin (peerchemist), Namecoin (JeremyRand), Zcoin (a-bezrukov), Simplicity,
Mice (ComputerCraftr), Sibcoin testnet (TriKriSta), Odin (Manbearpixel),
* other: h2o10, osagga, Sombernight, breign, pedr0-fr, wingsuit
Version 1.13.0 (26 Sep 2019)
============================
* daemon: use a single connection for all requests rather than a connection per request.
Distinguish handling of JSON and HTTP errors
* recognise OP_FALSE OP_RETURN scripts as unspendable
* peers - attempt to bind to correct local IP address
* improve name support (domob1812)
* coin additions / updates: BitZeny (y-chan), ZCoin (a-bezrukov), Emercoin (yakimka),
BSV (Roger Taylor), Bellcoin (streetcrypto7), Ritocoin (traysi), BTC (Sombernight),
PIVX (mrcarlanthony), Monacoin (wakiyamap)), NamecoinRegtest (JeremyRand), Axe (ddude1),
Xaya (domob1812), GZRO (MrNaif2018), Ravencoin (standard-error)
* other: gits7r
Version 1.12.0 (13 May 2019)
============================
* require aiorpcX 0.18.1. This introduces websocket support. The environment variables
changed accordingly; see :envvar:`SERVICES` and :envvar:`REPORT_SERVICES`.
* work around bug in recent versions of uvloop
* aiorpcX upgrade fixes from Shane M
* coin additions / updates: BitcoinSV, Bolivarcoin (Jose Luis Estevez), BTC Testnet (ghost43),
Odin (Pixxl)
Version 1.11.0 (18 Apr 2019)
============================
* require aiorpcX 0.15.x
* require aiohttp 3.3 or higher; earlier versions had a problematic bug
* add :envvar:`REQUEST_TIMEOUT` and :envvar:`LOG_LEVEL` environment variables
* mark 4 old environment variables obsolete. ElectrumX won't start until they are removed
* getinfo local RPC cleaned up and shows more stats
* miscellaneous fixes and improvements
* more efficient handling of some RPC methods, particularly
:func:`blockchain.transaction.get_merkle`
* coin additions / updates: BitcoinSV scaling testnet (Roger Taylor), Dash (zebra lucky),
* issues resolved: `#566`_, `#731`_, `#795`_
Version 1.10.1 (13 Apr 2019)
============================
* introduce per-request costing. See environment variables documentation for new
variables :envvar:`COST_SOFT_LIMIT`, :envvar:`COST_HARD_LIMIT`, :envvar:`REQUEST_SLEEP`,
:envvar:`INITIAL_CONCURRENT`, :envvar:`BANDWIDTH_UNIT_COST`. Sessions are placed in groups
with which they share some of their costs. Prior cost is remembered across reconnects.
* require aiorpcX 0.13.5 for better concurrency handling
* require clients use protocol 1.4 or higher
* handle transaction.get_merkle requests more efficiently (ghost43)
* Windows support (sancoder)
* peers improvements (ghost43)
* report mempool and block sizes in logs
* electrumx_rpc: timeout raised to 30s, fix session request counts
* other tweaks and improvements by Bjorge Dijkstra, ghost43, peleion,
* coin additions / updates: ECA (Jenova7), ECCoin (smogm), GXX (DEVCØN), BZX (2INFINITY),
DeepOnion (Liam Alford), CivX / EXOS (turcol)
Version 1.10.0 (15 Mar 2019)
============================
* extra countermeasures to limit BTC phishing effectiveness (ghost43)
* peers: mark blacklisted peers bad; force retry blacklisted peers (ghost43)
* coin additions / updates: Monacoin (wakiyamap), Sparks (Mircea Rila), ColossusXT,
Polis, MNPCoin, Zcoin, GINCoin (cronos), Grosetlcoin (gruve-p), Dash (konez2k),
Bitsend (David), Ravencoin (standard-error), Onixcoin (Jose Estevez), SnowGem
* coin removals: Gobyte, Moneci (cronos)
* minor tweaks by d42
* issues fixed `#660`_ - unclean shutdowns during initial sync
Version 1.9.5 (08 Feb 2019)
===========================
* server blacklist logic (ecdsa)
* require aiorpcX 0.10.4
* remove dead wallet code
* fix `#727`_ - not listing same peer twice
Version 1.9.4 (07 Feb 2019)
===========================
* require aiorpcX 0.10.3
* fix `#713`_
Version 1.9.3 (05 Feb 2019)
===========================
* ignore potential sybil peers
* coin additions / updates: BitcoinCashABC (cculianu), Monacoin (wakiyamap)
Version 1.9.2 (03 Feb 2019)
===========================
* restore protocol version 1.2 and send a warning for old BTC Electrum clients that they
need to upgrade. This is an attempt to protect users of old versions of Electrum from
the ongoing phishing attacks
* increase default MAX_SEND for AuxPow Chains. Truncate AuxPow for block heights covered
by a checkpoint. (jeremyrand)
* coin additions / updates: NMC (jeremyrand), Dash (zebra-lucky), PeerCoin (peerchemist),
BCH testnet (Mark Lundeberg), Unitus (ChekaZ)
* tighter RPC param checking (ghost43)
Version 1.9.1 (11 Jan 2019)
===========================
* fix `#684`_
Version 1.9.0 (10 Jan 2019)
===========================
* minimum protocol version is now 1.4
* coin additions / updates: BitcoinSV, SmartCash (rc125), NIX (phamels), Minexcoin (joesixpack),
BitcoinABC (mblunderburg), Dash (zebra-lucky), BitcoinABCRegtest (ezegom), AXE (slowdive),
NOR (flo071), BitcoinPlus (bushsolo), Myriadcoin (cryptapus), Trezarcoin (ChekaZ),
Bitcoin Diamond (John Shine),
* close `#554`_, `#653`_, `#655`_
* other minor tweaks (Michael Schmoock, Michael Taborsky)
Original author of ElectrumX:
**Neil Booth** kyuupichan@gmail.com https://github.com/kyuupichan
This fork maintained by:
**Electrum developers** electrumdev@gmail.com https://github.com/spesmilo
.. _#554: https://github.com/kyuupichan/electrumx/issues/554
.. _#566: https://github.com/kyuupichan/electrumx/issues/566
.. _#653: https://github.com/kyuupichan/electrumx/issues/653
.. _#655: https://github.com/kyuupichan/electrumx/issues/655
.. _#660: https://github.com/kyuupichan/electrumx/issues/660
.. _#684: https://github.com/kyuupichan/electrumx/issues/684
.. _#713: https://github.com/kyuupichan/electrumx/issues/713
.. _#727: https://github.com/kyuupichan/electrumx/issues/727
.. _#731: https://github.com/kyuupichan/electrumx/issues/731
.. _#795: https://github.com/kyuupichan/electrumx/issues/795
.. _#909: https://github.com/kyuupichan/electrumx/issues/909
.. _#5: https://github.com/spesmilo/electrumx/pull/5
.. _#11: https://github.com/spesmilo/electrumx/pull/11
.. _#22: https://github.com/spesmilo/electrumx/issues/22
.. _#24: https://github.com/spesmilo/electrumx/pull/24
.. _#35: https://github.com/spesmilo/electrumx/pull/35
.. _#67: https://github.com/spesmilo/electrumx/pull/67
.. _#70: https://github.com/spesmilo/electrumx/pull/70
.. _spesmilo/electrumx#75: https://github.com/spesmilo/electrumx/pull/75
.. _spesmilo/electrumx#88: https://github.com/spesmilo/electrumx/issues/88
.. _spesmilo/electrumx#122: https://github.com/spesmilo/electrumx/pull/122
.. _spesmilo/electrumx#248: https://github.com/spesmilo/electrumx/pull/248
.. _spesmilo/electrumx#273: https://github.com/spesmilo/electrumx/pull/273
.. _spesmilo/electrumx#298: https://github.com/spesmilo/electrumx/pull/298
.. _spesmilo/electrumx#301: https://github.com/spesmilo/electrumx/pull/301
.. _spesmilo/electrumx#312: https://github.com/spesmilo/electrumx/pull/312
.. _spesmilo/electrumx#316: https://github.com/spesmilo/electrumx/pull/316
.. _spesmilo/electrumx#317: https://github.com/spesmilo/electrumx/pull/317
.. _spesmilo/electrumx#322: https://github.com/spesmilo/electrumx/pull/322
.. _spesmilo/electrumx#323: https://github.com/spesmilo/electrumx/pull/323
.. _spesmilo/electrumx#326: https://github.com/spesmilo/electrumx/pull/326
.. _spesmilo/electrumx#328: https://github.com/spesmilo/electrumx/pull/328
.. _spesmilo/electrumx#339: https://github.com/spesmilo/electrumx/pull/339
.. _spesmilo/electrumx#340: https://github.com/spesmilo/electrumx/pull/340
.. _spesmilo/electrumx#344: https://github.com/spesmilo/electrumx/pull/344
.. _spesmilo/electrumx#345: https://github.com/spesmilo/electrumx/pull/345
.. _spesmilo/electrumx#347: https://github.com/spesmilo/electrumx/pull/347
.. _spesmilo/electrumx#348: https://github.com/spesmilo/electrumx/pull/348
.. _spesmilo/electrumx#349: https://github.com/spesmilo/electrumx/pull/349
.. _spesmilo/electrumx#351: https://github.com/spesmilo/electrumx/pull/351
.. _spesmilo/electrumx#352: https://github.com/spesmilo/electrumx/pull/352
.. _spesmilo/electrumx#361: https://github.com/spesmilo/electrumx/pull/361
.. _spesmilo/electrumx#369: https://github.com/spesmilo/electrumx/pull/369
.. _4b3f6510: https://github.com/spesmilo/electrumx/commit/4b3f6510e94670a013c1abe6247cdd2b0e7e6f8c
.. _a61136c5: https://github.com/spesmilo/electrumx/commit/a61136c596d6a0290a6be9d21fb7c095c3cea21e
.. _ca59151d: https://github.com/spesmilo/electrumx/commit/ca59151d7365aabd2394fb39aac1faa25949b49d
.. _25339328: https://github.com/spesmilo/electrumx/commit/25339328a7468235071d152f728b214df10d4c56
.. _8198ab99: https://github.com/spesmilo/electrumx/commit/8198ab99603a7e74083b3569c15dad13850c721e
.. _31489365: https://github.com/spesmilo/electrumx/commit/314893655a7cc0bfcc216b07900fa77b5f66e148
.. _a0d3053f: https://github.com/spesmilo/electrumx/commit/a0d3053f60074264b54b3ef7583f2600f4b73ead
.. _2969ab11: https://github.com/spesmilo/electrumx/commit/2969ab110412bebddc3f3e815467fb59538b613d
.. _265a5a87: https://github.com/spesmilo/electrumx/commit/265a5a87b8ad01f739049c0b1e80923aab318f58
.. _5f4dd2cd: https://github.com/spesmilo/electrumx/commit/5f4dd2cdb414464484407affbbaae6b7407696cb
.. _f5582b29: https://github.com/spesmilo/electrumx/commit/f5582b29792625e8cca7cf137a6718c2520bb9cb
.. _0ba87447: https://github.com/spesmilo/electrumx/commit/0ba87447cb293cfc4a8a26c1c27842b95666875a
.. _c9b72e7e: https://github.com/spesmilo/electrumx/commit/c9b72e7ea76bfb706b2df3e90c739116bf49678d
.. _spesmilo/electrum-protocol#2: https://github.com/spesmilo/electrum-protocol/pull/2
.. _spesmilo/electrum-protocol#6: https://github.com/spesmilo/electrum-protocol/pull/6
.. _free-threaded python: https://docs.python.org/3/howto/free-threading-python.html
docs/conf.py
+165
View File
@@ -0,0 +1,165 @@
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/stable/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
VERSION="ElectrumX 2.0.0"
# -- Project information -----------------------------------------------------
project = 'ElectrumX'
copyright = '2016-2026, various'
author = 'Electrum developers'
# The full version including branding
release = VERSION
# The short X.Y version
version = VERSION.split()[-1]
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = 'en'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
html_theme_options = {
'description': 'Lightweight Electrum Server in Python',
'github_user': 'spesmilo',
'github_repo': 'electrumx',
'github_button': True,
'github_type': 'star',
'github_banner': True,
}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
html_sidebars = {
'**': [
'about.html', 'navigation.html', 'searchbox.html',
]
}
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'ElectrumXdoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'ElectrumX.tex', 'ElectrumX Documentation',
'Neil Booth', 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'electrumx', 'ElectrumX Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'ElectrumX', 'ElectrumX Documentation',
author, 'ElectrumX', 'One line description of project.',
'Miscellaneous'),
]
docs/environment.rst
+510
View File
@@ -0,0 +1,510 @@
.. _environment:
=====================
Environment Variables
=====================
ElectrumX takes no command line arguments, instead its behaviour is controlled by
environment variables. Only a few are required to be given, the rest will have sensible
defaults if not specified. Many of the defaults around resource usage are conservative; I
encourage you to review them.
.. note:: set :envvar:`SERVICES` appropriately to be able to connect to your server. For
clients across the internet to know what services you offer you must advertize your
services with :envvar:`REPORT_SERVICES`.
Required
========
These environment variables are always required:
.. envvar:: DB_DIRECTORY
The path to the database directory. Relative paths should be
relative to the parent process working directory. This is the
directory of the `run` script if you use it.
.. envvar:: DB_ENGINE
Database engine for the UTXO and history database.
Choose one of ``leveldb`` or ``rocksdb``.
You will need to install the appropriate python package for your engine.
In ElectrumX 1.x versions, the default was leveldb.
Warning: It is not possible to switch back and forth,
the on-disk DB formats are not compatible with each other: you have to resync from genesis.
LevelDB was written with HDDs in mind. RocksDB is more modern
and on an SSD takes around ~25% less time than LevelDB to sync from genesis.
.. envvar:: DAEMON_URL
A comma-separated list of daemon URLs. If more than one is provided
ElectrumX will initially connect to the first, and failover to
subsequent ones round-robin style if one stops working.
The generic form of a daemon URL is::
http://username:password@hostname:port/
The leading ``http://`` is optional, as is the trailing slash. The
``:port`` part is also optional and will default to the standard RPC
port for :envvar:`COIN` and :envvar:`NET` if omitted.
.. note:: With the above set your server will run and index the chain. To enable incoming
connections you must set :envvar:`SERVICES`, and for others to be aware of your server
set :envvar:`REPORT_SERVICES`.
For the ``run`` script
======================
The following are required if you use the ``run`` script:
.. envvar:: ELECTRUMX
The path to the electrumx_server script. Relative paths should
be relative to the directory of the ``run`` script.
.. envvar:: USERNAME
The username the server will run as.
Services
========
These two environment variables are comma-separated lists of individual *services*.
A **service** has the general form::
protocol://host:port
*protocol* is case-insensitive. The recognised protocols are::
tcp Plaintext TCP sockets
ssl SSL-encrypted TCP sockets
ws Plaintext websockets
wss SSL-encrypted websockets
rpc Plaintext RPC
In a services list, a protocol can be specified multiple times, with different hosts or
ports. This might be useful for multi-homed hosts, or if you offer both Tor and clearnet
services.
*host* can be a hostname, an IPv4 address, or an IPv6 address enclosed in square brackets.
*port* is an integer from :const:`1` to :const:`65535` inclusive.
Where documented, one or more of *protocol*, *host* and *port* can be omitted, in which
case a default value will be assumed.
Here are some examples of valid services::
tcp://host.domain.tld:50001 # Hostname, lowercase protocol, port
SSL://23.45.67.78:50002 # An IPv4 address, upper-case protocol, port
rpC://localhost # Host as a string, mixed-case protocol, default port
ws://[1234:5678:abcd::5601]:8000 # Host as an IPv6 address
wss://h3ubaasdlkheryasd.onion:50001 # Host as a Tor ".onion" address
rpc://:8000 # Default host, port given
host.domain.tld:5151 # Default protocol, hostname, port
rpc:// # RPC protocol, default host and port
.. note:: ElectrumX will not serve any incoming connections until it has fully caught up
with your bitcoin daemon. The only exception is local **RPC** connections,
which are served at any time after the server has initialized.
.. envvar:: SERVICES
A comma-separated list of services ElectrumX will accept incoming connections for.
This environment variable determines what interfaces and ports the server listens on, so
must be set correctly for any connection to the server to succeed. If unset or empty,
ElectrumX will not listen for any incoming connections.
*protocol* can be any recognised protocol.
*host* defaults to all of the machine's interfaces, except if the protocol is **rpc**,
when it defaults to :const:`localhost`.
*port* can only be defaulted for **rpc** where the default is :const:`8000`.
On most Unix systems ports below 1024 require elevated privileges so choosing a higher
port is advisable. On Debian for example, this can be achieved by installing
libcap2-bin package::
sudo apt-get update && sudo apt-get -y install libcap2-bin
sudo setcap cap_net_bind_service=+ep /path/to/electrumx_server
If any listed service has protocol **ssl** or **wss** then :envvar:`SSL_CERTFILE` and
:envvar:`SSL_KEYFILE` must be defined.
Tor **onion** addresses are invalid in :envvar:`SERVICES`.
Here is an example value of the :envvar:`SERVICES` environment variable::
tcp://:50001,ssl://:50002,wss://:50004,rpc://
This serves **tcp**, **ssl**, **wss** on all interfaces on ports 50001, 50002 and 50004
respectively. **rpc** is served on its default host :const:`localhost` and default port
:const:`8000`.
.. envvar:: REPORT_SERVICES
A comma-separated list of services ElectrumX will advertize and other servers in the
server network (if peer discovery is enabled), and any successful connection.
This environment variable must be set correctly, taking account of your network,
firewall and router setup, for clients and other servers to see how to connect to your
server. If not set or empty, no services are advertized.
The **rpc** protocol, special IP addresses (including private ones if peer discovery is
enabled), and :const:`localhost` are invalid in :envvar:`REPORT_SERVICES`.
Here is an example value of the :envvar:`REPORT_SERVICES` environment variable::
tcp://example.com:50001,ssl://example.com:50002,wss://example.com:50004
This advertizes **tcp**, **ssl**, **wss** services at :const:`example.com` on ports
50001, 50002 and 50004 respectively.
.. note:: Certificate Authority-signed certificates don't work over Tor, so you should
only have Tor services` in :envvar:`REPORT_SERVICES` if yours is self-signed.
.. envvar:: SSL_CERTFILE
The filesystem path to your SSL certificate file.
:ref:`SSL certificates`
.. envvar:: SSL_KEYFILE
The filesystem path to your SSL key file.
:ref:`SSL certificates`
Miscellaneous
=============
These environment variables are optional:
.. envvar:: COIN
Must be a :attr:`NAME` from one of the :class:`Coin` classes in
`lib/coins.py`_. Defaults to ``Bitcoin``.
.. envvar:: NET
Must be a :attr:`NET` from one of the :class:`Coin` classes in
`lib/coins.py`_. Defaults to ``mainnet``.
.. envvar:: LOG_FORMAT
The Python logging `format string
<https://docs.python.org/3/library/logging.html#logrecord-attributes>`_
to use. Defaults to ``%(levelname)s:%(name)s:%(message)s``.
.. envvar:: LOG_LEVEL
The default Python logging level, a case-insensitive string. Useful values
are 'debug', 'info', 'warning' and 'error'.
.. envvar:: ALLOW_ROOT
Set this environment variable to anything non-empty to allow running
ElectrumX as root.
.. envvar:: DONATION_ADDRESS
The server donation address reported to Electrum clients. Defaults
to empty, which Electrum interprets as meaning there is none.
.. envvar:: BANNER_FILE
The path to a banner file to serve to clients in Electrum's
"Console" tab. Relative file paths must be relative to
:envvar:`DB_DIRECTORY`. The banner file is re-read for each new
client.
You can place several meta-variables in your banner file, which will be
replaced before serving to a client.
+ ``$SERVER_VERSION`` is replaced with the ElectrumX version you are
running, such as ``1.0.10``.
+ ``$SERVER_SUBVERSION`` is replaced with the ElectrumX user agent
string. For example, ``ElectrumX 1.0.10``.
+ ``$DAEMON_VERSION`` is replaced with the daemon's version as a
dot-separated string. For example ``0.12.1``.
+ ``$DAEMON_SUBVERSION`` is replaced with the daemon's user agent
string. For example, ``/BitcoinUnlimited:0.12.1(EB16; AD4)/``.
+ ``$DONATION_ADDRESS`` is replaced with the address from the
:envvar:`DONATION_ADDRESS` environment variable.
See `here <https://github.com/shsmith/electrumx-banner-updater>`_
for a script that updates a banner file periodically with useful
statistics about fees, last block time and height, etc.
.. envvar:: TOR_BANNER_FILE
As for :envvar:`BANNER_FILE` (which is also the default) but shown
to incoming connections believed to be to your Tor hidden service.
.. envvar:: ANON_LOGS
Set to anything non-empty to replace IP addresses in logs with
redacted text like ``xx.xx.xx.xx:xxx``. By default IP addresses
will be written to logs.
.. envvar:: LOG_SESSIONS
The number of seconds between printing session statistics to the
log. The output is identical to the :ref:`sessions` RPC command
except that :envvar:`ANON_LOGS` is honoured. Defaults to 3600. Set
to zero to suppress this logging.
.. envvar:: REORG_LIMIT
The maximum number of blocks to be able to handle in a chain
reorganisation. ElectrumX retains some fairly compact undo
information for this many blocks in levelDB. The default is a
function of :envvar:`COIN` and :envvar:`NET`; for Bitcoin mainnet it
is 200.
.. envvar:: EVENT_LOOP_POLICY
The name of an event loop policy to replace the default asyncio
policy, if any. At present only ``uvloop`` is accepted, in which
case you must have installed the `uvloop`_ Python package.
If you are not sure what this means leave it unset.
.. envvar:: DROP_CLIENT
Set a regular expression to disconnect any client based on their
version string. For example to drop versions from 1.0 to 1.2 use
the regex ``1\.[0-2]\.\d+``.
Resource Usage Limits
=====================
The following environment variables are all optional and help to limit
server resource consumption and prevent simple DoS.
Address subscriptions in ElectrumX are very cheap - they consume about
160 bytes of memory each and are processed efficiently. I feel the
two subscription-related defaults below are low and encourage you to
raise them.
.. envvar:: MAX_SESSIONS
The maximum number of incoming connections. Once reached, TCP and
SSL listening sockets are closed until the session count drops
naturally to 95% of the limit. Defaults to 1,000.
.. envvar:: MAX_RECV
The maximum size of an incoming message in bytes, the default is 1,000,000 bytes.
Note that the smallest sane/safe value for Bitcoin is ~800,100 bytes,
as the largest standard tx can have a weight of 400K but the protocol hex-encodes that,
plus there is a few bytes of protocol overhead. Setting this to lower than that
would preclude clients from broadcasting txs that could propagate over the network.
.. envvar:: MAX_SEND
The maximum size of a response message to send over the wire, in
bytes. Defaults to 8,100,000 (except for AuxPoW coins, which default
to 10,000,000). Values smaller than 350,000 are taken as 350,000
because standard Electrum protocol header "chunk" requests are almost
that large.
The Electrum protocol has a flaw in that address histories must be
served all at once or not at all, an obvious avenue for abuse.
:envvar:`MAX_SEND` is a stop-gap until the protocol is improved to
admit incremental history requests. Each history entry is
approximately 100 bytes, so the default is equivalent to a history
limit of around 80,000 entries, which should be ample for most
legitimate users. If you use a higher default, bear in mind one
client can request history for multiple addresses.
Also note that the largest raw transaction you will be able to serve to a client is
just under half of :envvar:`MAX_SEND`, as each raw byte becomes 2 hexadecimal
ASCII characters on the wire. For Bitcoin, the consensus limit for tx size is
4 M weight units, while the standardness (policy) limit is 400 K weight units.
When serialized as bytes, 4 M weight units is 4 MB, and when encoded as hex that is 8 MB.
The default was chosen so that such a tx could still be served.
(Note: in multi-party protocol, such as Lightning, a counterparty might collude with a miner
to get a tx mined. Being able to obtain such a raw tx might be part of the security
requirements of a protocol.)
.. envvar:: COST_SOFT_LIMIT
.. envvar:: COST_HARD_LIMIT
.. envvar:: REQUEST_SLEEP
.. envvar:: INITIAL_CONCURRENT
.. envvar:: SESSION_GROUP_BY_SUBNET_IPV4
.. envvar:: SESSION_GROUP_BY_SUBNET_IPV6
All values are integers. :envvar:`COST_SOFT_LIMIT` defaults to :const:`1,000`,
:envvar:`COST_HARD_LIMIT` to :const:`10,000`, :envvar:`REQUEST_SLEEP` to :const:`2,500`
milliseconds, and :envvar:`INITIAL_CONCURRENT` to :const:`10` concurrent requests.
The server prices each request made to it based upon an estimate of the resources needed
to process it. Factors include whether the request uses bitcoind, how much bandwidth
it uses, and how hard it hits the databases.
To set a base for the units, a :func:`blockchain.scripthash.subscribe` subscription to
an address with a history of 2 or fewer transactions is costed at :const:`1.0` before
considering the bandwidth consumed. :func:`server.ping` is costed at :const:`0.1`.
As the total cost of a session goes over the soft limit, its requests start to be
throttled in two ways. First, the number of requests for that session that the server
will process concurrently is reduced. Second, each request starts to sleep a little
before being handled.
Before throttling starts, the server will process up to :envvar:`INITIAL_CONCURRENT`
requests concurrently without sleeping. As the session cost ranges from
:envvar:`COST_SOFT_LIMIT` to :envvar:`COST_HARD_LIMIT`, concurrency drops linearly to
zero and each request's sleep time increases linearly up to :envvar:`REQUEST_SLEEP`
milliseconds. Once the hard limit is reached, the session is disconnected.
In order that non-abusive sessions can continue to be served, a session's cost gradually
decays over time. Subscriptions have an ongoing servicing cost, so the decay is slower
as the number of subscriptions increases.
If a session disconnects, ElectrumX continues to associate its cost with its IP address,
so if it immediately reconnects it will re-acquire its previous cost allocation.
Moreover, sessions are also grouped together based on their IP address subnets, and cost
is accrued for the whole group. What subnet sizes to use can be configured via
:envvar:`SESSION_GROUP_BY_SUBNET_IPV4` (by default /24) and
:envvar:`SESSION_GROUP_BY_SUBNET_IPV6` (by default /48).
A server operator should experiment with different values according to server loads. It
is not necessarily true that e.g. having a low soft limit, decreasing concurrency and
increasing sleep will help handling heavy loads, as it will also increase the backlog of
requests the server has to manage in memory. It will also give a much worse experience
for genuine connections.
.. envvar:: BANDWIDTH_UNIT_COST
The number of bytes, sent and received, by a session that is deemed to cost :const:`1.0`.
The default value :const:`5,000` bytes, meaning the bandwidth cost assigned to a response
of 100KB is 20. If your bandwidth is cheap you should probably raise this.
.. envvar:: REQUEST_TIMEOUT
An integer number of seconds defaulting to :const:`30`. If a request takes longer than
this to respond to, either because of request limiting or because the request is
expensive, the server rejects it and returns a timeout error to the client indicating
that the server is busy.
This can help prevent large backlogs of unprocessed requests building up under heavy load.
.. envvar:: SESSION_TIMEOUT
An integer number of seconds defaulting to :const:`600`. Sessions that have not sent a
request for longer than this are disconnected. Properly functioning clients should send
a :func:`server.ping` request once roughly 450 seconds have passed since the previous
request, in order to avoid disconnection.
Peer Discovery
==============
In response to the :func:`server.peers.subscribe` RPC call, ElectrumX
will only return peer servers that it has recently connected to and
verified basic functionality.
If you are not running a Tor proxy ElectrumX will be unable to connect
to onion server peers, in which case rather than returning no onion
peers it will fall back to a hard-coded list.
To give incoming clients a full range of onion servers, you will need
to be running a Tor proxy for ElectrumX to use.
ElectrumX will perform peer-discovery by default and announce itself
to other peers. If your server is private you may wish to disable
some of this.
.. envvar:: PEER_DISCOVERY
This environment variable is case-insensitive and defaults to
``on``.
If ``on``, ElectrumX will occasionally connect to and verify its
network of peer servers.
If ``off``, peer discovery is disabled and a hard-coded default list
of servers will be read in and served. If set to ``self`` then peer
discovery is disabled and the server will only return itself in the
peers list.
.. envvar:: PEER_ANNOUNCE
Set this environment variable to empty to disable announcing itself.
If not defined, or non-empty, ElectrumX will announce itself to
peers.
If peer discovery is disabled this environment variable has no
effect, because ElectrumX only announces itself to peers when doing
peer discovery if it notices it is not present in the peer's
returned list.
.. envvar:: FORCE_PROXY
By default peer discovery happens over the clear internet. Set this
to non-empty to force peer discovery to be done via the proxy. This
might be useful if you are running a Tor service exclusively and
wish to keep your IP address private.
.. envvar:: TOR_PROXY_HOST
The host where your Tor proxy is running. Defaults to
``localhost``.
If you are not running a Tor proxy just leave this environment
variable undefined.
.. envvar:: TOR_PROXY_PORT
The port on which the Tor proxy is running. If not set, ElectrumX
will autodetect any proxy running on the usual ports 9050 (Tor),
9150 (Tor browser bundle) and 1080 (socks).
.. envvar:: BLACKLIST_URL
URL to retrieve a list of blacklisted peers. If not set, a coin-
specific default is used.
Cache
=====
If synchronizing from the Genesis block your performance might change
by tweaking the cache size. Cache size is only checked roughly every
minute, so the cache can grow beyond the specified size. Moreover,
the Python process is often quite a bit fatter than the cache size,
because of Python overhead and also because leveldb consumes a lot of
memory when flushing. So I recommend you do not set this over 60% of
your available physical RAM:
.. _CACHE:
.. envvar:: CACHE_MB
The amount of cache, in MB, to use. The default is 1,200.
A portion of the cache is reserved for unflushed history, which is
written out frequently. The bulk is used to cache UTXOs.
Larger caches probably increase performance a little as there is
significant searching of the UTXO cache during indexing. However, I
don't see much benefit in my tests pushing this too high, and in
fact performance begins to fall, probably because LevelDB already
caches, and also because of Python GC.
I do not recommend raising this above 2000.
.. _lib/coins.py: https://github.com/spesmilo/electrumx/blob/master/src/electrumx/lib/coins.py
.. _uvloop: https://pypi.python.org/pypi/uvloop
docs/features.rst
+59
View File
@@ -0,0 +1,59 @@
Features
========
- Efficient, lightweight reimplementation of electrum-server
- Fast synchronization of bitcoin mainnet from Genesis. Recent
hardware should synchronize in well under 24 hours. The fastest
time to height 448k (mid January 2017) reported is under 4h 30m. On
the same hardware JElectrum would take around 4 days and
electrum-server probably around 1 month.
- Various configurable means of controlling resource consumption and
handling bad clients and denial of service attacks. These include
maximum connection counts, subscription limits per-connection and
across all connections, maximum response size, per-session bandwidth
limits, and session timeouts.
- Minimal resource usage once caught up and serving clients; tracking the
transaction mempool appears to be the most expensive part.
- Mostly asynchronous processing of new blocks, mempool updates, and
client requests. Busy clients should not noticeably impede other
clients' requests and notifications, nor the processing of incoming
blocks and mempool updates.
- Daemon failover. More than one daemon can be specified, and
ElectrumX will failover round-robin style if the current one fails
for any reason.
- Peer discovery protocol removes need for IRC
- Coin abstraction makes compatible altcoin and testnet support easy.
Implementation
==============
ElectrumX does not do any pruning or throwing away of history. I want
to retain this property for as long as it is feasible, and it appears
efficiently achievable for the foreseeable future with plain Python.
The following all play a part in making it efficient as a Python
blockchain indexer:
- aggressive caching and batching of DB writes
- more compact and efficient representation of UTXOs, address index,
and history. Electrum Server stores full transaction hash and
height for each UTXO, and does the same in its pruned history. In
contrast ElectrumX just stores the transaction number in the linear
history of transactions. ElectrumX can determine block
height from a simple binary search of tx counts stored on disk.
ElectrumX stores historical transaction hashes in a linear array on
disk.
- placing static append-only metadata indexable by position on disk
rather than in levelDB. It would be nice to do this for histories
but I cannot think of a way.
- avoiding unnecessary or redundant computations, such as converting
address hashes to human-readable ASCII strings with expensive bignum
arithmetic, and then back again.
- better choice of Python data structures giving lower memory usage as
well as faster traversal
- leveraging asyncio for asynchronous prefetch of blocks to mostly
eliminate CPU idling. As a Python program ElectrumX is unavoidably
single-threaded in its essence; we must keep that CPU core busy.
Python's :mod:`asyncio` means ElectrumX has no (direct) use for threads
and associated complications.
docs/index.rst
+71
View File
@@ -0,0 +1,71 @@
=========
ElectrumX
=========
.. image:: https://github.com/spesmilo/electrumx/actions/workflows/ci.yml/badge.svg?branch=master
:target: https://github.com/spesmilo/electrumx/actions/workflows/ci.yml
This project is a fork of `kyuupichan/electrumx <https://github.com/kyuupichan/electrumx>`_.
The original author dropped support for Bitcoin, which we intend to keep.
ElectrumX allows users to run their own Electrum server. It connects to your
full node and indexes the blockchain, allowing efficient querying of history of
arbitrary addresses. The server can be exposed publicly, and joined to the public network
of servers via peer discovery. As of May 2020, a significant chunk of the public
Electrum server network runs ElectrumX.
The current version is |release|.
Source Code
===========
The project is hosted on `GitHub
<https://github.com/spesmilo/electrumx/>`_. and uses `GitHub Actions
<https://github.com/spesmilo/electrumx/actions>`_ for Continuous
Integration.
Please submit an issue on the `bug tracker
<https://github.com/spesmilo/electrumx/issues>`_ if you have found a
bug or have a suggestion to improve the server.
Authors and License
===================
Neil Booth wrote the vast majority of the code; see :ref:`Authors`.
Python version at least 3.10 is required.
The code is released under the `MIT Licence
<https://github.com/spesmilo/electrumx/LICENCE>`_.
Getting Started
===============
See :ref:`HOWTO`.
There is also an `installer`_ available that simplifies the
installation on various Linux-based distributions, and a `Dockerfile`_
available.
.. _installer: https://github.com/bauerj/electrumx-installer
.. _Dockerfile: https://github.com/lukechilds/docker-electrumx
Documentation
=============
.. toctree::
features
changelog
HOWTO
environment
protocol
peer_discovery
rpc-interface
architecture
authors
Indices and tables
==================
* :ref:`genindex`
* :ref:`search`
docs/make.bat
+36
View File
@@ -0,0 +1,36 @@
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
set SPHINXPROJ=ElectrumX
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd
docs/peer_discovery.rst
+154
View File
@@ -0,0 +1,154 @@
.. _Peer Discovery:
Peer Discovery
==============
This was implemented in ElectrumX as of version 0.11.0. Support for
IRC peer discovery was removed in ElectrumX version 1.2.1.
The :dfn:`peer database` is an in-memory store of peers with at least
the following information about a peer, required for a response to the
:func:`server.peers.subscribe` RPC call:
* host name
* ip address
* TCP and SSL port numbers
* protocol version
* pruning limit, if any
Hard-coded Peers
----------------
A list of hard-coded, well-known peers seeds the peer discovery
process. Ideally it should have at least 4 servers that have shown
commitment to reliable service.
In ElectrumX this is a per-coin property in `lib/coins.py
<https://github.com/spesmilo/electrumx/blob/master/electrumx/lib/coins.py>`_.
server.peers.subscribe
----------------------
:func:`server.peers.subscribe` is used by Electrum clients to get a
list of peer servers, in preference to a hard-coded list of peer
servers in the client, which it will fall back to if necessary.
The server should craft its response in a way that reduces the
effectiveness of server sybil attacks and peer spamming.
The response should only include peers it has successfully connected
to recently. Only reporting recent good peers ensures that those that
have gone offline will be forgotten quickly and not be passed around
for long.
In ElectrumX, "recently" is taken to be the last 24 hours. Only one
peer from each IPv4/16 netmask is returned, and the number of onion
peers is limited.
Maintaining the Peer Database
-----------------------------
In order to keep its peer database up-to-date and fresh, after some
time has passed since the last successful connection to a peer, an
Electrum server should make another attempt to connect, choosing
either the TCP or SSL port.
On connecting it should issue :func:`server.peers.subscribe`,
:func:`blockchain.headers.subscribe`, and :func:`server.features` RPC
calls to collect information about the server and its peers. If the
peer seems to not know of you, you can issue a :func:`server.add_peer`
call to advertise yourself. Once this is done and replies received,
terminate the connection.
The peer database should view information obtained from an outgoing
connection as authoritative, and prefer it to information obtained
from any other source.
On connecting, a server should confirm the peer is serving the same
network, ideally via the genesis block hash of the
:func:`server.features` RPC call below. Also the height reported by
the peer should be within a small number of the expected value. If a
peer is on the wrong network it should never be advertised to clients
or other peers. Such invalid peers should perhaps be remembered for a
short time to prevent redundant revalidation if other peers persist in
advertising them, and later forgotten.
If a connection attempt fails, subsequent reconnection attempts should
follow some kind of exponential backoff.
If a long period of time has elapsed since the last successful
connection attempt, the peer entry should be removed from the
database. This ensures that all peers that have gone offline will
eventually be forgotten by the network entirely.
ElectrumX will connect to the SSL port if both ports are available.
If that fails it will fall back to the TCP port. It tries to
reconnect to a good peer at least once every 24 hours, and a failing
after 5 minutes but with exponential backoff. It forgets a peer
entirely if a few days have passed since a successful connection.
ElectrumX attempts to connect to onion peers through a Tor proxy that
can be configured or that it will try to autodetect.
server.features
---------------
:func:`server.features` is a fairly new RPC call that a server can use
to advertise what services and features it offers. It is intended for
use by Electrum clients as well as other peers. Peers will use it to
gather peer information from the peer itself.
The call takes no arguments and returns a dictionary keyed by feature
name whose value gives details about the feature where appropriate.
If a key is missing the feature is presumed not to be offered.
server.add_peer
---------------
:func:`server.add_peer` is intended for a new server to get itself in
the connected set.
A server receiving a :func:`server.add_peer` call should not replace
existing information about the host(s) given, but instead schedule a
separate connection to verify the information for itself.
To prevent abuse a server may do nothing with second and subsequent
calls to this method from a single connection.
The result should be True if accepted and False otherwise.
Notes for Implementors
----------------------
* it is very important to only accept peers that appear to be on the
same network. At a minimum the genesis hash should be compared (if
the peer supports :func:`server.features`), and also that the peer's
reported height is within a few blocks of your own server's height.
* care should be taken with the :func:`server.add_peer` call.
Consider only accepting it once per connection. Clearnet peer
requests should check the peer resolves to the requesting IP
address, to prevent attackers from being able to trigger arbitrary
outgoing connections from your server. This doesn't work for onion
peers so they should be rate-limited.
* it should be possible for a peer to change their port assignments -
presumably connecting to the old ports to perform checks will not
work.
* peer host names should be checked for validity before accepting
them; and `localhost` should probably be rejected. If it is an IP
address it should be a normal public one (not private, multicast or
unspecified).
* you should limit the number of new peers accepted from any single
source to at most a handful, to limit the effectiveness of malicious
peers wanting to trigger arbitrary outgoing connections or fill your
peer tables with junk data.
* in the response to :func:`server.peers.subscribe` calls, consider
limiting the number of peers on similar IP subnets to protect
against sybil attacks, and in the case of onion servers the total
returned.
* you should not advertise a peer's IP address if it also advertises a
hostname (avoiding duplicates).
docs/protocol.rst
+7
View File
@@ -0,0 +1,7 @@
Electrum Protocol
=================
See rendered `protocol docs here <https://electrum-protocol.readthedocs.io/>`_,
and `git repository here <https://github.com/spesmilo/electrum-protocol>`_.
docs/release_process.md
+33
View File
@@ -0,0 +1,33 @@
# Release process
### 1. Add changelog
### 2. Update version number
- `docs/conf.py`
- `electrumx/__init__.py`
### 3. Tag
```
$ git tag -s "$VERSION" -m "$VERSION"
$ git push "$REMOTE_ORIGIN" tag "$VERSION"
```
### 4. Build sdist
see [`contrib/sdist/`](contrib/sdist)
```
$ ELECBUILD_COMMIT=HEAD ELECBUILD_NOCACHE=1 ./contrib/sdist/build.sh
```
### 5. Upload to PyPI
```
// $ python3 -m twine upload --repository testpypi dist/$DISTNAME
$ python3 -m twine upload dist/$DISTNAME
```
This will prompt for a username and password. For the username, use `__token__`.
For the password, use API token from https://pypi.org/manage/account/ , including the pypi- prefix.
docs/requirements-docs.txt
+1
View File
@@ -0,0 +1 @@
Sphinx
docs/rpc-interface.rst
+280
View File
@@ -0,0 +1,280 @@
RPC Interface
=============
You can query the status of a running server, and affect its behaviour
by sending **JSON RPC** commands to the LocalRPC port it is listening
on. This is best done using the :file:`electrumx_rpc` script
provided.
The general form of invocation is::
electrumx_rpc [-p PORT] <command> [arg1 [arg2...]]
The port to send the commands to can be specified on the command line,
otherwise the environment variable :envvar:`RPC_PORT` is used, and if
that is not set then **8000** is assumed.
The following commands are available:
add_peer
--------
Add a peer to the peers list. ElectrumX will schedule an immediate
connection attempt. This command takes a single argument: the peer's
"real name" as it used to advertise itself on IRC::
$ electrumx_rpc add_peer "ecdsa.net v1.0 s110 t"
"peer 'ecdsa.net v1.0 s110 t' added"
daemon_url
----------
This command takes an optional argument that is interpreted
identically to the :envvar:`DAEMON_URL` environment variable. If
omitted, the default argument value is the process's existing
:envvar:`DAEMON_URL` environment variable.
This command replaces the daemon's URL at run-time, and also
forcefully rotates to the first URL in the list.
For example, in case ElectrumX has previously failed over to a
secondary daemon and you want to revert to the primary having resolved
the connectivity issue, invoking this command without an argument will
have that effect.
disconnect
----------
Disconnect the given session IDs or group names.
Session IDs can be obtained in the logs or with the `sessions`_ RPC command. Group
names can be obtained with the `groups`_ RPC command.
The special string :const:`all` disconnects all sessions.
Example::
$ electrumx_rpc disconnect 209.59.102 34 2
[
"disconnecting session 34",
"disconnecting group 209.59.102"
"unknown: 2",
]
getinfo
-------
Return a summary of server state. This command takes no arguments.
A typical result is as follows (with annotated comments)::
$ electrumx_rpc getinfo
{
"coin": "Bitcoin",
"daemon": "127.0.0.1:9334/",
"daemon height": 572154, # The daemon's height when last queried
"db height": 572154, # The height to which the DB is flushed
"groups": 586, # The number of session groups
"history cache": "185,014 lookups 9,756 hits 1,000 entries",
"merkle cache": "280 lookups 54 hits 213 entries",
"peers": { # Peer information
"bad": 1,
"good": 51,
"never": 2,
"stale": 0,
"total": 54
},
"pid": 11804, # Process ID
"request counts": { # Count of RPC requests by method name
"blockchain.block.header": 245,
"blockchain.block.headers": 70,
"blockchain.estimatefee": 12776,
"blockchain.headers.subscribe": 2825,
"blockchain.relayfee": 740,
"blockchain.scripthash.get_history": 196,
"blockchain.scripthash.subscribe": 184626,
"blockchain.transaction.broadcast": 19,
"blockchain.transaction.get": 213,
"blockchain.transaction.get_merkle": 289,
"getinfo": 3,
"groups": 1,
"mempool.get_fee_histogram": 3194,
"mempool.get_info": 121,
"server.add_peer": 9,
"server.banner": 740,
"server.donation_address": 754,
"server.features": 50,
"server.peers.subscribe": 792,
"server.ping": 6412,
"server.version": 2866
},
"request total": 216820, # Total requests served
"sessions": { # Live session stats
"count": 670,
"count with subs": 45,
"errors": 0,
"logged": 0,
"paused": 0,
"pending requests": 79, # Number of requests currently being processed
"subs": 36292 # Total subscriptions
},
"tx hashes cache": "289 lookups 38 hits 213 entries",
"txs sent": 19, # Transactions broadcast
"uptime": "01h 39m 04s",
"version": "ElectrumX 1.10.1"
}
Each ill-formed request, or one that does not follow the Electrum
protocol, increments the error count of the session that sent it.
:ref:`logging <session logging>` of sessions can be enabled by RPC.
For more information on peers see :ref:`here <peers>`.
Clients that are slow to consume data sent to them are :dfn:`paused` until their socket
buffer drains sufficiently, at which point processing of requests resumes.
Apart from very short intervals, typically after a new block or when a client has just
connected, the number of unprocessed requests should be low, say 250 or fewer. If it is
over 1,000 the server is overloaded.
Sessions are put into groups, primarily as an anti-DoS measure. Each session goes into
a group based on its IP subnet (e.g. /24 for IPv4). Each member of a group incurs a fraction
of the costs of the other group members. This appears in the `sessions_` list under the
column XCost.
groups
------
Return a list of all current session groups. Takes no arguments.
The output is quite similar to the `sessions`_ command.
log
---
Toggle logging of the given session IDs or group names. All incoming requests for a
logged session are written to the server log. The arguments are case-insensitive.
When a group is specified, logging is toggled for its current members only; there is no
effect on future group members.
Session IDs can be obtained in the logs or with the `sessions`_ RPC command. Group
names can be obtained with the `groups`_ RPC command.
The special string :const:`all` turns on logging of all current and future sessions,
:const:`none` turns off logging of all current and future sessions, and :const:`new`
toggles logging of future sessions.
Example::
$ electrumx_rpc log new 6 t0 z
[
"logging new sessions",
"logging session 6",
"logging session 3",
"logging session 57",
"logging session 12"
"unknown: z",
]
In the above command sessions 3, 12 and 57 were in group `t0` (in fact, session 6 was
too).
.. _peers:
peers
-----
Return a list of peer Electrum servers serving the same coin network.
This command takes no arguments.
Peer data is obtained via a peer discovery protocol documented
:ref:`here <Peer Discovery>`::
$ electrumx_rpc peers
Host Status TCP SSL Server Min Max Pruning Last Good Last Try Tries Source IP Address
bch.tedy.pw good 50001 50002 ElectrumX 1.2.1 0.9 1.2 07h 29m 23s 07h 30m 40s 0 peer 185.215.224.26
shsmithgoggryfbx.onion good 60001 60002 ElectrumX 1.2.1 0.9 1.2 07h 30m 34s 07h 30m 38s 0 peer
bccarihace4jdcnt.onion good 52001 52002 ElectrumX 1.2.1 0.9 1.2 07h 30m 34s 07h 30m 39s 0 peer
[...]
electroncash.checksum0.com good 50001 50002 ElectrumX 1.2.1 0.9 1.1 07h 30m 40s 07h 30m 41s 0 peer 149.56.198.233
.. _query:
query
-----
Run a query of the UTXO and history databases against one or more
addresses, hex scripts or ASCII names (for coins that have an index
on names like Namecoin). `--limit <N>` or `-l <N>` limits the output
for each kind to that many entries. History is printed in blockchain
order; UTXOs in an arbitrary order.
For example::
$ electrumx_rpc query --limit 5 76a91462e907b15cbf27d5425399ebf6f0fb50ebb88f1888ac
Script: 76a91462e907b15cbf27d5425399ebf6f0fb50ebb88f1888ac
History #1: height 123,723 tx_hash 3387418aaddb4927209c5032f515aa442a6587d6e54677f08a03b8fa7789e688
History #2: height 127,280 tx_hash 4574958d135e66a53abf9c61950aba340e9e140be50efeea9456aa9f92bf40b5
History #3: height 127,909 tx_hash 8b960c87f9f1a6e6910e214fcf5f9c69b60319ba58a39c61f299548412f5a1c6
History #4: height 127,943 tx_hash 8f6b63012753005236b1b76e4884e4dee7415e05ab96604d353001662cde6b53
History #5: height 127,943 tx_hash 60ff2dfdf67917040139903a0141f7525a7d152365b371b35fd1cf83f1d7f704
UTXO #1: tx_hash 9aa497bf000b20f5ec5dc512bb6c1b60b68fc584d38b292b434e839ea8807bf0 tx_pos 0 height 254,148 value 5,500
UTXO #2: tx_hash 1c998142a5a5aae6f8c1eab245351413fe8d4032a3f14345f9943a0d0bc90ec0 tx_pos 0 height 254,161 value 5,500
UTXO #3: tx_hash 53345491b4829140be53f30079c6e4556a18545343b122900ebbfa158f9ca97a tx_pos 0 height 254,163 value 5,500
UTXO #4: tx_hash c71ad947ac46af217da3cd5521113cbd03e36ddada2b4452afe6c15f944d2529 tx_pos 0 height 372,916 value 1,000
UTXO #5: tx_hash c944a6acac054275a5e294e746d9ce79f6dcae91f3b4f5a84561aee6404a55b3 tx_pos 0 height 254,148 value 5,500
Balance: 17.8983303 BCH
reorg
-----
Force a block chain reorganisation, primarily for debugging purposes.
This command takes an optional argument - the number of blocks to
reorg - which defaults to 3.
That number of blocks will be backed up - using undo information
stored in ElectrumX's database - and then ElectrumX will move forwards
on the daemon's main chain to its current height.
.. _sessions:
sessions
--------
Return a list of all current sessions. Takes no arguments::
ID Flags Client Proto Cost XCost Reqs Txs Subs Recv Recv KB Sent Sent KB Time Peer
1 S6 1.1.1 1.4 0 16 0 0 0 3 0 3 0 05m42s 165.255.191.213:22349
2 S6 all_seeing_eye 1.4 0 16 0 0 0 2 0 2 0 05m40s 67.170.52.226:24995
4 S6 3.3.2 1.4 0 16 0 0 34 45 5 45 3 05m40s 185.220.100.252:40463
3 S6 1.1.2 1.4 0 16 0 0 0 3 0 3 0 05m40s 89.17.142.28:59241
The columns show information by session: the session ID, flags (see below), how the client
identifies itself - typically the Electrum client version, the protocol version
negotiated, the session cost, the additional session cost accrued from its groups, the
number of unprocessed requests, the number of transactions sent, the number of address
subscriptions, the number of requests received and their total size, the number of
messages sent and their size, how long the client has been connected, and the client's IP
address (if anonymous logging is disabled).
The flags are:
* ``S`` an SSL connection
* ``T`` a TCP connection
* ``R`` a local RPC connection
* ``L`` a logged session
* ``C`` a connection that is being closed
* the non-negative number is the connection "cost", with lower
numbers having higher priority. RPC connections have cost ``0``,
normal connections have cost at least ``1``.
stop
----
Flush all cached data to disk and shut down the server cleanly, as if
sending the `KILL` signal. Be patient - during initial sync flushing
all cached data to disk can take several minutes. This command takes
no arguments.
.. _session logging: