-Main update is to add support for express v5.x which required some code changes to a few of the public and wallet api routes
-Removed the body-parser dependency and replaced with express which now has the same functionaliuty built-in and is no longer necessary to have a dedicated dependency
-All dependencies in the package.json have been updated to reference the most up-to-date version numbers
-An optional "Extracted By" column can now be added to the homepage tx data, the block page, tx page and/or in the /ext/getlasttxs api
-Added 4 new settings to allow displaying the "Extracted By" data on the homepage, block page, transaction page and/or in the /ext/getlasttxs api
-Fixed an issue with the get_txs function where it wasn't properly searching by txid
-The rl_labels.pug file has been updated to consolidate similar code without being duplicated
-Updated the README with new verbiage for the extracted by data
-A number of functions have been rewritten to be more optimized and faster: calculate_total, is_unique, convert_to_satoshi, get_input_addresses, processVoutAddresses, prepare_vout, prepare_vin
-Txes are now written to database via bulk writes which helps improve the sync speed and also controls memory usage with batching to write data once a certain threshold is reached
-update_address function changed to update_addresses since it now bulk writes the addresses in batches to improve sync speed and also controls memory usage with batching to write data once a certain threshold is reached
-The syncLoop function has been completely removed from the project and replaced with async library loops or even normal "for" loops in some cases which greatly improves sync speeds over large batches of data
-Fixed an issue with the flattened count of txes that is saved to the coinstats collection which could save incorrectly when using more than 1 thread
-Fixed an issue with the block sync which caused an unwanted delay when syncing less blocks than the amount of threads used to sync the data
-Fixed an issue with vout data processing that could sometimes populate data out of order
-Added a new sync.batch_size setting used to determine how many records (txes, addresses, addresstxes) should be saved in a single database transaction
-Added a new wait_for_bulk_database_save setting used to increase the block sync speed at the cost of not returning any error msgs for data that failed to save
-get_input_addresses function no longer returns in the exports section of the explorer.js file since it is only referenced in that file
-Updated explorerspec tests to use the newest function changes for any tests that needed to be updated
Special thanks to Karzo from Pepecoin for help with the bulkwrite code changes!
-The "Maximum call stack size exceeded" error is now handled internally by the block sync script in a way which will capture the error and re-launch the sync using a larger stack size and have the sync resume from where it left off. If the re-launch still doesn't have enough memory it will continue re-launching with more and more memory until the sync can finish without errors and then it will return to sync with a lower memory footprint for future syncs
-Added a new option for sync.elastic_stack_size which is used to determine how much memory should be used to increase the stack size for the block sync after encountering the "Maximum call stack size exceeded" error
-Fixed an issue with the block sync when using more than 1 thread that could sometimes cause the flattened txes value in the coinstats database to be written incorrectly (Use `npm run reindex-txcount` to fix this issue without needing to reindex the entire database)
-Updated the benchmark script so that it can also benefit from being able to capture the "Maximum call stack size exceeded" error even though the timing will be off so it outputs a new warning in that scenario which instructs to run the benchmark again with a higher stack size to properly capture the benchmark time
-Removed the "Maximum call stack size exceeded" error notes from the "Known Issues" section of the README
-The block_parallel_tasks feature has been improved and fixed so that it is now safe to cancel (Ctrl+C) or kill (kill cmd not kill -9) the task and resume the sync later without missing transactions. The new block_parallel_tasks default is 8 threads which seems to be the sweet spot for any type of cpu
-Numerous improvements to the benchamark script to utilize new benchmark settings, auto-add credentials to the benchmark database, reuse the same sync code as the regular block sync instead of using a copy of the code and more
-Added a new cmd to run the benchmark script `npm run benchmark`
-README updated to include the new benchmark script instrutions + include multi-threaded sync as a feature
-Chart.js has been updated to v4.4.7
-The chartjs-plugin-crosshair chart plugin has been updated to v2.0.5 via a forked version that has a working sync feature which is now available as a new setting option for use with the network charts
-Added a new max_hours setting to display chart data for a certain number of hours instead of a fixed set of records which can help reveal holes in the sync process for the explorer and/or blockchain
-Added a new timestamp field to the network history collection for use with the max_hours setting chart data
-Added a number of new network chart settings to control display of the chart title, legend, a new vertical block line option, chart height, an option to force 2 charts to appear on their own row or beside each other, and an option to force a chart to take up all available space in the chart box without extra padding
-Added a new dependency chartjs-plugin-annotation v3.1.0 to display block lines in new hourly charts
-Plugins can now be enabled via settings.json after dropping the plugin files into the new plugins directory
-Enabling plugins will allow extending the normal functionality of the explorer with new database collections, menus, pages and apis + open up a new url for data to be sent from the plugin to the explorer
-A new plugins section was added to the settings with a definition for the generic-snapshots plugin
-Locale strings are now loaded and shared out via the settings so there is generally no more need to explicitly include the locale.js file
-The locale object has been updated to localization within the explorer
-A number of new locale strings have been added and their values replaced with the locale string within the explorer
-Added plugin support verbiage and a link to the generic-snapshots crowdfunding task to the README
-Removed the ability to claim an address that has 0 transactions
-The "Claim" button is now disabled after submitting to help prevent double submissions
-Supported captchas include Google reCaptcha v3 (score-based), Google reCaptcha v2 (checkbox and invisible) and hCaptcha ("Always Challenge" mode)
-Captcha options are global to the explorer even though the only form submission page is the "Claim Address" feature which takes full advantage of the new captcha options
-The market sync could crash when calculating the USD price if the base pair of the markets_page.default_exchange.trading_pair value was not found in coingecko's vs_currencies list from https://api.coingecko.com/api/v3/simple/supported_vs_currencies (for example, using USDT as the default base pair). A different calculation is now used in this case which does not require any additional api credits to be used.
-New settings allow reading of the hash algorithm used to mine a particular block for coins that support this feature and have the algorithm data stored in the raw block data
-An "Algorithm" column has been added to the block page and main transaction homepage when multi-algo data is enabled
-The /ext/getlasttxs api will now return the hash algorithm if reading of the multi-algo data is enabled
-This fix prevents an error from being thrown that crashes the explorer when trying to use a masternode count that is a sinlge number. A single number masternode count is not valid or usable since it cannot differentiate the number of good and bad nodes, but the explorer will no longer crash when given this data
-Added support for a couple masternode list and count formats that do not have a separate fieldname for each value
-Masternode grid columns are now shown or hidden based on whether there is any data
-As of Feb 2024 the free "keyless" coingecko api will be deprecated and will likely stop working. There is a free demo api key that can be used which can now be plugged into the explorer settings to allow the coingecko api to continue working. Read more here: https://support.coingecko.com/hc/en-us/articles/21880397454233
-The previous market price calculation setting was hardcoded to only display market and USD prices for a single exchange and trading pair which was not very accurate for coins listed on multiple exchanges or with multiple trading pairs. The new default is to average the market prices for all supported exchanges and trading pairs
-The coingecko market price option was added to allow fetching the market price directly from the coingecko api instead of calculating it via supported exchanges known to the explorer
-Added a new root setting option for default_coingecko_ids which allows presetting symbols to their associated internal coingecko id to help prevent matching to the wrong currency with same symbol via coingecko api calls
-Fixed an issue where the explorer would fail to start with an enabled exchange that had no defined trading pairs
-Javascript in the layout.pug has been moved around in an attempt to only load code into the client based on certain settings such as the type of menu (side or top) in order to slim down the loading of pages by not including unnecessary code
-Fixed a bug on the richlist page where the last updated date and logo spinning animation would fail to load if the distribution chart was disabled in the settings
-The .ext/getsummary api has been modified to only return the connection and block counts when necessary
-The getmasternodecount rpc api will no longer fetch data unless it is enabled in settings
-Fixed a bug with the freiexchange pair volume value when paired with LTC
-Added a default value of zero for many of the fields for each market in the event that the data returned for a particular field is not numeric
-All external market apis have been normalized with a similar coding format, improved error handling with new wait times in between api calls to prevent abuse issues with sending too many requests too quickly
-All general market code has been reviewed and improved where necessary to help prevent sync issues
-Inactive markets are now removed from the markets collection on startup of the explorer to help prevent bloating the database
-The yobit api url has been changed to the new url
-The poloniex market has been updated to use the newest api
-The southxchange exchange trading link has been changed to the new url
-Miscellaneous locale string changes and reorganization
-jQuery has been updated to v3.7.1
-Luxon has been updated to v3.4.3. Also updated the link to the documentation showing how to format dates in the settings.json.template and settings.js
-Datatables has been updated to v1.13.6
-Font Awesome has been updated to v6.4.2 and all icons in the project have been updated to use the newest v6.x codes
-OverlayScrollbars has been updated to v2.3.2
-flag-icons (previously called flag-icon-css) has been updated to v6.11.1
-Updated the main style.scss file with better color support for the new datatable loading animation color
-Exor theme has been updated to v3.0.2 with a small fix to support the new datatable loading animation color
-Created a new claimaddress collection to hold claim address data to make it easier to work with and preserve that data if necessary in the future without being cluttered into the address collection
-The database init function has been updated to move claim address data to the new collection and remove the data from the address collection. This process will update existing explorer databases automatically and is smart enough to only run this process one time to prevent from slowing down startup of the explorer on each run
-The claim name field has been changed from "name" to "claim_name" wherever possible to be easier to find in the future. Searching for the keyword "name" brings back way too many matches and makes it difficult to find all the correct code snippets for future changes
-Added a newer_claim_address field to the stats collection to determine if the claim address data needs to be moved to the new collection or not
-All previous claim address code has been updated to pull from the new table and/or join to the address table if/when necessary
-Moved the database initialize call out of the instance code into the prestart script so that it is now only called once when you start the explorer even if you are running multiple cpus/threads
-The database connection function now uses the default connection string in the event a connection string is not supplied
-Removed unused reference to lib/explorer.js code from the cluster file
-The find_masternodes function now searches for both txhash and address instead of only txhash to prevent matching different masternodes that were created in the same tx
-Moved the find_masternodes function out of the module.exports section of the database file since it doesn't need to be exported
-Added a new "powered_by_text" option to the page_footer settings which can be used to override the text in the middle of the footer without the need to modify the source code
-The word "Explorer" was removed from the powered by text for new installs
-Mongoose and mongodb dependencies have been updated to the most recent versions in package.json
-Updated the project for use with the newest mongoose major version which required fixing many breaking changes to a number of files in the explorer
-Removed the db.find_address_tx() function as it is unused
NOTE: This update requires mongoose 6.0.10 or higher. If you are getting errors running the explorer after upgrading, it is recommended to run an 'npm update' or manually update mongoose to a more modern version which should fix the problem
-The block sync will now remove orphaned block data from the txes collecton, address balances/sent/received data as well as addresstx data and now stores limited info in a new orphans collection
-Added a new optional page for viewing orphaned block data
-The coinbase address now updates sent totals from POS rewards and other transactions that have vout but no vin addresses
-Block and transaction pages now display a warning when viewing an orphaned block or tx
-Added a couple new fields to the coinstats collection for tracking orphaned blocks
-Added new locale strings for the orphaned block feature
-Suppress the "DeprecationWarning: Mongoose: the `strictQuery` option will be switched back to `false` by default in Mongoose 7" warning msg when starting the explorer and using any of the scripts that connect to mongo
-Small change to help sort the peers list a bit better when filtering ports so that lower protocols drop off the list if there are multiple connections from the same ip with different protocols
-The port_filter settings for all 3 network page tables now support a -1 value which will completely hide the port from the associated table. If there are/were multiple ip addresses in the peers list with connections from different ports, this new option will consolidate those multiple records into a single entry since the port is no longer visible
-A new side menu option was added for displaying menu items vertically on the left-hand side of the page
-Added a new script called OverlayScrollbars v1.13.3 for displaying custom scrollbars on the mobile menu (both top and side menu) in the event that they extend beyond the visible screen
-Updated the market menu dropdown colors to better match the base menu colors (default, light and dark colors only)
-Updated the Exor theme to v3.0.1 with a few small fixes for displaying menu items in the side menu and updated market dropdown colors
-Updated how the richlist pie chart is sized to prevent it from creating unnecessary scrollbars because of the side menu
-Search field and button are now hidden in the top menu when the mobile menu is expanded to make it easier to scroll through menu items
-Added a tooltip title to the top menu search and menu toggler buttons
-The SIGINT and SIGTERM signals are now being caught and handled in the sync.js file so that most options for syncing blocks, markets, peers, masternodes, etc. are now being gracefully shut down instead of killed in the middle of the process. This should help prevent data anomalies when you need to stop or kill a sync process
-The update_tx_db function was moved from the database.js file into the sync.js file so that block syncs can now be gracefully stopped. The update_tx_db function was also copied to the benchmark.js
-The save_tx function was moved into the module.exports for the database.js file so that it can now be called from outside the database.js file
Joe Uhren