-The "Maximum call stack size exceeded" error is now handled internally by the block sync script in a way which will capture the error and re-launch the sync using a larger stack size and have the sync resume from where it left off. If the re-launch still doesn't have enough memory it will continue re-launching with more and more memory until the sync can finish without errors and then it will return to sync with a lower memory footprint for future syncs
-Added a new option for sync.elastic_stack_size which is used to determine how much memory should be used to increase the stack size for the block sync after encountering the "Maximum call stack size exceeded" error
-Fixed an issue with the block sync when using more than 1 thread that could sometimes cause the flattened txes value in the coinstats database to be written incorrectly (Use `npm run reindex-txcount` to fix this issue without needing to reindex the entire database)
-Updated the benchmark script so that it can also benefit from being able to capture the "Maximum call stack size exceeded" error even though the timing will be off so it outputs a new warning in that scenario which instructs to run the benchmark again with a higher stack size to properly capture the benchmark time
-Removed the "Maximum call stack size exceeded" error notes from the "Known Issues" section of the README
-The block_parallel_tasks feature has been improved and fixed so that it is now safe to cancel (Ctrl+C) or kill (kill cmd not kill -9) the task and resume the sync later without missing transactions. The new block_parallel_tasks default is 8 threads which seems to be the sweet spot for any type of cpu
-Numerous improvements to the benchamark script to utilize new benchmark settings, auto-add credentials to the benchmark database, reuse the same sync code as the regular block sync instead of using a copy of the code and more
-Added a new cmd to run the benchmark script `npm run benchmark`
-README updated to include the new benchmark script instrutions + include multi-threaded sync as a feature
-Plugins can now be enabled via settings.json after dropping the plugin files into the new plugins directory
-Enabling plugins will allow extending the normal functionality of the explorer with new database collections, menus, pages and apis + open up a new url for data to be sent from the plugin to the explorer
-A new plugins section was added to the settings with a definition for the generic-snapshots plugin
-Locale strings are now loaded and shared out via the settings so there is generally no more need to explicitly include the locale.js file
-The locale object has been updated to localization within the explorer
-A number of new locale strings have been added and their values replaced with the locale string within the explorer
-Added plugin support verbiage and a link to the generic-snapshots crowdfunding task to the README
-New settings allow reading of the hash algorithm used to mine a particular block for coins that support this feature and have the algorithm data stored in the raw block data
-An "Algorithm" column has been added to the block page and main transaction homepage when multi-algo data is enabled
-The /ext/getlasttxs api will now return the hash algorithm if reading of the multi-algo data is enabled
-Added support for a couple masternode list and count formats that do not have a separate fieldname for each value
-Masternode grid columns are now shown or hidden based on whether there is any data
-As of Feb 2024 the free "keyless" coingecko api will be deprecated and will likely stop working. There is a free demo api key that can be used which can now be plugged into the explorer settings to allow the coingecko api to continue working. Read more here: https://support.coingecko.com/hc/en-us/articles/21880397454233
-The previous market price calculation setting was hardcoded to only display market and USD prices for a single exchange and trading pair which was not very accurate for coins listed on multiple exchanges or with multiple trading pairs. The new default is to average the market prices for all supported exchanges and trading pairs
-The coingecko market price option was added to allow fetching the market price directly from the coingecko api instead of calculating it via supported exchanges known to the explorer
-Added a new root setting option for default_coingecko_ids which allows presetting symbols to their associated internal coingecko id to help prevent matching to the wrong currency with same symbol via coingecko api calls
-Fixed an issue where the explorer would fail to start with an enabled exchange that had no defined trading pairs
-All external market apis have been normalized with a similar coding format, improved error handling with new wait times in between api calls to prevent abuse issues with sending too many requests too quickly
-All general market code has been reviewed and improved where necessary to help prevent sync issues
-Inactive markets are now removed from the markets collection on startup of the explorer to help prevent bloating the database
-The yobit api url has been changed to the new url
-The poloniex market has been updated to use the newest api
-The southxchange exchange trading link has been changed to the new url
-Miscellaneous locale string changes and reorganization
-The delete-database script now checks the claimaddresses collection for data and if at least 1 record exists, it will ask an additional question to see if you want to preserve that data or delete everything
-The delete-database script now issues the prompt for the reindex with a slightly different msg than the delete without reindex
-The prompt for deleting now happens after the locks have been properly checked
-Reindex now calls the delete-databse.js script which is much faster and more complete than the previous manual delete code the resync used to use
-The delete-database.js script now accepts a parameter that suppresses the delete prompt when being called from the resync process
-Reindex and normal block sync now reuse the same block sync code
-Reworked the sync script to no longer require the readline-sync dependency
-The readline-sync dependency has been completely removed from package.json since it is no longer used
-Although the explorer still seems to work fine with older node.js versions, the mongodb 6.x docs indicate that the minimum supported version is v16.20.1 and therefore the explorer has been updated to also enforce this as the min version to help prevent future problems
-A small fix was applied to the prestart script that was previously not stopping execution of the explorer when an older version of node.js was found. This was bug only affected running the explorer with `npm start`
-The initialize_data_startup() function is now being called in the update_explorer script to ensure that new changes to the database structure are automaticaly run after the explorer code is updated
-npm 9.7.2 added better color support detection which effectively broke the explorer's detection of whether pm2 or forever modules are installed due to extra color codes in the `npm list` output. This has been corrected so that pm2 and forever are now detected properly on newer and older npm versions
-Moved the database initialize call out of the instance code into the prestart script so that it is now only called once when you start the explorer even if you are running multiple cpus/threads
-The database connection function now uses the default connection string in the event a connection string is not supplied
-Removed unused reference to lib/explorer.js code from the cluster file
-The different methods of starting the explorer now set NODE_ENV to "development" or "production" depending on which options were chosen
-pm2 is now loaded from within the prestart script and will reload the config instead of starting if it was previously loaded which allows changing the NODE_ENV value if necessary
NOTE: if running via pm2 or forever when updating the explorer with this change, it is necessary to manually stop the explorer with `npm run stop-pm2` or `npm run stop-forever` and then start again after the code has been updated to allow NODE_ENV to be set to "production". Without doing this, the explorer will still be run in "development" mode until pm2 or forever is fully stopped and restarted.
-Mongoose and mongodb dependencies have been updated to the most recent versions in package.json
-Updated the project for use with the newest mongoose major version which required fixing many breaking changes to a number of files in the explorer
-Removed the db.find_address_tx() function as it is unused
NOTE: This update requires mongoose 6.0.10 or higher. If you are getting errors running the explorer after upgrading, it is recommended to run an 'npm update' or manually update mongoose to a more modern version which should fix the problem
-The block sync will now remove orphaned block data from the txes collecton, address balances/sent/received data as well as addresstx data and now stores limited info in a new orphans collection
-Added a new optional page for viewing orphaned block data
-The coinbase address now updates sent totals from POS rewards and other transactions that have vout but no vin addresses
-Block and transaction pages now display a warning when viewing an orphaned block or tx
-Added a couple new fields to the coinstats collection for tracking orphaned blocks
-Added new locale strings for the orphaned block feature
-Suppress the "DeprecationWarning: Mongoose: the `strictQuery` option will be switched back to `false` by default in Mongoose 7" warning msg when starting the explorer and using any of the scripts that connect to mongo
-Removed the check for country and country_code since not all ip addresses have country information and can cause a peer drop loop as a result
-Updated the error msg text when port data is not found for existing peers
-The SIGINT and SIGTERM signals are now being caught and handled in the sync.js file so that most options for syncing blocks, markets, peers, masternodes, etc. are now being gracefully shut down instead of killed in the middle of the process. This should help prevent data anomalies when you need to stop or kill a sync process
-The update_tx_db function was moved from the database.js file into the sync.js file so that block syncs can now be gracefully stopped. The update_tx_db function was also copied to the benchmark.js
-The save_tx function was moved into the module.exports for the database.js file so that it can now be called from outside the database.js file
-Use the cmd `npm run update-explorer "explorer-only"` to update the explorer code only, without checking for out-of-date dependencies
-Use the cmd `npm run update-explorer "dependencies-only"` to update outdated dependencies only, without checking for explorer code updates
-Updated the README with the 2 new script arguments
-The update_explorer.js script has been improved with better spacing and the ability to restart the explorer automatically to ensure new changes take effect immidiately (works with npm start, pm2 and forever)
-The code to compile scss to css has been moved from the prestart script into its own compile_css.js script which is now called from the update-explorer.js script to apply css changes after update
-The cluster code now handles a custom restart msg which is used to restart the explorer from the update explorer process
-Pm2 and Forever are now referenced by the name 'explorer' instead of ./bin/instance or ./bin/cluster [SEE IMPORTANT NOTE BELOW]
-Added reload/restart scripts to the package.json for pm2 and forever
-Pm2 and forever now write a pid file to the tmp directory when started. NOTE: Forever is now started from the prestart script due to a bug in forever that prevents the pid from being written to a different directory without the absolute path
-Fixed a bug which caused the prestart script to be run twice when starting the explorer with `npm start`
-The cluster code now accepts a numeric argument to force a specific number of instances to be loaded
-The `npm run start-instance` cmd now loads using the cluster code with a single instance
-The is_locked function now accepts an optional 'silent' argument to prevent displaying msgs while checking for pid/lock files
-Added some process.exit statements to the stop_explorer.js file
-Updated the README with cmd changes from package.json and updated description of the "Update Explorer Script"
IMPORTANT NOTE: It is strongly recommended to stop the explorer before performing this update. If the explorer is running while you perform this update, you will need to stop and restart the explorer for this update to fully take effect. Because of the changes in this commit, stopping the explorer using the built-in pm2 and/or forever stop cmds will not work and you will need to type out the full stop cmd this one time only, and going forward from now on you should no longer need to even stop the explorer for any update as it is now built into the update cmd.
If running using pm2 and you cannot stop the explorer, you can use stop using the following full cmd syntax:
Windows:
pm2 stop ./bin/instance
Linux and other OS's:
node node_modules/pm2/bin/pm2 stop ./bin/instance
If running using forever and you cannot stop the explorer, you can use stop using the following full cmd syntax:
All OS's:
node node_modules/forever/bin/forever stop ./bin/cluster
-Added a new port_filter setting to the 3 network_page tables which allows showing only results from peers on the selected port
-Added a new hide_protocols setting to the 3 network_page tables which allows hiding results from peers on the selected protocols
-Added a Port column to the data table on the Network page Connections tab
-The data table on the Network page Connections tab is now sortable
-The peer sync now refreshes data for peers that existed since last sync
-Adding and updating peers now displays the port # in the log output
-The /ext/getnetworkpeers api is now sorted by ip4/ip6, address and port which also means all data on the Network page is sorted this way by default as well
-The /ext/getnetworkpeers api no longer requires the internal argument for calls originating from the Network page
-The Network page now only makes a single call to the /ext/getnetworkpeers api instead of 3 calls
-The find_peer and drop_peer functions now requiresa port to distinguish between multiple connections to the same peer on different ports
-The custom.scss file can be used to add custom css changes without affecting the ability to do a git pull update
-Added a quick explanation of using the custom.scss file in the "Configure Explorer Settings" section of the README
-Added new script to A) grab the latest code form github, B) update all dependencies, and C) run the database initialize function to ensure new changes take effect immidiately
-Updated README to explain the new explorer update function
-freegeoip.app now requires an api key so the peer sync has been updated to use reallyfreegeoip.org instead
-Added extra error checking to the peer sync so that it will error out when the geolocation service does not return a proper object
-Added locks to all sync processes (blocks, markets, peers, masternodes) as well as "create backup", "restore backup" and "delete database" functions. This helps prevent problems with syncing data while a backup is in progress for example
-The code to initialize certain database collections on startup was moved into database.js and is now called from restore_backup.js and delete_database.js. This effectively allows the database to be deleted or restored to a completely different backup while the explorer is still running
-Lock functions (create_lock, remove_lock, is_locked) were moved into explorer.js for better reusability and rewriten to be synchronous
-is_locked function now accepts an array of lock files to be able to check for multiple locks in a single call
-remove_sync_message() function was moved into database.js so that restore_backup.js and delete_database.js can also check for and remove the sync msg if it exists
-Useful Scripts section updated in the README to make it clear that the explorer no longer needs to be stopped for these scripts to be run
-Most if not all log messages now start with a capitlal letter
-All shell scripts have been removed and replaced with javascript equivalents which allows for better platform independence
-All scripts have been improved over the older shell scripts to be more functional and dynamic
-Updated all applicable cmds in the package.json to target the new js scripts + added the backup, restore and delete database functions to the list
-Removed the json and strip-json-comments-cli packages as they are no longer needed
-Added a new package realine-sync
-Updated cluster code with better Windows support when shutting down the explorer
-Backup and Restore scripts now use mongo's own encryption instead of tar.gz by default. Older tar.gz backups can still be restored as long as the tar.gz suffix is explicitly added
-Backup and Restore scripts now support connecting to remote database based on the mongo details in settings.json
-Updated README to include a new line for the 'platform independence' feature, updated script cmds and notes about certain steps being Linux only
-Hashrate chart is populated from get_hashrate
-Difficulty chart is populated from get_difficulty
-Added a networkhistories collection to hold the new network data
-Index sync and resync functions now check for and update network history data at the end of the sync process
-restore_backup.sh and delete_database.sh scripts now drop the new networkhistories collection
-Added a new javascript library chartjs-plugin-crosshair which adds crosshair functionality to the new chart.js line charts
-Added new settings to settings.json for controlling things like collecting network_history data, independently enabling/disabling the network charts and changing colors of various aspects of the charts
-Updated README to introduce the network charts and added a line for the chartjs-plugin-crosshair library
-Shifted some of the code around to be run in a better order (for example, blockchain_last_updated date is set right after finishing the blockchain sync instead of at the very end of the reindex)
-update_heavy has been moved into a proper callback function which is now only called for index and reindex syncs right near the end of the process instead of at the beginning
-Added an "Are you sure" prompt before starting the reindex to make it clear that this is a destructive sync option
-All necessary coin stat data is cleared before starting the resync (txes, blockchain_last_updated and richlist_last_updated were not previously being cleared)
-More and better messages about what is being done during the process
-Shifted some of the code around to be run in a better order (for example, blockchain_last_updated date is set right after finishing the blockchain sync instead of at the very end of the reindex)
-Older versions of nodejs will throw all kinds of different errors depending on which version you are running. This check should hopefully prevent confusion about why the explorer is not working by giving a clearer error message before the explorer starts
-All node dependencies are up-to-date in package.json
-Updated semantic versioning of some node dependencies in package.json
-Applied fixes to necessary explorer files to fix breaking changes to mongoose dependency
-Update README with new Node.js and MongoDB version recommendations
NOTE: Node.js must be updated to at least v14.13.1 to fix breaking errors in a few of the updated dependencies
-Also add output displaying which block is currently being checked as before it was almost impossible to determine what, if anything was happening in check mode
-Market sync now sets the last USD price based on the default market pair even if it is not BTC
-Removed the coindesk api which was only really useful for getting the USD price of BTC
-Added a couple coingecko apis which are used to get the current USD value of the default market pair
-Updated the readme, settings file comments and info/api page with better verbiage regarding the market price which was previously only measured in BTC
-Many blockchains allow you to use addnodes without a port # and it will assume the default port internally, but the explorer was failing to parse these peers because it was expecting a port. This is now fixed and will display the peer data without a port in these cases
-Automatically detect if using newer or older Dash masternode format and display applicable columns and data
-Added new fields 'ip_address' and 'last_paid_block' to the Masternode collection
-More graceful shutdown of node cluster on 'npm stop' with better cleanup of resources on exit
-Added new stop_explorer.sh script which looks up the explorer port # via settings file and closes the application running on that port # instead of saving and killing the process by pid as it did before
-Added support for pm2 and forever using 'npm run start-pm2' and 'npm run start-forever' respectively
-pm2 is automatically installed when starting with 'npm run start-pm2' if it is not already installed
-forever is automatically installed when starting with 'npm run start-forever' if it is not already installed
-Updated existing npm commands in package.json by replacing hardcoded 'node' with '$(which node)'
-/path/to/nodejs changed to /path/to/node in the /settings.json.template, /lib/settings.js and /scripts/sync.js files
-README updates:
-Added a new 'Start/Stop the Explorer' section
-Added PM2 instructions to the 'Start/Stop the Explorer' section
-Moved Start/Stop Explorer instructions to the 'Start/Stop the Explorer' section
-Moved Forever instructions to the 'Start/Stop the Explorer' section
-/path/to/nodejs changed to /path/to/node
-Some additional small misc fixes
-I figure this should be changed now while it's fresh on my mind and may come in handy down the road to have the ability to auto-run 'npm update' when new packages are added and/or to more easily do any other future pre-run processing
-Greatly increases speed of database restore by removing the 'remove' cmds which are not only unnecessary but can be extremely time-consuming to run on large database collections while drop statements run very fast and free up everything required
-Greatly increases speed of database deletion by removing the 'remove' cmds which are not only unnecessary but can be extremely time-consuming to run on large database collections while drop statements run very fast and free up everything required
-Replace tabs with double-spaces
-Betting spacing and lining up of code functions
-Add missing semi-colons
-Remove extra characters and spaces where applicable
-Remove commented-out code fragments
-Add missing 2021 date to LICENSE
-Small touchups and other misc nitpickings
-Improved the display of the market page so that as long as one section has data (stats, buys, sells, trade history) the page will load properly even if it's only one section that isn't blank. Previously it would display an error if just one section was blank.
-Added the coin name beside the market pages volume header column in the 24h stats section
-Fixed a bug that was displaying an error when the summary value from the 24h stats section was supplied but was a blank string or '-' character
-Better success/error message during market sync
Joe Uhren