18 Commits

Author SHA1 Message Date
Joe Uhren 0b0ef817f1 "Maximum call stack size exceeded" error bug fixed
-The "Maximum call stack size exceeded" error is now handled internally by the block sync script in a way which will capture the error and re-launch the sync using a larger stack size and have the sync resume from where it left off. If the re-launch still doesn't have enough memory it will continue re-launching with more and more memory until the sync can finish without errors and then it will return to sync with a lower memory footprint for future syncs
-Added a new option for sync.elastic_stack_size which is used to determine how much memory should be used to increase the stack size for the block sync after encountering the "Maximum call stack size exceeded" error
-Fixed an issue with the block sync when using more than 1 thread that could sometimes cause the flattened txes value in the coinstats database to be written incorrectly (Use `npm run reindex-txcount` to fix this issue without needing to reindex the entire database)
-Updated the benchmark script so that it can also benefit from being able to capture the "Maximum call stack size exceeded" error even though the timing will be off so it outputs a new warning in that scenario which instructs to run the benchmark again with a higher stack size to properly capture the benchmark time
-Removed the "Maximum call stack size exceeded" error notes from the "Known Issues" section of the README
2025-01-09 20:00:37 -07:00
Joe Uhren 454fb0a7d7 Fixed multi-threaded sync + related improvements
-The block_parallel_tasks feature has been improved and fixed so that it is now safe to cancel (Ctrl+C) or kill (kill cmd not kill -9) the task and resume the sync later without missing transactions. The new block_parallel_tasks default is 8 threads which seems to be the sweet spot for any type of cpu
-Numerous improvements to the benchamark script to utilize new benchmark settings, auto-add credentials to the benchmark database, reuse the same sync code as the regular block sync instead of using a copy of the code and more
-Added a new cmd to run the benchmark script `npm run benchmark`
-README updated to include the new benchmark script instrutions + include multi-threaded sync as a feature
2025-01-01 19:20:13 -07:00
Joe Uhren d86beee960 Add new settings to save+display multi-algo data
-New settings allow reading of the hash algorithm used to mine a particular block for coins that support this feature and have the algorithm data stored in the raw block data
-An "Algorithm" column has been added to the block page and main transaction homepage when multi-algo data is enabled
-The /ext/getlasttxs api will now return the hash algorithm if reading of the multi-algo data is enabled
2024-02-06 19:44:11 -07:00
Joe Uhren 1a0ef69749 Added support for newer mongoose versions
-Mongoose and mongodb dependencies have been updated to the most recent versions in package.json
-Updated the project for use with the newest mongoose major version which required fixing many breaking changes to a number of files in the explorer
-Removed the db.find_address_tx() function as it is unused

NOTE: This update requires mongoose 6.0.10 or higher. If you are getting errors running the explorer after upgrading, it is recommended to run an 'npm update' or manually update mongoose to a more modern version which should fix the problem
2023-05-07 20:55:29 -06:00
Joe Uhren 4047587990 Fix strictQuery warning from new mongoose versions
-Suppress the "DeprecationWarning: Mongoose: the `strictQuery` option will be switched back to `false` by default in Mongoose 7" warning msg when starting the explorer and using any of the scripts that connect to mongo
2022-12-16 19:17:37 -07:00
Joe Uhren 1178038710 Better handling of stop/kill sync processes
-The SIGINT and SIGTERM signals are now being caught and handled in the sync.js file so that most options for syncing blocks, markets, peers, masternodes, etc. are now being gracefully shut down instead of killed in the middle of the process. This should help prevent data anomalies when you need to stop or kill a sync process
-The update_tx_db function was moved from the database.js file into the sync.js file so that block syncs can now be gracefully stopped. The update_tx_db function was also copied to the benchmark.js
-The save_tx function was moved into the module.exports for the database.js file so that it can now be called from outside the database.js file
2022-07-17 16:49:02 -06:00
Joe Uhren 40d367510d Better special char support for mongo user/pass
-URL encoding the mongo username and password helps prevent connection issues when using certain special characters such as % or @ for example
2022-06-13 19:37:48 -06:00
Joe Uhren d7c18a48f5 Improved (sync/backup) scripts + misc updates
-Added locks to all sync processes (blocks, markets, peers, masternodes) as well as "create backup", "restore backup" and "delete database" functions. This helps prevent problems with syncing data while a backup is in progress for example
-The code to initialize certain database collections on startup was moved into database.js and is now called from restore_backup.js and delete_database.js. This effectively allows the database to be deleted or restored to a completely different backup while the explorer is still running
-Lock functions (create_lock, remove_lock, is_locked) were moved into explorer.js for better reusability and rewriten to be synchronous
-is_locked function now accepts an array of lock files to be able to check for multiple locks in a single call
-remove_sync_message() function was moved into database.js so that restore_backup.js and delete_database.js can also check for and remove the sync msg if it exists
-Useful Scripts section updated in the README to make it clear that the explorer no longer needs to be stopped for these scripts to be run
-Most if not all log messages now start with a capitlal letter
2022-04-30 20:53:10 -06:00
Joe Uhren 59cee58cb0 Update node dependencies + fix breaking changes
-All node dependencies are up-to-date in package.json
-Updated semantic versioning of some node dependencies in package.json
-Applied fixes to necessary explorer files to fix breaking changes to mongoose dependency
-Update README with new Node.js and MongoDB version recommendations

NOTE: Node.js must be updated to at least v14.13.1 to fix breaking errors in a few of the updated dependencies
2021-12-04 12:44:35 -07:00
Joe Uhren e5e1887401 Check mode no longer updates the last block height
-Also add output displaying which block is currently being checked as before it was almost impossible to determine what, if anything was happening in check mode
2021-11-21 18:09:06 -07:00
joeuhren 20c0a382a3 Code cleanup
-Replace tabs with double-spaces
-Betting spacing and lining up of code functions
-Add missing semi-colons
-Remove extra characters and spaces where applicable
-Remove commented-out code fragments
-Add missing 2021 date to LICENSE
-Small touchups and other misc nitpickings
2021-03-17 17:54:09 -06:00
joeuhren 4790764e2c Massive overhaul and cleanup of settings
-Restructured the settings.json.template and settings.js files with better groupings
-Added better comments for improved explanations of all settings
-Better handling of page length options on all datatables; Page length options are now configurable and also only display options based on limits set by certain items_per_page and max_items_per_query settings
-Markets have been extended to support unlimited trading pairs for each exchange
-Added coin_symbol and pair_symbol to the markets schema to support multiple trading pairs
-Added a burned field to the richlist schema for tracking burned coin addresses
-Added a locale string for the new richlist wealth distribution "Top 1-100 Total" table row
-Updated a couple locale strings that were incomplete or needed fixing (api_getblockhash and api_getnextrewardwhenstr)
-Added a new css class to remove some redundant inline styles for the richlist wealth distribution color boxes
-Richlist page now allows better separation of the top list(s) and the wealth distribution table and chart (you can now hide or show sections as desired)
-Richlist page now allows for better burned coin support (Hide burned coin addresses from lists and totals even when the burned coin amounts are still included in the total coin supply value)
-Block page now only displays the raw block link if the api_page.public_apis.rpc.getblock.enabled settings is set to true
-Transaction page now only displays the raw tx link if the api_page.public_apis.rpc.getrawtransaction.enabled setting is set to true
-Rate limiting class has been made global and is now used to limit market requests
-getnetworkhashps rpc call now returns a '-' if shared_pages.show_hashrate is not set to true
-Searching for unsycned blocks/txs no longer saves the data locally but instead still shows the data on screen. This helps prevent syncing data out-of-order (This eliminates the need for db_index.pid which has been removed)
-Coin supply is now always taken from the stats collection database instead of from the wallet via rpc command in all cases except when syncing
-Lots of misc fixes and code cleanup changes
-List of changed settings:
  -title -> shared_pages.page_title
  -coin -> coin.name
  -symbol -> coin.symbol
  -logo -> shared_pages.logo
  -headerlogo -> shared_pages.page_header.home_link_logo
  -favicon -> shared_pages.favicon
  -homelink -> shared_pages.page_header.home_link
  -logoheight -> shared_pages.page_header.home_link_logo_height
  -sticky_header -> shared_pages.page_header.sticky_header
  -sticky_footer -> shared_pages.page_footer.sticky_footer
  -footer_height_desktop -> shared_pages.page_footer.footer_height_desktop
  -footer_height_tablet -> shared_pages.page_footer.footer_height_tablet
  -footer_height_mobile -> shared_pages.page_footer.footer_height_mobile
  -social_link_percent_height_desktop -> shared_pages.page_footer.social_link_percent_height_desktop
  -social_link_percent_height_tablet -> shared_pages.page_footer.social_link_percent_height_tablet
  -social_link_percent_height_mobile -> shared_pages.page_footer.social_link_percent_height_mobile
  -theme -> shared_pages.theme
  -port -> webserver.port
  -update_timeout -> sync.update_timeout
  -check_timeout -> sync.check_timeout
  -block_parallel_tasks -> sync.block_parallel_tasks
  -use_rpc -> api_cmds.use_rpc
  -confirmations -> shared_pages.confirmations
  -display.api -> api_page.enabled
  -display.markets -> markets_page.enabled
  -display.richlist -> richlist_page.enabled
  -display.search -> shared_pages.page_header.show_search
  -display.movement -> movement_page.enabled
  -display.network -> network_page.enabled
  -display.masternodes -> masternodes_page.enabled
  -display.claim_address -> claim_address_page.enabled
  -display.claim_address_header_menu -> claim_address_page.show_header_menu
  -display.page_header_bgcolor -> shared_pages.page_header.bgcolor
  -display.page_footer_bgcolor -> shared_pages.page_footer.bgcolor
  -display.table_header_bgcolor -> shared_pages.table_header_bgcolor
  -display.networkpnl -> shared_pages.page_header.panels.network_panel.display_order
  -display.difficultypnl -> shared_pages.page_header.panels.difficulty_panel.display_order
  -display.masternodespnl -> shared_pages.page_header.panels.masternodes_panel.display_order
  -display.coinsupplypnl -> shared_pages.page_header.panels.coin_supply_panel.display_order
  -display.pricepnl -> shared_pages.page_header.panels.price_panel.display_order
  -display.marketcappnl -> shared_pages.page_header.panels.market_cap_panel.display_order
  -display.logopnl -> shared_pages.page_header.panels.logo_panel.display_order
  -index.show_last_updated -> index_page.show_last_updated
  -index.show_hashrate -> shared_pages.show_hashrate
  -index.difficulty -> shared_pages.difficulty
  -index.last_txs -> api_page.public_apis.ext.getlasttxs.max_items_per_query
  -index.txs_per_page -> index_page.transaction_table.items_per_page (for index page) AND movement_page.movement_table.items_per_page (for movement page)
  -reward_page.show_last_updated -> blockchain_specific.heavycoin.reward_page.show_last_updated
  -api.blockindex -> api_page.sample_data.blockindex
  -api.blockhash -> api_page.sample_data.blockhash
  -api.txhash -> api_page.sample_data.txhash
  -api.address -> api_page.sample_data.address
  -markets.exchange -> markets_page.default_exchange.trading_pair
  -markets.default -> markets_page.default_exchange.exchange_name
  -markets.market_dropdown_menu -> markets_page.show_market_dropdown_menu
  -markets.market_select_visible -> markets_page.show_market_select
  -richlist.distribution -> richlist_page.wealth_distribution.show_distribution_chart
  -richlist.received -> richlist_page.show_received_coins
  -richlist.balance -> richlist_page.show_current_balance
  -movement.min_amount -> movement_page.movement_table.min_amount
  -movement.low_flag -> movement_page.movement_table.low_warning_flag
  -movement.high_flag -> movement_page.movement_table.high_warning_flag
  -social_links -> shared_pages.page_footer.social_links
  -social_links[x].image_url -> shared_pages.page_footer.social_links[x].image_path
  -genesis_tx -> transaction_page.genesis_tx
  -genesis_block -> block_page.genesis_block
  -heavy -> blockchain_specific.heavycoin.enabled
  -save_stats_after_sync_blocks -> sync.save_stats_after_sync_blocks
  -txcount -> api_page.public_apis.ext.getaddresstxs.max_items_per_query
  -txcount_per_page -> address_page.history_table.items_per_page
  -show_sent_received -> address_page.show_sent_received
  -supply -> sync.supply
  -nethash -> shared_pages.page_header.panels.network_panel.nethash
  -nethash_units -> shared_pages.page_header.panels.network_panel.nethash_units
  -usecors -> webserver.cors.enabled
  -corsorigin -> webserver.cors.corsorigin
  -burned_coins -> richlist_page.burned_coins.addresses
  -public_api.rpc.getdifficulty -> api_page.public_apis.rpc.getdifficulty.enabled
  -public_api.rpc.getconnectioncount -> api_page.public_apis.rpc.getconnectioncount.enabled
  -public_api.rpc.getblockcount -> api_page.public_apis.rpc.getblockcount.enabled
  -public_api.rpc.getblockhash -> api_page.public_apis.rpc.getblockhash.enabled
  -public_api.rpc.getblock -> api_page.public_apis.rpc.getblock.enabled
  -public_api.rpc.getrawtransaction -> api_page.public_apis.rpc.getrawtransaction.enabled
  -public_api.rpc.getnetworkhashps -> api_page.public_apis.rpc.getnetworkhashps.enabled
  -public_api.rpc.getvotelist -> api_page.public_apis.rpc.getvotelist.enabled
  -public_api.rpc.getmasternodecount -> api_page.public_apis.rpc.getmasternodecount.enabled
  -public_api.rpc.getmaxmoney -> blockchain_specific.heavycoin.public_apis.getmaxmoney.enabled
  -public_api.rpc.getmaxvote -> blockchain_specific.heavycoin.public_apis.getmaxvote.enabled
  -public_api.rpc.getvote -> blockchain_specific.heavycoin.public_apis.getvote.enabled
  -public_api.rpc.getphase -> blockchain_specific.heavycoin.public_apis.getphase.enabled
  -public_api.rpc.getreward -> blockchain_specific.heavycoin.public_apis.getreward.enabled
  -public_api.rpc.getsupply -> blockchain_specific.heavycoin.public_apis.getsupply.enabled
  -public_api.rpc.getnextrewardestimate -> blockchain_specific.heavycoin.public_apis.getnextrewardestimate.enabled
  -public_api.rpc.getnextrewardwhenstr -> blockchain_specific.heavycoin.public_apis.getnextrewardwhenstr.enabled
  -public_api.ext.getmoneysupply -> api_page.public_apis.ext.getmoneysupply.enabled
  -public_api.ext.getdistribution -> api_page.public_apis.ext.getdistribution.enabled
  -public_api.ext.getaddress -> api_page.public_apis.ext.getaddress.enabled
  -public_api.ext.getaddresstxs -> api_page.public_apis.ext.getaddresstxs.enabled
  -public_api.ext.gettx -> api_page.public_apis.ext.gettx.enabled
  -public_api.ext.getbalance -> api_page.public_apis.ext.getbalance.enabled
  -public_api.ext.getlasttxs -> api_page.public_apis.ext.getlasttxs.enabled
  -public_api.ext.getcurrentprice -> api_page.public_apis.ext.getcurrentprice.enabled
  -public_api.ext.getbasicstats -> api_page.public_apis.ext.getbasicstats.enabled
  -public_api.ext.getsummary -> api_page.public_apis.ext.getsummary.enabled
  -public_api.ext.getnetworkpeers -> api_page.public_apis.ext.getnetworkpeers.enabled
  -public_api.ext.getmasternodelist -> api_page.public_apis.ext.getmasternodelist.enabled
  -public_api.ext.getmasternoderewards -> api_page.public_apis.ext.getmasternoderewards.enabled
  -public_api.ext.getmasternoderewardstotal -> api_page.public_apis.ext.getmasternoderewardstotal.enabled
  -api_cmds.heavies.getmaxmoney -> blockchain_specific.heavycoin.api_cmds.getmaxmoney
  -api_cmds.heavies.getmaxvote -> blockchain_specific.heavycoin.api_cmds.getmaxvote
  -api_cmds.heavies.getvote -> blockchain_specific.heavycoin.api_cmds.getvote
  -api_cmds.heavies.getphase -> blockchain_specific.heavycoin.api_cmds.getphase
  -api_cmds.heavies.getreward -> blockchain_specific.heavycoin.api_cmds.getreward
  -api_cmds.heavies.getnextrewardestimate -> blockchain_specific.heavycoin.api_cmds.getnextrewardestimate
  -api_cmds.heavies.getnextrewardwhenstr -> blockchain_specific.heavycoin.api_cmds.getnextrewardwhenstr
  -api_cmds.heavies.getsupply -> blockchain_specific.heavycoin.api_cmds.getsupply
-List of new settings:
  -shared_pages.page_header.panels.network_panel.enabled: allow enabling/disabling of the network panel
  -shared_pages.page_header.panels.difficulty_panel.enabled: allow enabling/disabling of the difficulty panel
  -shared_pages.page_header.panels.masternodes_panel.enabled: allow enabling/disabling of the masternodes panel
  -shared_pages.page_header.panels.coin_supply_panel.enabled: allow enabling/disabling of the coin supply panel
  -shared_pages.page_header.panels.price_panel.enabled: allow enabling/disabling of the price panel
  -shared_pages.page_header.panels.market_cap_panel.enabled: allow enabling/disabling of the market cap panel
  -shared_pages.page_header.panels.logo_panel.enabled: allow enabling/disabling of the logo panel
  -index_page.transaction_table.page_length_options: specify the page length options that determine how many items/records to display in the table at any given time
  -index_page.transaction_table.reload_table_seconds: the time in seconds to automatically reload the table data from the server
  -address_page.history_table.page_length_options: specify the page length options that determine how many items/records to display in the table at any given time
  -masternodes_page.masternode_table.page_length_options: specify the page length options that determine how many items/records to display in the table at any given time
  -masternodes_page.masternode_table.items_per_page: the default amount of items/records to display in the table at any given time
  -movement_page.movement_table.page_length_options: specify the page length options that determine how many items/records to display in the table at any given time
  -movement_page.movement_table.reload_table_seconds: the time in seconds to automatically reload the table data from the server
  -network_page.connections_table.page_length_options: specify the page length options that determine how many items/records to display in the table at any given time
  -network_page.connections_table.items_per_page: the default amount of items/records to display in the table at any given time
  -network_page.addnodes_table.page_length_options: specify the page length options that determine how many items/records to display in the table at any given time
  -network_page.addnodes_table.items_per_page: the default amount of items/records to display in the table at any given time
  -network_page.onetry_table.page_length_options: specify the page length options that determine how many items/records to display in the table at any given time
  -network_page.onetry_table.items_per_page: the default amount of items/records to display in the table at any given time
  -richlist_page.wealth_distribution.show_distribution_table: show/hide the wealth distribution summary table
  -richlist_page.wealth_distribution.colors: a list of html color codes to represent the top 100 groupings in the wealth distribution table and pie chart
  -richlist_page.burned_coins.include_burned_coins_in_distribution: determine whether to include burned coins in the wealth distribution section or not
  -markets_page.exchanges.altmarkets.enabled: enable/disable the altmarkets exchange
  -markets_page.exchanges.altmarkets.trading_pairs: a list of market trading pair symbols
  -markets_page.exchanges.bittrex.enabled: enable/disable the bittrex exchange
  -markets_page.exchanges.bittrex.trading_pairs: a list of market trading pair symbols
  -markets_page.exchanges.bleutrade.enabled: enable/disable the bleutrade exchange
  -markets_page.exchanges.bleutrade.trading_pairs: a list of market trading pair symbols
  -markets_page.exchanges.crex.enabled: enable/disable the crex exchange
  -markets_page.exchanges.crex.trading_pairs: a list of market trading pair symbols
  -markets_page.exchanges.poloniex.enabled: enable/disable the poloniex exchange
  -markets_page.exchanges.poloniex.trading_pairs: a list of market trading pair symbols
  -markets_page.exchanges.stex.enabled: enable/disable the stex exchange
  -markets_page.exchanges.stex.trading_pairs: a list of market trading pair symbols
  -markets_page.exchanges.yobit.enabled: enable/disable the yobit exchange
  -markets_page.exchanges.yobit.trading_pairs: a list of market trading pair symbols
  -claim_address_page.enable_bad_word_filter: enable/disable the "bad word" filter for claimed addresses, so that trying to claim an address with a bad word will fail
  -sync.show_sync_msg_when_syncing_more_than_blocks: show the sync msg at the top of all pages during index syncronization if there are more than this many blocks to process
  -labels[x].enabled: allow enabling/disabling of specific wallet address labels
  -blockchain_specific.heavycoin.reward_page.enabled: enable/disable the reward page
  -blockchain_specific.zksnarks.enabled: enable/disable Zcash zk-SNARKs private transaction support (WIP - 90% complete)
-List of deleted settings
  -address: unnecessary setting has been replaced by dynamic values via http request
  -markets.coin: replaced by a richer set of settings that allow choosing the coin for each market
  -markets.enabled -> replaced by a richer set of settings that allow enabling/disabling each market specifically
  -lock_during_index: no longer possible to save unsynced blocks/txs via search so this settings is now obsolete/unused
2021-01-22 15:04:32 -07:00
joeuhren 6d74c3e6de Flatten tx count value for faster lookups with large blockchains 2020-12-03 14:16:34 -07:00
joeuhren e7ce0cef27 Fix all collection.count is deprecated warnings 2020-11-28 14:44:42 -07:00
joeuhren ea54a79de0 Fix mongoose usefindandmodify depreciation msg 2020-11-22 17:39:46 -07:00
joeuhren df35341be2 Update Mongoose calls: remove > deleteMany 2020-11-20 14:18:32 -07:00
joeuhren f500de2696 Fix for Server Discovery and Monitoring engine Deprecation Warning 2020-03-22 13:34:28 -06:00
Joe Uhren 60fb8e29b4 Initial release 2019-05-27 10:33:22 -07:00