davide/purple-explorer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Various core improvements and easier updating

Browse Source 
-The update_explorer.js script has been improved with better spacing and the ability to restart the explorer automatically to ensure new changes take effect immidiately (works with npm start, pm2 and forever)
-The code to compile scss to css has been moved from the prestart script into its own compile_css.js script which is now called from the update-explorer.js script to apply css changes after update
-The cluster code now handles a custom restart msg which is used to restart the explorer from the update explorer process
-Pm2 and Forever are now referenced by the name 'explorer' instead of ./bin/instance or ./bin/cluster [SEE IMPORTANT NOTE BELOW]
-Added reload/restart scripts to the package.json for pm2 and forever
-Pm2 and forever now write a pid file to the tmp directory when started. NOTE: Forever is now started from the prestart script due to a bug in forever that prevents the pid from being written to a different directory without the absolute path
-Fixed a bug which caused the prestart script to be run twice when starting the explorer with `npm start`
-The cluster code now accepts a numeric argument to force a specific number of instances to be loaded
-The `npm run start-instance` cmd now loads using the cluster code with a single instance
-The is_locked function now accepts an optional 'silent' argument to prevent displaying msgs while checking for pid/lock files
-Added some process.exit statements to the stop_explorer.js file
-Updated the README with cmd changes from package.json and updated description of the "Update Explorer Script"

IMPORTANT NOTE: It is strongly recommended to stop the explorer before performing this update. If the explorer is running while you perform this update, you will need to stop and restart the explorer for this update to fully take effect. Because of the changes in this commit, stopping the explorer using the built-in pm2 and/or forever stop cmds will not work and you will need to type out the full stop cmd this one time only, and going forward from now on you should no longer need to even stop the explorer for any update as it is now built into the update cmd.

If running using pm2 and you cannot stop the explorer, you can use stop using the following full cmd syntax:

Windows:
pm2 stop ./bin/instance

Linux and other OS's:
node node_modules/pm2/bin/pm2 stop ./bin/instance

If running using forever and you cannot stop the explorer, you can use stop using the following full cmd syntax:

All OS's:
node node_modules/forever/bin/forever stop ./bin/cluster
This commit is contained in:
Joe Uhren
2022-07-03 19:13:50 -06:00
parent 8fa337f6f9
commit bae4d50087
9 changed files with 310 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+41 -7
View File
@@ -38,8 +38,10 @@ Table of Contents
- [Start Explorer Using PM2 (Recommended for Production)](#start-explorer-using-pm2-recommended-for-production)
- [Start Explorer Using PM2 and Log Viewer](#start-explorer-using-pm2-and-log-viewer)
- [Stop Explorer Using PM2 (Recommended for Production)](#stop-explorer-using-pm2-recommended-for-production)
- [Reload Explorer Using PM2 (Recommended for Production)](#reload-explorer-using-pm2-recommended-for-production)
- [Start Explorer Using Forever (Alternate Production Option)](#start-explorer-using-forever-alternate-production-option)
- [Stop Explorer Using Forever (Alternate Production Option)](#stop-explorer-using-forever-alternate-production-option)
- [Reload Explorer Using Forever (Alternate Production Option)](#reload-explorer-using-forever-alternate-production-option)
- [Syncing Databases with the Blockchain](#syncing-databases-with-the-blockchain)
- [Commands for Manually Syncing Databases](#commands-for-manually-syncing-databases)
- [Sample Crontab](#sample-crontab)
@@ -311,7 +313,7 @@ npm run start-instance
or (useful for crontab):
```
cd /path/to/explorer && /path/to/npm run prestart && /path/to/node --stack-size=10000 ./bin/instance
cd /path/to/explorer && /path/to/npm run prestart && /path/to/node --stack-size=10000 ./bin/cluster 1
```
#### Stop Explorer (Use for Testing)
@@ -341,7 +343,7 @@ npm run start-pm2
or (useful for crontab):
```
cd /path/to/explorer && /path/to/npm run prestart "pm2" && /path/to/pm2 start ./bin/instance -i 0 --node-args="--stack-size=10000"
cd /path/to/explorer && /path/to/npm run prestart "pm2" && /path/to/pm2 start ./bin/instance -i 0 -n explorer -p "./tmp/pm2.pid" --node-args="--stack-size=10000"
```
**NOTE:** Use the following cmd to find the install path for PM2 (Linux only):
@@ -361,7 +363,7 @@ npm run start-pm2-debug
or (useful for crontab):
```
cd /path/to/explorer && /path/to/npm run prestart "pm2" && /path/to/pm2 start ./bin/instance -i 0 --node-args="--stack-size=10000" && /path/to/pm2 logs
cd /path/to/explorer && /path/to/npm run prestart "pm2" && /path/to/pm2 start ./bin/instance -i 0 -n explorer -p "./tmp/pm2.pid" --node-args="--stack-size=10000" && /path/to/pm2 logs
```
#### Stop Explorer Using PM2 (Recommended for Production)
@@ -375,7 +377,23 @@ npm run stop-pm2
or (useful for crontab):
```
cd /path/to/explorer && /path/to/pm2 stop ./bin/instance
cd /path/to/explorer && /path/to/pm2 stop explorer
```
#### Reload Explorer Using PM2 (Recommended for Production)
The explorer can be stopped and restarted in a single cmd when it is running via PM2, which is often necessary after updating the explorer code for example. Use one of the following terminal cmds to reload the explorer (be sure to run from within the explorer directory):
**NOTE:** Assuming the explorer has access to 2 or more cpus, this reload will be done in such a way that there will be zero-downtime while the restart is being performed. If you only have a single cpu then the explorer will be inaccessible for a few seconds while the restart is being performed.
```
npm run reload-pm2
```
or (useful for crontab):
```
cd /path/to/explorer && /path/to/pm2 reload explorer
```
#### Start Explorer Using Forever (Alternate Production Option)
@@ -391,7 +409,7 @@ npm run start-forever
or (useful for crontab):
```
cd /path/to/explorer && /path/to/npm run prestart && /path/to/forever start ./bin/cluster
cd /path/to/explorer && /path/to/npm run prestart "forever"
```
**NOTE:** Use the following cmd to find the install path for forever (Linux only):
@@ -411,7 +429,23 @@ npm run stop-forever
or (useful for crontab):
```
cd /path/to/explorer && /path/to/forever stop ./bin/cluster
cd /path/to/explorer && /path/to/forever stop "explorer"
```
#### Reload Explorer Using Forever (Alternate Production Option)
The explorer can be stopped and restarted in a single cmd when it is running via forever, which is often necessary after updating the explorer code for example. Use one of the following terminal cmds to reload the explorer (be sure to run from within the explorer directory):
**NOTE:** The explorer will be inaccessible for a few seconds while the restart is being performed.
```
npm run reload-forever
```
or (useful for crontab):
```
cd /path/to/explorer && /path/to/forever restart "explorer"
```
### Syncing Databases with the Blockchain
@@ -726,7 +760,7 @@ jQuery(document).ready(function($) {
#### Update Explorer Script
Automatically download and install the newest explorer source code, update out-of-date dependencies and initialize new changes with a single command. In most cases this update script can be safely run while the explorer is actively running to prevent needing to shut down to do updates, but please note that it may be possible for certain updates with large changes to require a reboot to the explorer for all changes to take effect properly. If you notice the explorer having issues after updating, try shutting down and restarting the explorer before seeking support.
Automatically download and install the newest explorer source code, update out-of-date dependencies and reload the explorer with a single command. This update script can be safely run while the explorer is actively running to prevent needing to manually shut down to do updates, but please note that the website may be inaccessible for a few seconds or more while the explorer is being updated.
**NOTE:** Only explorer installations that were installed via cloning the source from git can be automatically updated. Be sure to follow the [Quick Install Instructions](#quick-install-instructions) to set up the explorer for optimum use with this update script.
app.js
+15 -2
View File
@@ -10,8 +10,7 @@ var express = require('express'),
lib = require('./lib/explorer'),
db = require('./lib/database'),
package_metadata = require('./package.json'),
locale = require('./lib/locale'),
request = require('postman-request');
locale = require('./lib/locale');
var app = express();
var apiAccessList = [];
const { exec } = require('child_process');
@@ -602,6 +601,20 @@ app.use('/ext/getnetworkchartdata', function(req, res) {
});
});
app.use('/system/restartexplorer', function(req, res, next) {
// check to ensure this special cmd is only executed by the local server
if (req._remoteAddress != null && req._remoteAddress.indexOf('127.0.0.1') > -1) {
// send a msg to the cluster process telling it to restart
process.send('restart');
res.end();
} else {
// show the error page
var err = new Error('Not Found');
err.status = 404;
next(err);
}
});
var market_data = [];
var market_count = 0;
bin/cluster
+50 -21
View File
@@ -1,31 +1,42 @@
var cluster = require('cluster');
var fs = require('fs');
const isWinOS = process.platform == 'win32';
if (cluster.isMaster) {
const isWinOS = process.platform == 'win32';
const lib = require('../lib/explorer');
const instances = (process.argv[2] != null && process.argv[2] != '' && !isNaN(process.argv[2]) && Number.isInteger(parseFloat(process.argv[2])) ? parseInt(process.argv[2]) : require('os').cpus().length);
console.log('Starting cluster with pid: ' + process.pid);
// ensure workers exit cleanly
process.on('SIGINT', () => {
console.log('Cluster shutting down..');
function startWorkers() {
// create worker instances
for (var i = 0; i < instances; i += 1)
cluster.fork();
}
function killAllWorkers(isRestart) {
// send kill cmd to all workers
for (var id in cluster.workers) {
console.log('Worker (' + id + ') shutting down...');
if (!isWinOS) {
// only kill the worker if not on windows (otherwise an error is displayed)
if (!isWinOS || isRestart) {
// only kill the worker if not on windows (otherwise an error is displayed) or if the cluster is being restarted
process.kill(cluster.workers[id]['process']['pid'], 'SIGINT');
}
}
}
}
function waitForWorkerShutdown () {
if (Object.keys(cluster.workers).length > 0) {
// continue waiting since worker threads are still open
setTimeout(waitForWorkerShutdown, 100);
function waitForWorkerShutdown(isRestart) {
if (Object.keys(cluster.workers).length > 0) {
// continue waiting since worker threads are still open
setTimeout(waitForWorkerShutdown, 100, isRestart);
} else {
// all worker threads have closed
// check if the cluster process should stop or if the workers should be restarted
if (isRestart) {
// start new worker threads
startWorkers();
} else {
// all worker threads have closed
// now exit the master process
// exit the master process
if (isWinOS) {
// command line in windows doesn't seem to release itself
@@ -36,16 +47,20 @@ if (cluster.isMaster) {
process.exit(0);
}
}
}
waitForWorkerShutdown();
// ensure workers exit cleanly
process.on('SIGINT', () => {
console.log('Cluster shutting down..');
// send kill cmd to all workers
killAllWorkers(false);
// wait for workers to stop
waitForWorkerShutdown(false);
});
// count the machine's CPUs
var cpuCount = require('os').cpus().length;
// create a worker for each CPU
for (var i = 0; i < cpuCount; i += 1)
cluster.fork();
startWorkers();
// listen for dying workers
cluster.on('exit', function (worker, code, signal) {
@@ -56,5 +71,19 @@ if (cluster.isMaster) {
cluster.fork();
}
});
// listen for the restart cmd
cluster.on('message', function (data, msg) {
// check if this was the restart cmd
if (msg != null && msg == 'restart') {
console.log('Cluster restarting..');
// send kill cmd to all workers
killAllWorkers(true);
// wait for workers to stop
waitForWorkerShutdown(true);
}
});
} else
require('./instance');
lib/explorer.js
+6 -3
View File
@@ -1371,7 +1371,7 @@ module.exports = {
}
},
is_locked: function(lock_array) {
is_locked: function(lock_array, silent = false) {
const fs = require('fs');
const path = require('path');
var retVal = false;
@@ -1418,11 +1418,14 @@ module.exports = {
try {
fs.rmSync(pidFile);
} catch(err) {
console.log(`Failed to delete lock file ${pidFile}: ${err}`);
if (!silent)
console.log(`Failed to delete lock file ${pidFile}: ${err}`);
}
} else {
// script is running
console.log(`${lock_array[i]} script is running..`);
if (!silent)
console.log(`${lock_array[i]} script is running..`);
retVal = true;
break;
package.json
+9 -7
View File
@@ -3,14 +3,16 @@
"version": "1.100.0",
"private": true,
"scripts": {
"start": "npm run prestart && node --stack-size=10000 ./bin/cluster",
"start": "node --stack-size=10000 ./bin/cluster",
"stop": "node ./scripts/stop_explorer.js",
"start-instance": "npm run prestart && node --stack-size=10000 ./bin/instance",
"start-forever": "npm run prestart \"forever\" && forever start ./bin/cluster",
"stop-forever": "forever stop ./bin/cluster",
"start-pm2": "npm run prestart \"pm2\" && pm2 start ./bin/instance -i 0 --node-args=\"--stack-size=10000\"",
"start-pm2-debug": "npm run prestart \"pm2\" && pm2 start ./bin/instance -i 0 --node-args=\"--stack-size=10000\" && pm2 logs",
"stop-pm2": "pm2 stop ./bin/instance",
"start-instance": "npm run prestart && node --stack-size=10000 ./bin/cluster 1",
"start-forever": "npm run prestart \"forever\"",
"stop-forever": "forever stop \"explorer\"",
"reload-forever": "forever restart \"explorer\"",
"start-pm2": "npm run prestart \"pm2\" && pm2 start ./bin/instance -i 0 -n explorer -p \"./tmp/pm2.pid\" --node-args=\"--stack-size=10000\"",
"start-pm2-debug": "npm run prestart \"pm2\" && pm2 start ./bin/instance -i 0 -n explorer -p \"./tmp/pm2.pid\" --node-args=\"--stack-size=10000\" && pm2 logs",
"stop-pm2": "pm2 stop explorer",
"reload-pm2": "pm2 reload explorer",
"test": "node ./node_modules/jasmine/bin/jasmine.js test/*Spec.js",
"prestart": "node ./scripts/prestart.js",
"sync-blocks": "node ./scripts/sync.js index update",
scripts/compile_css.js
+24
View File
@@ -0,0 +1,24 @@
const fs = require('fs');
const settings = require('../lib/settings');
console.log('Compiling CSS.. Please wait..');
// ensure the selected theme is properly installed
fs.writeFile('./public/css/_theme-selector.scss', `$theme-name: "${settings.shared_pages.theme}";`, function (err) {
const sass = require('sass');
// generate minified css from style.scss file
const minified = sass.compile('./public/css/style.scss', {style: 'compressed'});
// save the minified css to file
fs.writeFile('./public/css/style.min.css', minified.css, function (err) {
// generate minified css from custom.scss file
const custom_minified = sass.compile('./public/css/custom.scss', {style: 'compressed'});
// save the minified css to file
fs.writeFile('./public/css/custom.min.css', custom_minified.css, function (err) {
// finished compiling css
process.exit(0);
});
});
});
scripts/prestart.js
+24 -26
View File
@@ -33,6 +33,8 @@ if (!(nodeVersionMajor > minNodeVersionMajor || (nodeVersionMajor == minNodeVers
}
function check_argument_passed(cb) {
const pidName = (process.argv[2] != null && process.argv[2] != '' && (process.argv[2] == 'pm2' || process.argv[2] == 'forever') ? process.argv[2] : 'node');
// check 1st argument
if (process.argv[2] != null) {
const { exec } = require('child_process');
@@ -54,11 +56,11 @@ function check_argument_passed(cb) {
// install pm2
exec(`npm install pm2@latest${(isWinOS ? ' -g' : '')}`, (err, stdout, stderr) => {
// always return true for now without checking results
return cb(true);
// always return the pidName for now without checking results
return cb(pidName);
});
} else
return cb(true);
return cb(pidName);
});
break;
case 'forever':
@@ -73,42 +75,38 @@ function check_argument_passed(cb) {
// install forever
exec('npm install forever', (err, stdout, stderr) => {
// always return true for now without checking results
return cb(true);
// always return the pidName for now without checking results
return cb(pidName);
});
} else
return cb(true);
return cb(pidName);
});
break;
default:
// argument not passed or unknown argument
return cb(true);
return cb(pidName);
}
} else
return cb(true);
return cb(pidName);
}
// check if an argument was passed into this script
check_argument_passed(function(retVal) {
const fs = require('fs');
const settings = require('../lib/settings');
check_argument_passed(function(pidName) {
const execSync = require('child_process').execSync;
// ensure the selected theme is properly installed
fs.writeFile('./public/css/_theme-selector.scss', `$theme-name: "${settings.shared_pages.theme}";`, function (err) {
const sass = require('sass');
// compile scss to css
execSync('node ./scripts/compile_css.js', {stdio : 'inherit'});
// generate minified css from style.scss file
const minified = sass.compile('./public/css/style.scss', {style: 'compressed'});
// check if the webserver should be started from here based on the pidName
if (pidName == 'forever') {
const path = require('path');
// save the minified css to file
fs.writeFile('./public/css/style.min.css', minified.css, function (err) {
// generate minified css from custom.scss file
const custom_minified = sass.compile('./public/css/custom.scss', {style: 'compressed'});
// there is a long-time bug or shortcoming in forever that still exists in the latest version which requires the absolute path to the pid file option
// more info: https://github.com/foreversd/forever/issues/421
// forever is therefore started from here to be able to more easily resolve the absolute path
execSync(`forever start --append --uid "explorer" --pidFile "${path.resolve('./tmp/forever.pid')}" ./bin/cluster`, {stdio : 'inherit'});
}
// save the minified css to file
fs.writeFile('./public/css/custom.min.css', custom_minified.css, function (err) {
// Finished pre-loading
});
});
});
// finished pre-loading
process.exit(0);
});
scripts/stop_explorer.js
+3
View File
@@ -48,13 +48,16 @@ if (validate_port(settings.webserver.port) == true) {
exec(killcmd, (err, stdout, stderr) => {
// show shutdown msg
console.log('Explorer shutting down... Please wait...');
process.exit(0);
});
} else {
// webserver is not running
console.log('Error: Cannot stop explorer because it is not currently running');
process.exit(1);
}
});
} else {
// invalid port number
console.log('Error: webserver.port value not found in settings.json.');
process.exit(1);
}
scripts/update_explorer.js
+138 -44
View File
@@ -1,78 +1,172 @@
const { execSync } = require('child_process');
const fs = require('fs');
const mongoose = require('mongoose');
// exit function used to cleanup before finishing script
function exit(exitCode) {
// disconnect mongo connection
mongoose.disconnect();
var reloadWebserver = false;
// exit process
process.exit(exitCode);
function exit() {
console.log('Explorer update complete');
process.exit(0);
}
var response;
function compile_css() {
// compile scss to css
execSync('node ./scripts/compile_css.js', {stdio : 'inherit'});
}
// check if the .git directory and .git/refs/heads/master file exist
if (fs.existsSync('./.git') && fs.existsSync('./.git/refs/heads/master')) {
// get the current commit hash
var commit = fs.readFileSync('./.git/refs/heads/master');
// check if the .git directory exists
if (fs.existsSync('./.git')) {
// update to newest explorer source
console.log('Downloading newest explorer code.. Please wait..');
console.log('Downloading newest explorer code.. Please wait..\n');
try {
response = execSync('git pull');
console.log('Git response:');
execSync('git pull', {stdio : 'inherit'});
// split response string by new line
var splitResponse = (response == null ? '' : response.toString()).split('\n').filter(element => element);
// get the current commit hash to see if it has changed
var new_commit = fs.readFileSync('./.git/refs/heads/master');
// check if the response was a single line which indicates it was already up-to-date
if (splitResponse.length == 1) {
// check if the commit values are the same
if (new_commit.toString() == commit.toString()) {
// explorer code was already up-to-date
console.log('Explorer code is already up-to-date');
console.log('\nExplorer code is already up-to-date');
} else {
console.log(response.toString().trim());
console.log('Explorer code successfully updated');
console.log('\nExplorer code successfully updated');
reloadWebserver = true;
}
} catch(err) {
console.log('Error updating explorer code. Maybe git is not installed globally?');
console.log('\nError updating explorer code. Maybe git is not installed globally or you made some custom changes to the explorer code?');
}
} else {
console.log('WARNING: Explorer code not cloned from github and cannot be automatically updated!');
console.log('Skipping explorer code update');
}
// update npm modules to latest versions according to package.json rules
console.log('Updating out-of-date explorer packages.. Please wait..');
execSync('npm update');
var outdatedPkgs = null;
// check for outdated packages
try {
console.log('\nChecking for outdated packages.. Please wait..');
execSync('npm outdated');
// all packages are up-to-date
console.log('All explorer packages are up-to-date');
console.log('\nAll explorer packages are up-to-date');
} catch (err) {
console.log(`The following packages are still out-of-date:\n${err.stdout.toString().trim()}`);
outdatedPkgs = err.stdout.toString().trim();
}
// load database and settings files after being updated
const db = require('../lib/database');
const settings = require('../lib/settings');
// add a new line for better spacing
console.log('');
var dbString = 'mongodb://' + encodeURIComponent(settings.dbsettings.user);
dbString = dbString + ':' + encodeURIComponent(settings.dbsettings.password);
dbString = dbString + '@' + settings.dbsettings.address;
dbString = dbString + ':' + settings.dbsettings.port;
dbString = dbString + '/' + settings.dbsettings.database;
// check if there were any outdated packages
if (outdatedPkgs != null) {
// update npm modules to latest versions according to package.json rules
console.log('Updating out-of-date explorer packages.. Please wait..\n');
execSync('npm update');
// connect to mongo database
mongoose.connect(dbString, function(err) {
if (err) {
console.log('Error: Unable to connect to database: %s', dbString);
exit(999);
} else {
// initialize the database
db.initialize_data_startup(function() {
exit(0);
});
// check for outdated packages (again)
try {
execSync('npm outdated');
// all packages are up-to-date
console.log('All explorer packages are up-to-date\n');
reloadWebserver = true;
} catch (err) {
console.log(`The following packages are still out-of-date:\n${err.stdout.toString().trim()}\n`);
// check if any of the packages were updated
if (err.stdout.toString().trim() == outdatedPkgs)
reloadWebserver = true;
}
});
}
// check if the web server should be reloaded
if (reloadWebserver == true) {
console.log('Checking if webserver is running.. Please wait..\n');
const path = require('path');
const lib = require('../lib/explorer');
var pidActive = false;
// get a list of all files in the tmp directory
var tmpFiles = fs.readdirSync('./tmp');
// get a list of all pm2 pid files
var pm2Files = tmpFiles
.filter(file => file.startsWith('pm2') && file.endsWith('.pid'))
.map(file => path.basename(file, '.pid'));
// loop through the pm2 pid files and check if at least one is valid/active by testing the pid to see if it is running
for (var i = 0; i < pm2Files.length; i++) {
// check if the current pm2.pid file is valid
if (lib.is_locked([pm2Files[i]], true) == true) {
// this pid is active so stop checking
pidActive = true;
break;
}
}
// check if any pm2 pids were active
if (pidActive == true) {
// compile css
compile_css();
console.log('\nReloading the explorer.. Please wait..\n');
// reload pm2 using the zero-downtime reload function
execSync(`pm2 reload explorer`, {stdio : 'inherit'});
// add a new line for better spacing
console.log('');
// finish the script
exit();
} else {
// check if the forever pid file exists and is valid
if (fs.existsSync('./tmp/forever.pid') && lib.is_locked(['forever'], true) == true) {
// this pid is active
pidActive = true;
}
// check if the forever.pid is active
if (pidActive == true) {
// compile css
compile_css();
console.log('\nReloading the explorer.. Please wait..\n');
// reload forever using the restart function
execSync(`forever restart explorer`, {stdio : 'inherit'});
// add a new line for better spacing
console.log('');
// finish the script
exit();
} else {
const request = require('postman-request');
const settings = require('../lib/settings');
// try executing the restart explorer api
request({uri: `http://localhost:${settings.webserver.port}/system/restartexplorer`, timeout: 1000}, function (error, response, summary) {
// check if there was an error
if (error != null)
console.log('Webserver is not runnning\n');
else {
// compile css
compile_css();
console.log('\nReloading the explorer.. Please wait..\n');
}
// finish the script
exit();
});
}
}
} else {
// finish the script
exit();
}