Allow backup and restore of a single collection

-Both the backup and restore scripts now support a new optional parameter that allows backing up and restoring a single collection only
-Added new verbiage and examples to the backup and restore script sections of the README
-Removed extra "the"'s from some of the restore database examples in the README

README.md

@@ -873,7 +873,13 @@ cd /path/to/explorer && /path/to/node ./scripts/update_explorer.js "dependencies
 
 #### Backup Database Script
 
-Make a complete backup of an eIquidus mongo database and save to compressed file. A built-in locking mechanism prevents data from being updated or changed while a backup is in process. Backups can be safely created while the explorer is actively running and/or while the explorer is turned off. The following backup scenarios are supported:
+Make a complete backup of an eIquidus mongo database or single collection and save to compressed file. A built-in locking mechanism prevents data from being updated or changed while a backup is in process. Backups can be safely created while the explorer is actively running and/or while the explorer is turned off.
+
+Parameters:
+1. Backup path or filename (optional)
+2. Collection name (optional) **NOTE:** This parameter is useful for backing up a single database collection such as the `claimaddresses` or plugin-related collections that can later be restored into an existing database without affecting any other database collections.
+
+The following backup scenarios are supported:
 
 **Backup Database (No filename specified)**
 
@@ -881,26 +887,38 @@ Make a complete backup of an eIquidus mongo database and save to compressed file
 
 **Backup Database (Partial filename specified)**
 
-`npm run create-backup test`: Backs up the the explorer/backups directory by default with the filename test.bak
+`npm run create-backup test`: Backs up the explorer/backups directory by default with the filename test.bak
 
 **Backup Database (Full filename specified)**
 
-`npm run create-backup today.bak`: Backs up the the explorer/backups directory by default with the filename today.bak
+`npm run create-backup today.bak`: Backs up the explorer/backups directory by default with the filename today.bak
 
 **Backup Database (Full path with partial filename specified)**
 
-`npm run create-backup /usr/local/bin/abc`: Backs up the the /usr/local/bin directory with the filename abc.bak
+`npm run create-backup /usr/local/bin/abc`: Backs up the /usr/local/bin directory with the filename abc.bak
 
 **Backup Database (Full path and filename specified)**
 
-`npm run create-backup ~/new.bak`: Backs up the the users home directory with the filename new.bak
+`npm run create-backup ~/new.bak`: Backs up the users home directory with the filename new.bak
+
+**Backup Database (Filename and collection both specified)**
+
+`npm run create-backup test claimaddresses`: Backs up only the `claimaddresses` collection to the explorer/backups directory by default with the filename test.bak
+
+**Backup Database (No filename specified, and backup a single collection only)**
+
+`npm run create-backup "" masternodes`: Backs up only the `masternodes` collection to the explorer/backups directory by default with the current date as the filename in the format yyyy-MMM-dd.bak 
 
 #### Restore Database Script
 
-Restore a previously saved eIquidus mongo database backup. :warning: **WARNING:** This will completely overwrite your existing eIquidus mongo database, so be sure to make a full backup before proceeding. A built-in locking mechanism prevents data from being updated or changed while a backup is being restored. Backups can be safely restored while the explorer is actively running and/or while the explorer is turned off.
+Restore a previously saved eIquidus mongo database backup. :warning: **WARNING:** Unless a single collection name is specified, this will completely overwrite your existing eIquidus mongo database, so be sure to make a full backup before proceeding. A built-in locking mechanism prevents data from being updated or changed while a backup is being restored. Backups can be safely restored while the explorer is actively running and/or while the explorer is turned off.
 
 **NOTE:** Older v1.x eIquidus database backups were compressed into tar.gz files. These older tar.gz backups can still be restored, but you must specifically add the .tar.gz suffix. Example: `npm run restore-backup /path/to/old_backup.tar.gz`
 
+Parameters:
+1. Backup path or filename (optional)
+2. Collection name (optional) **NOTE:** This parameter is useful for restoring a single database collection such as the `claimaddresses` or plugin-related collections without affecting any other database collections. This option can be used with a single collection backup or full database backup and will restore only the specified collection.
+
 The following restore scenarios are supported:
 
 **Restore Database (Partial filename specified)**
@@ -919,6 +937,10 @@ The following restore scenarios are supported:
 
 `npm run restore-backup ~/archive.bak`: Restores the ~/archive.bak file
 
+**Restore Database (Filename and collection both specified)**
+
+`npm run restore-backup test claimaddresses`: Restores only the `claimaddresses` collection from the explorer/scripts/backups/test.bak file
+
 #### Delete Database Script
 
 Wipe the eIquidus mongo database clean to start again from scratch. :warning: **WARNING:** This will completely destroy all data in your existing eIquidus mongo database, so be sure to make a full backup before proceeding. A built-in locking mechanism prevents data from being updated or changed while the database is being deleted. The process to delete the database can be executed while the explorer is actively running and/or while the explorer is turned off.

scripts/create_backup.js

@@ -6,6 +6,7 @@ const backupLockName = 'backup';
 const settings = require('../lib/settings');
 let backupPath = path.join(path.dirname(__dirname), 'backups');
 let backupFilename;
+let singleCollection = '';
 let lockCreated = false;
 
 // exit function used to cleanup lock before finishing script
@@ -20,6 +21,42 @@ function exit(exitCode) {
   }
 }
 
+// verify that the collection exists
+function verify_collection_exists(cb) {
+  // check if the backup will be for a single collection
+  if (singleCollection != null && singleCollection != '') {
+    const mongoose = require('mongoose');
+    const dbString = `mongodb://${encodeURIComponent(settings.dbsettings.user)}:${encodeURIComponent(settings.dbsettings.password)}@${settings.dbsettings.address}:${settings.dbsettings.port}/${settings.dbsettings.database}`;
+
+    console.log('Connecting to database..');
+
+    mongoose.set('strictQuery', true);
+
+    // connect to mongo database
+    mongoose.connect(dbString).then(() => {
+      // lookup the collection in the list of collections
+      mongoose.connection.db.listCollections({ name: singleCollection }).toArray().then((collections) => {
+        // check if the collection exists
+        if (collections.length > 0) {
+          // collection exists
+          return cb(false);
+        } else {
+          // collection not found
+          return cb(true);
+        }
+      }).catch((err) => {
+        console.log(err);
+        return cb(true);
+      });
+    }).catch((err) => {
+      console.log('Error: Unable to connect to database: %s', err);
+      exit(999);
+      return cb(true);
+    });
+  } else
+    return cb(false);
+}
+
 // check if a backup filename was passed into the script
 if (process.argv[2] != null && process.argv[2] != '') {
   // use the backup filename passed into this script
@@ -32,6 +69,12 @@ if (process.argv[2] != null && process.argv[2] != '') {
   backupFilename = `${systemDate.getFullYear()}-${monthName[systemDate.getMonth()]}-${systemDate.getDate()}`;
 }
 
+// check if a collection name was passed into the script
+if (process.argv[3] != null && process.argv[3] != '') {
+  // save the collection name to a variable for later use
+  singleCollection = process.argv[3];
+}
+
 // check if backup filename has the archive suffix already
 if (backupFilename.endsWith(archiveSuffix)) {
   // remove the archive suffix from the backup filename
@@ -58,18 +101,23 @@ if (!fs.existsSync(path.join(backupPath, `${backupFilename}${archiveSuffix}`)))
   if (lib.is_locked([backupLockName]) == false) {
     // create a new backup lock before checking the rest of the locks to minimize problems with running scripts at the same time
     lib.create_lock(backupLockName);
+
     // ensure the lock will be deleted on exit
     lockCreated = true;
+
     // check all other possible locks since backups should not run at the same time that data is being changed
     if (lib.is_locked(['restore', 'delete', 'index', 'markets', 'peers', 'masternodes', 'plugin']) == false) {
+      // check if the collection name exists
+      verify_collection_exists(function(collection_error) {
+        // check if there was an error finding the collection by name
+        if (!collection_error) {
           // all tests passed. OK to run backup
           console.log(`${settings.localization.script_launched }: ${process.pid}`);
 
           const { exec } = require('child_process');
-      const randomDirectoryName = Math.random().toString(36).substring(2, 15) + Math.random().toString(23).substring(2, 5);
 
           // execute backup
-      const backupProcess = exec(`mongodump --host="${settings.dbsettings.address}" --port="${settings.dbsettings.port}" --username="${settings.dbsettings.user}" --password="${settings.dbsettings.password}" --db="${settings.dbsettings.database}" --archive="${path.join(backupPath, backupFilename + archiveSuffix)}" --gzip`);
+          const backupProcess = exec(`mongodump --host="${settings.dbsettings.address}" --port="${settings.dbsettings.port}" --username="${settings.dbsettings.user}" --password="${settings.dbsettings.password}" --db="${settings.dbsettings.database}" --archive="${path.join(backupPath, backupFilename + archiveSuffix)}" --gzip${singleCollection == null || singleCollection == '' ? '' : ` --collection ${singleCollection}`}`);
 
           backupProcess.stdout.on('data', (data) => {
             console.log(data);
@@ -95,6 +143,12 @@ if (!fs.existsSync(path.join(backupPath, `${backupFilename}${archiveSuffix}`)))
               exit(0);
             }
           });
+        } else {
+          // the collection does not exist
+          console.log(`Collection "${singleCollection}" does not exist`);
+          exit(2);
+        }
+      });
     } else {
       // another script process is currently running
       console.log("Backup aborted");

scripts/restore_backup.js

@@ -7,7 +7,8 @@ const restoreLockName = 'restore';
 const tarModule = 'tar';
 const defaultBackupPath = path.join(path.dirname(__dirname), 'backups');
 const settings = require('../lib/settings');
-var lockCreated = false;
+let singleCollection = '';
+let lockCreated = false;
 
 // exit function used to cleanup lock before finishing script
 function exit(mongoose, exitCode) {
@@ -83,19 +84,13 @@ function drop_collection(mongoose, colName, cb) {
 }
 
 function delete_database(mongoose, cb) {
-  const dbString = `mongodb://${encodeURIComponent(settings.dbsettings.user)}:${encodeURIComponent(settings.dbsettings.password)}@${settings.dbsettings.address}:${settings.dbsettings.port}/${settings.dbsettings.database}`;
+  // check if a single collection is being restored
-
+  if (singleCollection != null && singleCollection == '') {
-  console.log('Connecting to database..');
-
-  mongoose.set('strictQuery', true);
-
-  // connect to mongo database
-  mongoose.connect(dbString).then(() => {
     // get the list of collections
     mongoose.connection.db.listCollections().toArray().then((collections) => {
       // check if there are any collections
       if (collections.length > 0) {
-        var counter = 0;
+        let counter = 0;
 
         // loop through all collections
         collections.forEach((collection) => {
@@ -122,10 +117,10 @@ function delete_database(mongoose, cb) {
       console.log(err);
       return cb(true);
     });
-  }).catch((err) => {
+  } else {
-    console.log('Error: Unable to connect to database: %s', err);
+    // do not delete any collections since mongorestore will drop the collection automatically
-    exit(mongoose, 999);
+    return cb(true);
-  });
+  }
 }
 
 function restore_backup(mongoose, backupPath, extractedPath, gZip) {
@@ -134,7 +129,7 @@ function restore_backup(mongoose, backupPath, extractedPath, gZip) {
   console.log(`${settings.localization.restoring_backup}.. ${settings.localization.please_wait}..`);
 
   // restore mongo database from backup
-  const restoreProcess = exec(`mongorestore --host="${settings.dbsettings.address}" --port="${settings.dbsettings.port}" --username="${settings.dbsettings.user}" --password="${settings.dbsettings.password}" --authenticationDatabase="${settings.dbsettings.database}" ${(gZip ? `--gzip --archive="${backupPath}"` : `"${extractedPath}"`)}`);
+  const restoreProcess = exec(`mongorestore --host="${settings.dbsettings.address}" --port="${settings.dbsettings.port}" --username="${settings.dbsettings.user}" --password="${settings.dbsettings.password}" --authenticationDatabase="${settings.dbsettings.database}" ${(gZip ? `--gzip --archive="${backupPath}"` : `"${extractedPath}"`)}${singleCollection == null || singleCollection == '' ? '' : ` --drop --db explorerdb --collection ${singleCollection}`}`);
 
   restoreProcess.stdout.on('data', (data) => {
     console.log(data);
@@ -173,9 +168,73 @@ function restore_backup(mongoose, backupPath, extractedPath, gZip) {
   });
 }
 
+function delete_prompt(cb) {
+  const readline = require('readline');
+  const rl = readline.createInterface({
+    input: process.stdin,
+    output: process.stdout
+  });
+
+  // check if a single collection is being restored
+  if (singleCollection != null && singleCollection != '')
+    console.log(`You are about to delete and restore the "${singleCollection}" collection from backup.`);
+  else
+    console.log('You are about to delete the current eIquidus database and restore from backup.');
+
+  // prompt for restoring explorer database
+  rl.question(`${settings.localization.are_you_sure}: `, function (restoreAnswer) {
+    // stop prompting
+    rl.close();
+
+    // return the answer
+    return cb(restoreAnswer);
+  });
+}
+
+// verify that the collection exists
+function verify_collection_exists(mongoose, cb) {
+  const dbString = `mongodb://${encodeURIComponent(settings.dbsettings.user)}:${encodeURIComponent(settings.dbsettings.password)}@${settings.dbsettings.address}:${settings.dbsettings.port}/${settings.dbsettings.database}`;
+
+  console.log('Connecting to database..');
+
+  mongoose.set('strictQuery', true);
+
+  // connect to mongo database
+  mongoose.connect(dbString).then(() => {
+    // check if the restore will be for a single collection
+    if (singleCollection != null && singleCollection != '') {
+      // lookup the collection in the list of collections
+      mongoose.connection.db.listCollections({ name: singleCollection }).toArray().then((collections) => {
+        // check if the collection exists
+        if (collections.length > 0) {
+          // collection exists
+          return cb(false);
+        } else {
+          // collection not found
+          return cb(true);
+        }
+      }).catch((err) => {
+        console.log(err);
+        return cb(true);
+      });
+    } else
+      return cb(false);
+  }).catch((err) => {
+    console.log('Error: Unable to connect to database: %s', err);
+    exit(999);
+    return cb(true);
+  });
+}
+
 // check if a backup filename was passed into the script
 if (process.argv[2] != null && process.argv[2] != '') {
-  var backupPath = process.argv[2];
+  let backupPath = process.argv[2];
+
+  // check if a collection name was passed into the script
+  if (process.argv[3] != null && process.argv[3] != '') {
+    // save the collection name to a variable for later use
+    singleCollection = process.argv[3];
+  }
 
   // check if the backup filename already has a path
   if (!fs.existsSync(`${backupPath}`)) {
@@ -197,19 +256,14 @@ if (process.argv[2] != null && process.argv[2] != '') {
 
   // check for the backup file (last time)
   if (fs.existsSync(`${backupPath}`)) {
-    const readline = require('readline');
+    const mongoose = require('mongoose');
-    const rl = readline.createInterface({
-      input: process.stdin,
-      output: process.stdout
-    });
-
-    console.log('You are about to delete the current eIquidus database and restore from backup.');
-
-    // prompt for restoring explorer database
-    rl.question(`${settings.localization.are_you_sure}: `, function (restoreAnswer) {
-      // stop prompting
-      rl.close();
 
+    // check if the collection name exists
+    verify_collection_exists(mongoose, function(collection_error) {
+      // check if there was an error finding the collection by name
+      if (!collection_error) {
+        // prompt for deleting and restoring the database
+        delete_prompt(function(restoreAnswer) {
           // determine if the explorer database should be restored
           switch ((restoreAnswer == null ? '' : restoreAnswer).toLowerCase()) {
             case settings.localization.short_yes:
@@ -218,8 +272,10 @@ if (process.argv[2] != null && process.argv[2] != '') {
               if (lib.is_locked([restoreLockName]) == false) {
                 // create a new restore lock before checking the rest of the locks to minimize problems with running scripts at the same time
                 lib.create_lock(restoreLockName);
+
                 // ensure the lock will be deleted on exit
                 lockCreated = true;
+
                 // check all other possible locks since restoring backups should not run at the same time that data is being changed
                 if (lib.is_locked(['backup', 'delete', 'index', 'markets', 'peers', 'masternodes', 'plugin']) == false) {
                   // all tests passed. OK to run restore
@@ -251,8 +307,6 @@ if (process.argv[2] != null && process.argv[2] != '') {
 
                         // check if this is a valid backup archive now that the files have been extracted
                         if (fs.existsSync(`${path.join(extractedPath, settings.dbsettings.database)}`)) {
-                      const mongoose = require('mongoose');
-
                           // delete all collections from existing database
                           delete_database(mongoose, function(retVal) {
                             if (retVal) {
@@ -269,7 +323,7 @@ if (process.argv[2] != null && process.argv[2] != '') {
                             // do nothing
                           } finally {
                             console.log(`${path.basename(backupPath)} is not a valid backup file`);
-                        exit(null, 1);
+                            exit(mongoose, 1);
                           }
                         }
                       });
@@ -278,18 +332,24 @@ if (process.argv[2] != null && process.argv[2] != '') {
                 } else {
                   // another script process is currently running
                   console.log("Restore aborted");
-              exit(null, 2);
+                  exit(mongoose, 2);
                 }
               } else {
                 // restore process is already running
                 console.log("Restore aborted");
-            exit(null, 2);
+                exit(mongoose, 2);
               }
 
               break;
             default:
               console.log(`${settings.localization.process_aborted}. ${settings.localization.nothing_was_restored}.`);
-          exit(null, 2);
+              exit(mongoose, 2);
+          }
+        });
+      } else {
+        // the collection does not exist
+        console.log(`Collection "${singleCollection}" does not exist`);
+        exit(mongoose, 2);
       }
     });
   } else {