hex7c0 / mongodb-backup Goto Github PK

View Code? Open in Web Editor NEW

134.0 8.0 49.0 271 KB

backup data for mongodb for Nodejs

Home Page: https://github.com/hex7c0/mongodb-backup

License: Apache License 2.0

JavaScript 100.00%

mongodb backup nodejs

mongodb-backup's Introduction

mongodb-backup

Backup for mongodb

Look at mongodb-backup-cli for command line usage, similar to mongodump

Look at mongodb-restore for restore data

Installation

Install through NPM

npm install mongodb-backup

git clone git://github.com/hex7c0/mongodb-backup.git

[email protected] has been pulled out, so versions >= 1.3.0 and <= 1.4.1 are deprecated

API

inside nodejs project

var backup = require('mongodb-backup');

backup({
  uri: 'uri', // mongodb://<dbuser>:<dbpassword>@<dbdomain>.mongolab.com:<dbport>/<dbdatabase>
  root: __dirname
});

backup(options)

options

uri - String URI for MongoDb connection (default "required")
root- String Path where save data (default "required")
[parser] - String | Function Data parser (bson, json) or custom (default "bson")
[collections] - Array Select which collections save (default "disabled")
[callback] - Function Callback when done (default "disabled")
[stream]- Object Send .tar file to Node stream (default "disabled")
[tar] - String Pack files into a .tar file (default "disabled")
[query] - Object Query that optionally limits the documents included (default "{}")
[numCursors] - Number Set number of cursor for parallelCollectionScan without query (default "disabled")
[logger] - String Path where save a .log file (default "disabled")
[metadata] - Boolean Save metadata of collections as Index, ecc (default "false")
[options] - Object MongoDb options (default)

Examples

Take a look at my examples

mongodb-backup's People

Contributors

Stargazers

Watchers

mongodb-backup's Issues

Makes use of the filesystem

Even when using streams, it uses the filesystem which makes it unsuitable for cloud platforms like Heroku and AWS Lambda.

Absolute path for tar archive

What do you think to handle also an absolute path?

Error [MongoError]: Unsupported OP_QUERY command: listCollections.

Error [MongoError]: Unsupported OP_QUERY command: listCollections. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal
at Function.MongoError.create (C:\Users\user\Desktop\poetry\node_modules\mongodb-core\lib\error.js:31:11)
at queryCallback (C:\Users\user\Desktop\poetry\node_modules\mongodb-core\lib\cursor.js:212:36)
at C:\Users\user\Desktop\poetry\node_modules\mongodb-core\lib\connection\pool.js:455:18
at processTicksAndRejections (node:internal/process/task_queues:78:11) {
ok: 0,
errmsg: 'Unsupported OP_QUERY command: listCollections. The client driver may require an upgrade. For more details see https://dochub.mongodb.org/core/legacy-opcode-removal',
code: 352,
codeName: 'UnsupportedOpQueryCommand'
}

"mongodb-backup": "^1.6.9"

Dump is not creating when 'tar' key is exist

This code works, creating directory in storageDir with collection dumps
backup({
uri: mongodb.location,
root: storageDir,
});
But
This code does not works:
backup({
uri: mongodb.location,
root: storageDir,
tar: 'mongodumpname.tar'
});
Nothing happening

P.S
"mongodb-backup": "^1.6.9",

it will create seperate json file for every document is there any way to create only one json file having all document of collection

How to backup all database of mongodb uri without specify ?

How to backup all database of mongodb uri without specify ?
I need to back up all without define them by array because . I may forget to add the db name to when have a new database . So the best way is back up all db . How can i do it ?

Link to examples is broken

Hi,

First I want to thank you for your great work sofar on mongodb-backup.
I came across a small issue, as the title says: The link to examples is broken. Could this be fixed?

Thanx,
Stephan

update package with mongo driver 3

mongo driver 2 is old and there is some problems on query with _id...
please update packege with mongo driver 3...
it need some changes such as snapshot stream or mongo client connection.

Logger callback function

Is it possible to modify the input logger option? To make it either a string filename like before or let it be a callback function. This bypasses the logger object and lets the user handle the logging itself.
Similar in mongodb-restore.

Is there possible to rename backuped collection name ?

Hi,

its working good, but i need to rename collection name once back completed, becase i have restoring same data into different table based on using cron job.

let fmdate = new Date("11 May 2019 00:00:01 UTC");
let todate = new Date("11 May 2019 23:59:59 UTC");
backup({
  uri: "mongodb://demoroot:demopass@localhost:27017/dbDemo",
  root: `${__dirname}/bkp/`,
  parser: "bson", //use only bson to get same mongo format
  collections: ["tblBook"],
  callback: (err, data)=>{
    console.log(err);
    console.log(this)
    console.log(data);
  },
  query: {
    publisedDate: {
      $gte: fmdate,
      $lt: todate
    }
  }
});

Gopal R.

How to use "query" option for multiple collections?

I would like to use query option for multiple collections

backup({
  uri: 'uri', // mongodb://<dbuser>:<dbpassword>@<dbdomain>.mongolab.com:<dbport>/<dbdatabase>
  root: __dirname
  collections: ['collectionA', 'collectionB']
  query: {{...A...}, {...B...}} // break for collectionA, and B 
});

Please example for me.

Generates backup file with empty collections

Generates empty folders

I use this library inside my nest.js service.
Collection folders get created but they are all empty.
This is my code:

`var backup = require('mongodb-backup');
import { Injectable, Logger } from "@nestjs/common";

@Injectable()
export class BackupDbService {
private readonly logger = new Logger(BackupDbService.name);
mongoUrl = 'mongodb://localhost:27017/admin';

backupDb() {
    try {
        backup({
            uri: this.mongoUrl, // mongodb://<dbuser>:<dbpassword>@<dbdomain>.mongolab.com:<dbport>/<dbdatabase>
            collections: ['rates'],
            root: 'db',
            callback: function(err) {
                if (err) {
                  console.error(err);
                } else {
                  console.log('finish');
                }
              }
          });
    } catch (err) {
        this.logger.error(err);
    }
}

} `

Write backup to any stream

It would be great to be able to dump the backup to any nodejs stream, not only files.

In some cloud environments you don't have access to a filesystem but you could easily write it as a response to a HTTP request.

I haven't evaluated how hard it would be to implement this.

License (GPL)

Hi there, thanks for writing the awesome library. Would you consider changing the license to MIT for broader commercial uses? 😄

Version pulled out [email protected]

Hey @hex7c0,

[email protected] has been pulled out, could you remove the hard dep in your packages.json and push a new release to npm ? (cf mongodb/js-bson#147)

    "bson": "0.4.11",

would become

    "bson": "~0.4",

Thank you :)

Problem on Meteor JS?

I use it on Meteor JS by hitchcott/app-dump.
But I have problem with backup:

20151002-11:53:00.787(7)? (STDERR)                                                                                                              
20151002-11:53:00.790(7)? (STDERR) C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\node_modules\mongodb-backup\node_
ules\mongodb\lib\utils.js:97                                                                                                                    
20151002-11:53:00.790(7)? (STDERR)     process.nextTick(function() { throw err; });                                                             
20151002-11:53:00.790(7)? (STDERR)                                         ^                                                                    
20151002-11:53:00.791(7)? (STDERR) Error: UNKNOWN, unknown error 'C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\no
modules\mongodb-backup\dump\meteor\testing\gxCnw5uYSHNxW3hpx.bson'                                                                              
20151002-11:53:00.791(7)? (STDERR)     at Object.fs.openSync (fs.js:439:18)                                                                     
20151002-11:53:00.791(7)? (STDERR)     at Object.fs.writeFileSync (fs.js:978:15)                                                                
20151002-11:53:00.791(7)? (STDERR)     at C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\node_modules\mongodb-backu
ndex.min.js:56:19                                                                                                                               
20151002-11:53:00.791(7)? (STDERR)     at Array.forEach (native)                                                                                
20151002-11:53:00.792(7)? (STDERR)     at toBson (C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\node_modules\mongo
backup\index.min.js:55:37)                                                                                                                      
20151002-11:53:00.792(7)? (STDERR)     at C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\node_modules\mongodb-backu
ndex.min.js:71:82                                                                                                                               
20151002-11:53:00.792(7)? (STDERR)     at handleCallback (C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\node_modul
mongodb-backup\node_modules\mongodb\lib\utils.js:95:12)                                                                                         
20151002-11:53:00.793(7)? (STDERR)     at C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\node_modules\mongodb-backu
ode_modules\mongodb\lib\cursor.js:772:16                                                                                                        
20151002-11:53:00.793(7)? (STDERR)     at handleCallback (C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\node_modul
mongodb-backup\node_modules\mongodb\node_modules\mongodb-core\lib\cursor.js:244:5)                                                              
20151002-11:53:00.793(7)? (STDERR)     at isCursorDeadButNotkilled (C:\Users\Theara\AppData\Local\.meteor\packages\hitchcott_app-dump\0.4.1\npm\
e_modules\mongodb-backup\node_modules\mongodb\node_modules\mongodb-core\lib\cursor.js:506:5)                                                    
> Meteor server restarted                                                                                                                       

ttp.js:732                                                                                                                                      
   throw new Error('Can\'t render headers after they are sent to the client.'                                                                   
         ^                                                                                                                                      
rror: Can't render headers after they are sent to the client.                                                                                   
 at ServerResponse.OutgoingMessage._renderHeaders (http.js:732:11)                                                                              
 at ServerResponse.writeHead (http.js:1153:20)                                                                                                  
 at ProxyServer.<anonymous> (C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.4\mt-os.windows.x86_32\tools\run-proxy.js:96:21)    
 at ProxyServer.emit (C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.4\mt-os.windows.x86_32\dev_bundle\lib\node_modules\eventemi
r3\index.js:100:27)                                                                                                                             
 at ClientRequest.proxyError (C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.4\mt-os.windows.x86_32\dev_bundle\lib\node_modules\
p-proxy\lib\http-proxy\passes\web-incoming.js:140:16)                                                                                           
 at ClientRequest.emit (events.js:117:20)                                                                                                       
 at Socket.socketErrorListener (http.js:1552:9)                                                                                                 
 at Socket.emit (events.js:95:17)                                                                                                               
 at net.js:441:14                                                                                                                               
 at process._tickCallback (node.js:442:13)

I think that it have problem with size of DB.
I try with size ~ 100 K it is Ok, but size ~ 1M it don't work.

Error while backup large collection

Hello!

It seems this issue still not resolved: #16

Node: v7.2.1
mongodb-backup: v1.6.8

{ Error: Cannot find module 'tar'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at /Users/bushev/IdeaProjects/proj/node_modules/mongodb-backup/index.min.js:195:42
    at /Users/bushev/IdeaProjects/proj/node_modules/mongodb-backup/index.min.js:22:20
    at /Users/bushev/IdeaProjects/proj/node_modules/mongodb-backup/node_modules/graceful-fs/polyfills.js:287:18
    at FSReqWrap.oncomplete (fs.js:111:15) code: 'MODULE_NOT_FOUND' }
fs.js:557
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: EMFILE: too many open files, open '/Users/bushev/IdeaProjects/proj/logs/error.log'
    at Object.fs.openSync (fs.js:557:18)
    at Object.fs.writeFileSync (fs.js:1222:33)
    at Object.fs.appendFileSync (fs.js:1268:6)
    at process.on.err (/Users/bushev/IdeaProjects/proj/queue-server.js:50:19)
    at emitOne (events.js:96:13)
    at process.emit (events.js:188:7)
    at process._fatalException (bootstrap_node.js:292:26)

Also, Q: Why we need to create a separate file for each collection item?

EMFILE: Error with large number of db

Hello,
Thanks for the great work on nodejs packages mongodb-backup and restore. I'm currently using these two packages in my project.

I have some problems with large number of db. I try backup 100+ db in async.series. But I have error:
fs.js:60 throw err; // Forgot a callback but don't know where? Use NODE_DEBUG=fs ^ Error: EMFILE: too many open files, open '/var/www/test/node_modules/mongodb-backup/dump/db_1/pages/577bc802fed786f67fdc43b3.bson' at Error (native)

The size of each base about 100 mb
Could you please have a look at it ?

Thanks in advance.

Mongodb-backup stop mongod service

Hi,
My database host on AWS server.
When i start to backup mongodb, my mongod service is stop automatic.
I think this issue because of large database size(my db size is more than 1 GB).
What should i do to prevent to stop mongod service.

Can't install on Meteor package?

I try to create Meteor package

Package.describe({
  name: 'theara:app-dump',
  version: '0.0.2',
  // Brief, one-line summary of the package.
  summary: 'Dump',
  // URL to the Git repository containing the source code for this package.
  git: '',
  // By default, Meteor will default to using README.md for documentation.
  // To avoid submitting documentation, set this field to null.
  documentation: 'README.md'
});

Package.onUse(function(api) {
  api.versionsFrom('[email protected]');
  api.use('ecmascript');
  api.addFiles('app-dump.js');
});

Package.onTest(function(api) {
  api.use('ecmascript');
  api.use('tinytest');
  api.use('theara:app-dump');
  api.addFiles('app-dump-tests.js');
});

Npm.depends({
  "mongodb-backup": "1.4.1"
});

And then install it meteor add theara:app-dump, but show:

$ meteor add theara:app-dump                                                                                                                         
theara:app-dump: updating npm dependencies -- mongodb-backup...                                                                               
 => Errors while adding packages:                                                                                                                    

While building package theara:app-dump:                                                                                                              
error: couldn't install npm package [email protected]: Command failed: npm http GET https://registry.npmjs.org/mongodb-backup                     
npm http 304 https://registry.npmjs.org/mongodb-backup                                                                                               
npm http GET https://registry.npmjs.org/bson                                                                                                         
npm http GET https://registry.npmjs.org/fstream                                                                                                      
npm http GET https://registry.npmjs.org/mongodb                                                                                                      
npm http GET https://registry.npmjs.org/tar                                                                                                          
npm http GET https://registry.npmjs.org/logger-request                                                                                               
npm http 304 https://registry.npmjs.org/mongodb                                                                                                      
npm http 304 https://registry.npmjs.org/fstream                                                                                                      
npm http 304 https://registry.npmjs.org/bson                                                                                                         
npm http 304 https://registry.npmjs.org/logger-request                                                                                               
npm http 304 https://registry.npmjs.org/tar                                                                                                          
npm ERR! Error: version not found: [email protected]                                                                                                       
npm ERR!     at                                                                                                                                      
C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.9\mt-os.windows.x86_32\dev_bundle\bin\node_modules\npm\lib\cache.js:846:12            
npm ERR!     at saved                                                                                                                                
(C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.9\mt-os.windows.x86_32\dev_bundle\bin\node_modules\npm\node_modules\npm-registry-clie
nt\lib\get.js:148:7)                                                                                                                                 
npm ERR!     at                                                                                                                                      
C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.9\mt-os.windows.x86_32\dev_bundle\bin\node_modules\npm\node_modules\graceful-fs\polyfi
lls.js:133:7                                                                                                                                         
npm ERR!     at Object.oncomplete (fs.js:108:15)                                                                                                     
npm ERR! If you need help, you may report this *entire* log,                                                                                         
npm ERR! including the npm and node versions, at:                                                                                                    
npm ERR!     <http://github.com/npm/npm/issues>                                                                                                      

npm ERR! System Windows_NT 6.1.7600                                                                                                                  
npm ERR! command "C:\\Users\\Theara\\AppData\\Local\\.meteor\\packages\\meteor-tool\\1.1.9\\mt-os.windows.x86_32\\dev_bundle\\bin\\\\node.exe"       
"C:\\Users\\Theara\\AppData\\Local\\.meteor\\packages\\meteor-tool\\1.1.9\\mt-os.windows.x86_32\\dev_bundle\\bin\\node_modules\\npm\\bin\\npm-cli.js"

"install" "[email protected]"                                                                                                                     
npm ERR! cwd D:\meteor-app\dump_1.2\packages\app-dump\.npm\package-new-ycs0cr                                                                        
npm ERR! node -v v0.10.40                                                                                                                            
npm ERR! npm -v 1.4.9                                                                                                                                
npmnpm http GET https://registry.npmjs.org/mongodb-backup                                                                                            
npm http 304 https://registry.npmjs.org/mongodb-backup                                                                                               
npm http GET https://registry.npmjs.org/bson                                                                                                         
npm http GET https://registry.npmjs.org/fstream                                                                                                      
npm http GET https://registry.npmjs.org/mongodb                                                                                                      
npm http GET https://registry.npmjs.org/tar                                                                                                          
npm http GET https://registry.npmjs.org/logger-request                                                                                               
npm http 304 https://registry.npmjs.org/mongodb                                                                                                      
npm http 304 https://registry.npmjs.org/fstream                                                                                                      
npm http 304 https://registry.npmjs.org/bson                                                                                                         
npm http 304 https://registry.npmjs.org/logger-request                                                                                               
npm http 304 https://registry.npmjs.org/tar                                                                                                          
npm ERR! Error: version not found: [email protected]                                                                                                       
npm ERR!     at                                                                                                                                      
C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.9\mt-os.windows.x86_32\dev_bundle\bin\node_modules\npm\lib\cache.js:846:12            
npm ERR!     at saved                                                                                                                                
(C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.9\mt-os.windows.x86_32\dev_bundle\bin\node_modules\npm\node_modules\npm-registry-clie
nt\lib\get.js:148:7)                                                                                                                                 
npm ERR!     at                                                                                                                                      
C:\Users\Theara\AppData\Local\.meteor\packages\meteor-tool\1.1.9\mt-os.windows.x86_32\dev_bundle\bin\node_modules\npm\node_modules\graceful-fs\polyfi
lls.js:133:7                                                                                                                                         
npm ERR!     at Object.oncomplete (fs.js:108:15)                                                                                                     
npm ERR! If you need help, you may report this *entire* log,                                                                                         
npm ERR! including the npm and node versions, at:                                                                                                    
npm ERR!     <http://github.com/npm/npm/issues>                                                                                                      

npm ERR! System Windows_NT 6.1.7600                                                                                                                  
npm ERR! command "C:\\Users\\Theara\\AppData\\Local\\.meteor\\packages\\meteor-tool\\1.1.9\\mt-os.windows.x86_32\\dev_bundle\\bin\\\\node.exe"       
"C:\\Users\\Theara\\AppData\\Local\\.meteor\\packages\\meteor-tool\\1.1.9\\mt-os.windows.x86_32\\dev_bundle\\bin\\node_modules\\npm\\bin\\npm-cli.js"

"install" "[email protected]"                                                                                                                     
npm ERR! cwd D:\meteor-app\dump_1.2\packages\app-dump\.npm\package-new-ycs0cr                                                                        
npm ERR! node -v v0.10.40                                                                                                                            
npm ERR! npm -v 1.4.9                                                                                                                                
npm

tar property does not work

When I try to specify tar property in oprions object it's not working. I tried to install the different version of a package, but it repeats.
only I removed 'tar' property from options object it works.

Example:
{
uri: config.mongodb.location,
root: config.storageDir,
tar: 'myDump.tar'
}

Also I tried write tar string without '.tar' at the end but no any changes ;(

Cannot read property 'message' of undefined

So when i'm running a backup and specifying collections. It works when the array contains one collection, but when multiple are specified it chokes. I do however get a full tarball downloaded so it would appear the backup finished and sent the stream.

mongodb-backup is running in meteor and it ends up crashing the server and throwing this error. The bOpts are the options that are sent to backup ... you can see collections is set to the two collections i want to backup.

Thanks,

I20150415-00:00:17.398(-4)? bOpts.collections: object [ 'exercises', 'exerciseMedia' ]
W20150415-00:00:18.158(-4)? (STDERR) TypeError: Cannot read property 'message' of undefined
W20150415-00:00:18.158(-4)? (STDERR) at error (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/index.min.js:4:22)
W20150415-00:00:18.158(-4)? (STDERR) at /app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/index.min.js:86:82
W20150415-00:00:18.158(-4)? (STDERR) at /app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/index.min.js:55:32
W20150415-00:00:18.158(-4)? (STDERR) at Array.forEach (native)
W20150415-00:00:18.158(-4)? (STDERR) at toBson (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/index.min.js:52:42)
W20150415-00:00:18.159(-4)? (STDERR) at /app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/index.min.js:85:96
W20150415-00:00:18.159(-4)? (STDERR) at handleCallback (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/lib/utils.js:93:12)
W20150415-00:00:18.159(-4)? (STDERR) at /app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/lib/cursor.js:571:16
W20150415-00:00:18.159(-4)? (STDERR) at handleCallback (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:234:5)
W20150415-00:00:18.159(-4)? (STDERR) at setCursorDeadAndNotified (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:424:3)
W20150415-00:00:18.159(-4)? (STDERR) at [object Object].Cursor.next (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:585:7)
W20150415-00:00:18.159(-4)? (STDERR) at fetchDocs (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/lib/cursor.js:567:10)
W20150415-00:00:18.160(-4)? (STDERR) at /app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/lib/cursor.js:583:7
W20150415-00:00:18.160(-4)? (STDERR) at handleCallback (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:234:5)
W20150415-00:00:18.160(-4)? (STDERR) at [object Object].Cursor.next (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:603:5)
W20150415-00:00:18.160(-4)? (STDERR) at /app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:540:12
W20150415-00:00:18.160(-4)? (STDERR) at queryCallback (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/cursor.js:215:5)
W20150415-00:00:18.160(-4)? (STDERR) at [object Object].Callbacks.emit (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:84:3)
W20150415-00:00:18.160(-4)? (STDERR) at [object Object].messageHandler (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:209:23)
W20150415-00:00:18.161(-4)? (STDERR) at Socket. (/app/.meteor/local/isopacks/hitchcott_app-dump/npm/node_modules/mongodb-backup/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:256:22)
W20150415-00:00:18.161(-4)? (STDERR) at Socket.emit (events.js:95:17)
W20150415-00:00:18.161(-4)? (STDERR) at Socket. (stream_readable.js:765:14)
W20150415-00:00:18.161(-4)? (STDERR) at Socket.emit (events.js:92:17)
W20150415-00:00:18.161(-4)? (STDERR) at emitReadable (_stream_readable.js:427:10)
W20150415-00:00:18.161(-4)? (STDERR) at emitReadable (_stream_readable.js:423:5)
W20150415-00:00:18.161(-4)? (STDERR) at readableAddChunk (_stream_readable.js:166:9)
W20150415-00:00:18.162(-4)? (STDERR) at Socket.Readable.push (_stream_readable.js:128:10)
W20150415-00:00:18.162(-4)? (STDERR) at TCP.onread (net.js:529:21)

Suggestions?

High Severity Vulnerabilities

Please update dependencies. specially tar and fstream

tar >=2.2.2
fstream >=1.0.12

Delays server side functions on meteorJS

Hello,

first of all thanks for the great work on nodejs packages mongodb-backup and restore. I'm currently using these two packages in meteor js. And it suits my needs perfectly.

But, I've an issue that mongodb-dump delays my other serverside functions. I mean whenever I call mongodb-backup, and then call another my server-side function, my function works a few seconds later.

Could you please have a look at it ?

Thanks in advance.

Restoring the backup

I have tried and search but without luck. How do you go about restoring the backup this module creates?

An example would be appreciated. Thanks

how to tar.gz?

Empty collection folder

Hi there,
I faced this issue when export collections: in MondoDB I have not empty collections, but after export I have folders for all collections, but there are no files in these folders. What could be the reason of this issue?
Thanks

Installation issue

I am trying to install mongodb_backup. I have Windows7 machine.

I ram command - npm install --save mongodb_backup. But I am getting an error - Registry returned 404 for GET on http://registry.npmjs.org/mongodb_backup error and mongodb_backup' is not in the npm registry

Details on my environment are -

node v0.10.38
npm v2.11.1

Does it work on Windows?

It works perfectly on mac and linux but when I tried to run it on Windows, an error occurred Error: ENOENT: no such file or directory, open 'C:\Users\eldar\WebstormProjects\romanenkova\backups\2019-01-13T10:57:20.626Z\rom_prod\schedules\5c2cfcc23fb8387050a14958.bson'

Do I need to set something up or there is no way?

Please bump version to 1.4.2!

Unable to take back for this kind of URI as it from mongodb atlas, please help

mongodb+srv://username:[email protected]/dbName?retryWrites=true&w=majority

Getting the following error:

{
"message": "invalid schema, expected mongodb",
"error": {},
"title": "ERROR"
}

I tried with mongodb lab url string and it worked but with this not working

Does tar option replace the root dir?

This is about a documentation inconvenience for a first-time user. The README and the examples lack to answer how tar and root options are related. The reader (me) can form at least three different explanations.

after backup, both root dir and tar file exists. Tar has the content of root dir.
after backup, only tar file exists, root option does not affect anything.
root dir is created and then tar file is created and root dir is removed.

Would this inconvenience be worth of an additional sentence into README or tar example? I understand the user can test the correct explanation with relatively small amount of work or will eventually find it out. However for a new-comer, I think it is a bit confusing.

I am happy to make a PR about this.

Int64 _id gets exported as a TimeStamp

When exporting/backup-ing a document with ID like this:

"_id" : "1000576093407275579" (Int64)

they end up being transformed to this:

"_id" : Timestamp(1833010877, -705238994)

This is with the standard BSON parser.
As a workaround I tried using the JSON parser, but that screws up all datatypes.
This does not happen with the standard mongodump/mongorestore tools, but they're also using BSON as intermediate format.

bsondump --type=debug says, the _id field is of type 1, which is Double, from the mongodump BSON file,
but it's of type 17 (=TimeStamp) in the BSON file from mongodb-backup.

Support for `mongodump` like output

I was wondering whether it would be possible to transform the output to mongodump-like output.

The benefit of this is that you can use mongorestore on the CLI if you wanted to restore to a database created using mongodb-backup, and similarly, it's be cool if mongodb-restore had the ability to read a normal mongodump folder.

This is what mongodump spits out

collectionA.bson
collectionA.meta.json //this file is not required but contains index data
collectionB.bson
collectionB.meta.json
...

And this is what mongodb-backup currently creates

collectionA/
  _idA.bson
  _idB.bson
collectionB/
  _idC.bson
  _idD.bson
...

Would it possible to add directional compatibility for mongodump and mongorestore by implementing this transformation?

Backup creates empty files for big collection, reporting the completion prematurely.

Hi,
For small collections everything works fine. For collections with few hundred thousands documents, backup creates all the files, but most of them are empty when it reports that operation is complete (at least on Windows). Node app does not exit, so it looks like it flushes the internal buffers. If I wait until application exits, the files are Ok.
So, the problem is that it reports that everything is done when lots of files are still writing to disk...

Alex

Integration with GridFS collections?

When running mongobackup on GridFS collections, the .bson files are generated, but appear to be empty. When running mongorestore, deserializer.js throws the "corrupt bson message" error.

update - this is only when specifying GridFS collections only. When running mongobackup on the entire db, the GridFS collections have data