balmasi / migrate-mongoose Goto Github PK

There's nothing about how to setup migrate.json file in documentation

Hello!

I create a free mongodb into https://cloud.mongodb.com/ and I get an URL starting with mongodb+srv.
When I try to instantiate a MigrateMongoose, an error is thrown.

Code:

const MigrateMongoose = require("migrate-mongoose");

let migrator = new MigrateMongoose({
  dbConnectionUri: "mongodb+srv://user:[email protected]/DBName?retryWrites=true"
});

Error:

..\node_modules\migrate-mongoose\node_modules\mongoose\lib\drivers\node-mongodb-native\connection.js:49
  var server = new Server(this.host, this.port, this.options.server);
                                                             ^
TypeError: Cannot read property 'server' of null
    at NativeConnection.doOpen (..\node_modules\migrate-mongoose\node_modules\mongoose\lib\drivers\node-mongodb-native\connection.js:49:62)
    at NativeConnection.Connection._open (..\node_modules\migrate-mongoose\node_modules\mongoose\lib\connection.js:694:15)
    at NativeConnection.Connection._openWithoutPromise (..\node_modules\migrate-mongoose\node_modules\mongoose\lib\connection.js:359:8)
    at Mongoose.createConnection (..\node_modules\migrate-mongoose\node_modules\mongoose\lib\index.js:208:32)
    at new Migrator (..\node_modules\migrate-mongoose\dist\lib.js:97:56)
    ...

I've tried passing a connection by options, but another error is thrown. I think that just updating the mongoose to the latest version, already solve the problem.

Thanks!

Not a major concern but am getting this deprecation warning when I run migrations.

(node:10591) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.

Running on the following:

"mongodb": "^3.3.2", "mongoose": "^5.7.1"

Is there any way to get rid of it?

Problem statement

Running database migrations can frequently pose a problem in the form of breaking changes that happen when the state of the database does not match the state of the application code. This could happen when the migration code itself throws an error half-way through its execution and could result in the unknown state of the migration (i.e. partially migrated schema / partially seeded database).

Motivation

Running migrations as part of one's CI/CD pipeline usually means that the job has to be executed on par with the deployment stage (i.e. all-or-none). In this case, running a failing migration after the application is deployed can pose severe risk of bringing down the entire environment due to inconsistencies between the code and the state of the database. On the other hand, running a migration before the deployment can lead to the opposite problem - deployment script failure can break an existing environment, but the matching application code that has failed to deploy will also leave the environment in an out-of-sync state.

Solution

To decrease the likelihood of the above situations, I propose introducing a code that 'dry-runs' the migrations, giving the user ability to ensure that at the very least - the migration code will execute without syntax errors, even if external calls fail. This could be done in several ways.

Firstly the flag --dry, or similar one would have to be added to trigger the dry-run upon migration call. Secondly, the migration code itself should be checked for possible syntax errors etc. that would occur during runtime.

Alternatively, the migration code could explicitly specify the case of jobs that should run in case of dry-runs (as an example, external API calls that introduce the changes beyond the database itself). It would also be nice to perform full dry-runs in the way that does not alter the original database, but rather creates an exact copy of it and performs the migrations there.

When I run $ migrate I get the following error:

module.js:338
    throw err;
          ^
Error: Cannot find module 'babel-runtime/core-js/object/assign'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/usr/local/lib/node_modules/migrate-mongoose/dist/cli.js:4:15)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)

Any help?

Migration is a bit wonky when migrations exist on the DB from another microservice and don't exist in the current migration folder.

This is likely an architecture issue on my end as each microservice could have its own database.

The migration goes like this in this case:

Migrate from "APPS" service: Works fine

Migrate from "USERS" service:

root@8a2d067bf04b:/usr/src/app# npm run migrate up create-apps-collection

> [email protected] migrate /usr/src/app
> migrate "up" "create-apps-collection"

Synchronizing database with file system migrations...
? The following migrations exist in the migrations folder but not in the database. Select the ones you want to import into the database 1574819499749-create-apps-collection.js
Adding migration /usr/src/app/database/migrations/1574819499749-create-apps-collection.js into database from file system. State is DOWN
UP:    1574819499749-create-apps-collection.js
All migrations finished successfully.

Once you do this it's fine. But it makes it a bit difficult to hit space to accept the migration.

This is due to sort option:

  async run(direction = 'up', migrationName) {
    await this.sync();

    const untilMigration = migrationName ?
      await MigrationModel.findOne({name: migrationName}) :
      await MigrationModel.findOne().sort({createdAt: -1}); // <- here

// rest of the method omitted

sort({ createdAt: -1 }) works only for the 'up' direction. For 'down' direction we need to sort({ createdAt: 1 })

In the 4.0.0's version info it says:

Given that nodejs has come a long way and async/await and other modern ES features have long been released, I decided to remove the babel project. Babel was causing more trouble than solving at this point and makes the maintenance of this package difficult.

However after reading the code it is still unclear how migrate-mongoose can now be used with ES6 files containing import and export statements without a transpiler. Can you please add an example or explain how can you now run migrations with ES6 files as stated in the current README.md file here:

https://github.com/balmasi/migrate-mongoose/blob/master/README.md#migrations1459287720919-my-migrationjs

/home/ubuntu/code/novelia>migrate up
Synchronizing database with file system migrations...
Cannot find module 'babel-runtime/regenerator'

I have node 7.4.0.

npm install babel-runtime

This fixes it when run in the project's directory.

Hi,
when I install the package only local and run migrage -d ... list
I get following error:

module.js:472
    throw err;
    ^

Error: Cannot find module './lib'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/run/media/roland/mData/Dropbox/work/semantify-core/node_modules/.bin/migrate:26:12)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)

tried programming access as you explained in #2

const migration = require('migrate-mongoose');
        var migrator = new migration({
            dbConnectionUri: util.format('mongodb://%s/%s', config.get('database.mongodb.host'), config.get('database.mongodb.name'))
        });

it seems the package.json sets the default file as cli.js and cli is firing;

Not enough non-option arguments: got 0, need at least 1

how can instead access lib.js instead?

.run(name, 'up')

Why the function requires a migration name to go up?

.run('up') should go all the way up.

Hi, I'm really like your package but have some suggestions:
I think package should use config file by default without passing it as a param every time;
And the second - use a JS file that exports JSON-object because sometimes it's needed to do some operations like getting vars from env

I executed the command migrate up. Then, there's this error.

Require Babel "^7.0.0-0", but was loaded with "6.26.3". If you are sure you have a compatible version of @babel/core, it is likely that something in your build process is loading the wrong version. Inspect the stack trace of this error to look for the first entry that doesn't mention "@babel/core" or "babel-core" to see what is calling Babel.

My project's @babel/core dependency is "^7.3.4".

Hey, we use Typescript + Mongoose in our project and tried use migrate-mongoose for adding migrations. But migrate-mongoose filtering files by reg-pattern, /\d{13,}\-.+.js$/.test(file)) can we change a little /\d{13,}\-.+.(j|t)s$/.test(file)). Or create new config param for this.
Thanks

I'm missing a way to access the DB connection information inside the up and down functions in migrations.

When using the CLI tool there are several ways to pass connection information (db uri): environment variables, the config file, and command line arguments. This is great, and migrate-mongoose uses this information to connect to the database, update the migration collection, and to give the up and down functions of migrations access to mongoose models.

In some of the migrations, I'd like to get more direct access to the mongo db to rename collections. My problem is that I can't figure out how to get the connection information that I provided to migrate-mongoose inside the up and down functions. The this context only lets me create mongoose models.

Ideally, I'd like to be able to do something like the following:

async function up() {
  const uri = this.config.dbConnectionUri // attach config object to mongoose connection.model
  const mongoClient = new MongoClient(uri, { useNewUrlParser: true });
  ... 
}

OR

async function up() {
  const mongoClient = this.client // pass mongoose connection instead of connection.model
  ... 
}

Does this make sense or am I missing something that is already in place?

Mongoose's default connection logic is deprecated as of 4.11.0 so in here is the following warning:

Db.prototype.authenticate method will no longer be available in the next major release 3.x as MongoDB 3.6 will only allow auth against users in the admin db and will no longer allow multiple credentials on a socket.
Please authenticate using MongoClient.connect with auth credentials.

See The useMongoClient Option

obviously not a big issue, but instead of:

./node_modules/.bin/migrate [command] [options]

isn't it better to do:

npx migrate [command] [options]

Running with es6 set to true, I get this error:
Unexpected Token when parsing migration. If you are using an ES6 migration file, use option --es6

It'd be helpful to see the full error since it's not obvious - syntax looks ok to me.

While attempting to run the example, the last one that has the User model and migration script, everything executes as though it worked, but the database is not updated.

I read a Closed issue in which someone seemed to have a similar problem, and indicated that it is fixed by creating a connection in the migration script. This works, but has very unpredictable behavior.

This library is excellent and does exactly what I was looking for, but it is in need of a complete example than can be used "out-of-the-box".

Thanks.

I've noticed that when the database connection returns an error the CLI still exits with status 0. This is a bit confusing as the connection failed and migrations were not run. I believe the program should exit with an error status such as 1.

This would be helpful when running migrations in a CI environment for example or in other cases where you'd like to break execution if the migrations were not applied.

Is there a way to run revert/rollback a migration without specifying its name?

Currently I have to do this:

npx migrate down MIGRATION_FILE_NAME

would be really helpful if there is a command to revert the last migration. something like:

npx migrate down:undo

The same functionality is supported in sequelize by running:

npx sequelize db:migrate:undo

One in the README would help

Migration tool not works when Mongoose model defined with 3 arguments - mongoose.model('model', ModelSchema, 'model-collection');. It uses model collection model instead model-collection, when call this('model')

Example code:

require('../model1');
require('../model2');

const bucketBefore = 'aaa';
const bucketAfter = 'bbb';

function updateImages(Model, bucketBefore, bucketAfter, done, imgField = 'img') {
    let modelCursor = Model.find({}).cursor();

    modelCursor.eachAsync(doc => {
        if (doc[imgField] && doc[imgField].includes(bucketBefore)) {
            doc[imgField] = doc[imgField].replace(bucketBefore, bucketAfter);
            doc.save();
        }
    }, done);

    modelCursor.on('error', err => done('migration error: ' + err));
}

exports.up = function up (done) {
    // works fine for such kind of models -  mongoose.model('model1', ModelSchema1);
    // but not works for models with different collection name -
    //  mongoose.model('model2', ModelSchema2, 'model2-collection');

    updateImages(this('model1'), bucketBefore, bucketAfter, done, 'image'); // <- OK
    updateImages(this('model2'), bucketBefore, bucketAfter, done, 'image'); // <- Problem

};

exports.down = function down(done) {
// ...
};

			return migrator.list()
				.then(migrations => {

UP:       1462277559915-assign_juice_to_business.js
UP:       1463476921686-user-slugs.js
UP:       1465554485481-simplify-juice-ratings.js
UP:       1466692183680-lowercase_juice_tags.js
UP:       1469097774997-fix-juice-rating-field.js
UP:       1469097774998-fix-hardware-rating-field.js
UP:       1471876808801-calculate-vote-ranks.js
UP:       1472798933688-draft-blog-entries.js

im using it programmatically and list should be not printing to console actually :)

Hello,

When we have authorization enabled in security option at mongo side, we have no option to set the username, password. Is there any way to provide username & password for authenticating the user?

I have this migration:

require('../models/user');

/**
 * Make any changes you need to make to the database here
 */
export async function up() {
  await this('User').update({}, {
    $set: {
      role: 'user',
    },
  });
}

/**
 * Make any changes that UNDO the up function side effects here (if possible)
 */
export async function down() {
  await this('User').update({}, {
    $unset: {
      role: '',
    },
  });
}

I run it with --es6 and it never exits. If I use DEBUG=* I can see that it is doing things, but it just never exits. This is the final log of the down command:

  mquery update +14ms users {} { '$setOnInsert': { createdAt: 2016-06-12T18:30:49.032Z },
  '$set': { updatedAt: 2016-06-12T18:30:49.032Z },
  '$unset': { role: 0 } } {}

I also tried replacing await with return.

Lost a few hours banging my head against the keyboard with this one. :'(
According to my research the following line from your lib.js will not work because you are no longer using the default mongoose connection:

migrationFunctions[direction].call(mongoose.model.bind(mongoose)

Model access will be directly thru the connection.model and not mongoose.model

I probably misundestood something about your module, because this kind of code

'use strict';

var User = require('../models/user.js');

/**
 * Make any changes you need to make to the database here
 */
exports.up = function up(done) {
    User = this('User');
    var user = new User({
        username: "foo"
    });
    user.save()
        .then(function(response) {
            console.log(response);
            done();
        })
        .catch(function(err) {
            console.log(err);
            done(err);
        });
};

seems to block on the call to mongoose save. Did I miss something about how thinks should be done ?

Here is the code i'm trying to run;

'use strict';

const config = require('config');
const util = require('util');
const mongodb = require('mongodb');

/**
 * Make any changes you need to make to the database here
 */
exports.up = function up (done) {
    const uri = util.format('mongodb://%s/%s', config.get('database.mongodb.host'), config.get('database.mongodb.name'));

    require('mongodb').connect(uri, function (err, db) {
        if (err) return done(err);

        done();
    });
};

/**
 * Make any changes that UNDO the up function side effects here (if possible)
 */
exports.down = function down (done) {
    done();
};

the output

λ migrate -d mongodb://localhost/development up
Failed to run migration assign_business_juices due to an error.
Not continuing. Make sure your data is in consistent state
Cannot read property 'then' of undefined
TypeError: Cannot read property 'then' of undefined
    at C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\dist\lib.js:313:51
    at Promise._execute (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\bluebird\js\release\debuggability.js:272:9)
    at Promise._resolveFromExecutor (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\bluebird\js\release\promise.js:473:18)
    at new Promise (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\bluebird\js\release\promise.js:77:14)
    at Migrator._callee2$ (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\dist\lib.js:307:34)
    at tryCatch (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\regenerator\runtime.js:88:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\regenerator\runtime.js:341:22)
    at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\regenerator\runtime.js:121:21)
    at tryCatch (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\regenerator\runtime.js:88:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\regenerator\runtime.js:285:24)
    at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\regenerator\runtime.js:121:21)
    at step (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\helpers\asyncToGenerator.js:17:30)
    at C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\babel-runtime\helpers\asyncToGenerator.js:28:20
    at run (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\core-js\library\modules\es6.promise.js:89:22)
    at C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\core-js\library\modules\es6.promise.js:102:28
    at flush (C:\Users\shala\AppData\Roaming\nvm\v6.0.0\node_modules\migrate-mongoose\node_modules\core-js\library\modules\_microtask.js:18:9)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

Created a config (.js) and specified --config <path.js> when running, but the command fails with "Missing required argument: d".

-d should not be required when using a config file which specifies the dbConnectionUri.

If there is no migrations collection present in the database, but migration files exist,
call to migrator.list() (as in examples here) results in:

(node:2911) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: createdMigration.toJSON is not a function

The exception was thrown from lib.js:

       var createdMigration = MigrationModel.create({
           name: migrationName,
           createdAt: timestamp
       });
       return createdMigration.toJSON();  // This is a Promise, not document! (valango)

Would you be willing to put an example config json in the project?

there, i have two question
no 1: how to support ts file?
no 2: for midway, i cannot use " import outside model ", so what can i do to realize the point?

When I run migration and all migrations are already in Up status, an error is thrown. But since I need to run migration with an automated script, every time this error comes up, my script breaks. Is there any way to bypass this error?

I have made a migration with cmd: migrate -d create initDb
This created me a file 1543573729152-initDB.js in the application folder.
I have created my models in application/models/ but unable to import them to migration
I have used these codes for importing

1. require('../application/models/<modelName>/index.js');
2. require('mongoose').model(<modelName>).modelName;
3. require('mongoose').model(<modelName>).schema;

exports.up = async function up(done) {
// Instead of mongoose.model(<modelName>,<modelNameSchema>) also used direct import variable
    await mongoose.model(<modelName>,<modelNameSchema>).insertMany([
        { name: 'Diabetes' },
        { name: 'Hypertension' },
        { name: 'Cardiovascular Disease' },
        { name: 'Other' },
        { name: 'None' },
    ]);
done();
};

I am new to the service and I dont know what I am doing wrong..

I have a following model:

module.exports = mongoose.model('Project', ProjectSchema);

And a migration script:

export async function up () {
     this('Project').update({}, { // lowercase doesn't work as well
         $set: { imageUrl: '' }
    });
}

What I get in response is this:

Synchronizing database with file system migrations...
Failed to run migration add_projects_image_url due to an error.
Not continuing. Make sure your data is in consistent state
Schema hasn't been registered for model "project".
Use mongoose.model(name, schema)
error Command failed with exit code 1.

Collection in the DB is called projects, but the model is Project, can this be an issue?

When setting up migrate-mongoose with a working mongo:// db connection and running migrate up the command will be issued (I can see that it is picking up my migration file by introducing an error into my JS code) but will never get past the point of "Synchronizing database with file system migrations".

This might be related to #6 , but I can see the migrate skript connect to mongo in mongo's logs, I can also see a migration appearing in my DB in the migrations table, it's state stays 'DOWN'.

When running the command with a DEBUG=* prepended, the only thing logged after Synchronizing... is mquery findOne migrations {} { sort: { createdAt: -1 }, projection: {} } +0ms

When I run "migrate up" and there are no migrations to run the program returns a non-zero value, which usually indicates an "error".

I think, it should return 0 instead of an error when there are no migrations to run.

I can't create a migration without connecting to MongoDB, which is obviously unnecessary.

Hi,

So I setup everything, including a migrate.json file and created the migration using:

migrate create dark-theme-apply

/* eslint-disable no-param-reassign */
import Setting from '../src/server/models/Setting';

/**
 * Make any changes you need to make to the database here
 */
async function up() {
  // Write migration here
  await new Promise((resolve, reject) => {
    try {
      const settingsToChange = Setting.find({ darkTheme: { $exists: false } });

      settingsToChange.forEach((setting) => {
        setting.darkTheme = false;
        setting.save();
      });

      resolve();
    } catch (err) {
      reject(err);
    }
  });
}

/**
 * Make any changes that UNDO the up function side effects here (if possible)
 */
async function down() {
  // Write migration here
}

export default { up, down };

On running the following command

migrate up dark-theme-apply

I get the following error:

(node:10208) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.

Synchronizing database with file system migrations...
Unexpected identifier

EDIT: Apparently, the error goes away when I dont use import but then when I use require it still doesnt work

Would be great to have the ability to specify that a particular migration or folder is for development seeding purposes versus production updates.

Heyho, I'm looking for a nice migration-tool for my mongo/mongoose project and found this one. It looks really promising but on the other hand there are dozens of long existing issues which (in many cases) are one-line-fixes but still persisting. So I'm wondering if it's safe to bet on this project also in the future?

Would be glad to hear from you @balmasi 🙂

Hi，I am looking for a migration framework to record my mongoose ORM changes and apply them to mongo database.

However I cannot see it clearly how it works from the introduction markdown file. Could you explain it further with an example.

I want it this way:

I modify my models' schema in models/user.js. Then I run "migrate apply" which will generate migration patches and apply them step by step into my new database:

It drops the old database and then fills data values using a new relationship mapper.

Is it possible in migrate-mongoose?

I use module-alias in my project. And when i run migration up, i get error like in title of issue. Any ideas how to fix this without a lot of time?

When using the CLI tool it prints this deprecation warning:

(node:38084) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.

This does not impact functionality, but it is rather unsightly, and would be an easy fix (just add the specified option).

When I try to run migrate up it says 'Cannot use import statement outside a module'

is per environment config files possible? (one for development, one for production)

Can i programatically access migrate-mongoose?

let migratorConfig = {
	migrationsPath: './src/migrations/',
	collectionName: 'migrations',
	useES6: true,
	autosync: true,
	dbConnectionUri: `mongodb://${config.get('database.mongodb.host')}/${config.get('database.mongodb.name')}`
};

let migrator = new migration(migratorConfig);

exports.run = function() {
	return new Promise(function(resolve) {

		return migrator.list().then(migrations => {

			let name = '';
			let count = 0;

			_.forEach(migrations, function(migration) {
				if (migration.state === 'down') {
					count++;
					name = migration.name;
				}
			});

			if (count === 0) {
				winston.info('[DB] database is up-to-date.');
				return resolve();
			}

			return migrator.run(name, 'up')
				.then(function() {
					winston.info('[DB] executed %s migrations.', count);
					return resolve();
				});
		});
	});
};

promises are not working as they should with programmatic way.

.run() calls returns without a wait for the promise completition and .catch() doesnt catch an error.