Deployment failing on new instance about meteord HOT 30 CLOSED

If you want a quicker way of testing it without mup (and you have docker locally), try:

docker run -i -t \
    -e ROOT_URL=http://localhost:9090 \
    -v /some/path/on/your/machine/to/the/bundle/directory:/bundle \
    -p 9090:80 \
    abernix/meteord:base

Where /some/path/on/your/machine/to/the/bundle/directory is the directory produced by meteor build.

from meteord.

This should work...

$ cd your-meteor-app
$ meteor build ../my-meteor-build # waiiiiit...
$ cd ../my-meteor-build
$ pwd

The value of that last line should be used in place of /some/path/on/your/machine/to/the/bundle/directory.

from meteord.

Thanks a lot. Things are still failing, but my feedback loop is going to be much shorter

from meteord.

I re-run docker build -t gsabran/meteord . and it worked, so I'll see what I can do from here. (my image is 426.2MB large while the ones you've there are smaller, but I dunno if that's meaningful)

I've also tried to connect to the failing docker image doing docker run -i -t --entrypoint="/bin/bash" meteorhacks/meteord:base but I don't really know what this does and where this leads me :) I am within a new instance of the image, or within the failing container? Where do I edit meteor's node modules? if I do $ which meteor, or $ find / -name 'fibers' -type d from there I get nothing

from meteord.

Thanks for opening this issue.

I actually ended up toying around with this for a while last night and I found a problem, but it was related to me putting the --unsafe-perm flag in the meteor build command before Meteor 1.4.2 is actually used. (It breaks versions of Meteor which don't support it.) However, this is not the issue you're experiencing and it would have been fixed when I rolled back the image (the node-4.6.1 image would still have this problem).

So ultimately, I cannot reproduce this – I can deploy perfectly fine using the same image. The error isn't an error – it appears that something is just throw-ing the value of an object in string form (e.g. var object = {}; throw object;. And ultimately, debugging within fibers isn't going to reveal anything – the error is probably happening in async code and simply being caught there.

Do you mind throwing together a simple reproduction repo that this happens on with the same version of Meteor that you're using? Including a mup.js, etc. (I'll fill in my own EC2 host, obviously).

from meteord.

I just pulled the latest version of onbuild from Dockerhub, and it's still failing because of the --unsafe-perm flag with Meteor 1.4.1.

from meteord.

@christinedraper Hmm, I'll check it out right now. Thanks!

from meteord.

@christinedraper Sorry about that. Can you try it again? I think it should work fine now. When I did it last night I actually had to rebuild it from source and publish it and it wasn't as easy as I hoped, but I took advantage of the opportunity to temporarily fix meteor/meteor#7806 too. If you still have a problem, maybe open a new issue but let me know if it works for you.

from meteord.

@gsabran Just noticed you were using meteorhacks/meteord:base in the command you had above. Perhaps a mistake, but if you are in fact using the meteorhacks/meteord:base image you will most certainly have a problem. 😄

from meteord.

I'm using abernix/meteord:base-node-4.4.7 in my mup config. The meteorhacks is coming from what you suggested here zodern/meteor-up#172 (comment) and honestly I've little understanding of what I'm actually doing so I just copy-paste!

It looks like the error is thrown here so I was hopping that I could modify that line and log some more information on the error instead of just getting [object Object] to get a better understanding.

I'll try to get a simple repro, but since each deployement try takes me 5mn to run and I don't have any idea why things are failing, I anticipate that it'll take me a while to come to a good simple repro.

from meteord.

Thanks for paying attention!

from meteord.

Sorry, was definitely a typo on my part. I'm fairly confident that things are back how they were now, meaning the same configuration that's been in use for almost 3 months now. Waiting to hear back from @christinedraper for confirmation above though. Still working fine for me with the old image.

from meteord.

would /some/path/on/your/machine/to/the/bundle/directory be what I get with pwd from my meteor folder?

from meteord.

This is the end of my broadcast day so I'll be gone pretty soon, but I'll leave some more tips:

1. The same docker run command from above but with a --entrypoint=/bin/bash flag will drop you into the container without starting the Meteor App.
2. Once in that container, there should be a ./run_app.sh script in that directory (/opt/meteord). You can start the app with it and Ctrl-C it without the container getting reset.
3. Try with an empty app and see if you have similar results.
4. You should be able to modify the bundle on your local machine and see results on the next ./run_app.sh invocation.

Good luck. :)

from meteord.

Nice, that helps. I've changed the fiber code to get better logs. The error that is thrown is:

EventEmitter {
  id: 1,
  s: 
   { options: 
      { disconnectHandler: [Object],
        cursorFactory: [Object],
        reconnect: true,
        emitError: true,
        size: 5,
        socketOptions: {},
        ssl: true,
        auto_reconnect: true,
        clientInfo: [Object] },
     bson: {},
     Cursor: 
      { [Function]
        super_: [Object],
        define: [Object],
        INIT: 0,
        OPEN: 1,
        CLOSED: 2,
        GET_MORE: 3 },
     logger: { className: 'Mongos' },
     seedlist: [ [Object], [Object] ],
     haInterval: 10000,
     disconnectHandler: { s: [Object], length: [Getter] },
     index: 0,
     connectOptions: 
      { readPreference: [Object],
        native_parser: false,
        promiseLibrary: [Object] },
     debug: false,
     localThresholdMS: 15,
     clientInfo: 
      { driver: [Object],
        os: [Object],
        platform: 'Node.js v4.4.7, LE, mongodb-core: 2.0.11' } },
  authProviders: 
   { mongocr: { bson: {}, authStore: [] },
     x509: { bson: {}, authStore: [] },
     plain: { bson: {}, authStore: [] },
     gssapi: { bson: {}, authStore: [] },
     sspi: { bson: {}, authStore: [] },
     'scram-sha-1': { bson: {}, authStore: [], id: 0 } },
  state: 'destroyed',
  connectingProxies: [],
  connectedProxies: [],
  disconnectedProxies: 
   [ EventEmitter {
       domain: null,
       _events: [Object],
       _eventsCount: 7,
       _maxListeners: undefined,
       id: 3,
       s: [Object],
       ismaster: null,
       lastIsMasterMS: 6,
       monitoringProcessId: null,
       initalConnect: true,
       wireProtocolHandler: {},
       _type: 'server',
       clientInfo: [Object],
       lastUpdateTime: 0,
       lastWriteDate: 0,
       staleness: 0 },
     EventEmitter {
       domain: null,
       _events: [Object],
       _eventsCount: 7,
       _maxListeners: undefined,
       id: 2,
       s: [Object],
       ismaster: null,
       lastIsMasterMS: 2,
       monitoringProcessId: null,
       initalConnect: true,
       wireProtocolHandler: {},
       _type: 'server',
       clientInfo: [Object],
       lastUpdateTime: 0,
       lastWriteDate: 0,
       staleness: 0 } ],
  authenticating: false,
  index: 0,
  haTimeoutId: 
   { '0': null,
     _called: true,
     _idleTimeout: -1,
     _idlePrev: null,
     _idleNext: null,
     _idleStart: 1357,
     _onTimeout: null,
     _repeat: null },
  ismaster: null,
  domain: null,
  _events: 
   { timeout: { [Function: g] listener: [Function] },
     error: { [Function: g] listener: [Function] },
     connect: { [Function: g] listener: [Function] },
     joined: [Function],
     left: [Function],
     reconnect: [Function] },
  _eventsCount: 6,
  _maxListeners: undefined }

It looks like a mongo thing. Will keep looking.

from meteord.

It's just strange because that's not an error at all. It's just an EventEmitter object. Something is just throw-ing it – but it could be from anywhere. If you can print ex.stack that might have more info, maybe.

from meteord.

Yes, the stack I just got is coming from at runWithEnvironment (packages/meteor/dynamics_nodejs.js:115:1)

One step at a time!

from meteord.

Can you show the whole stack? That is an ambiguous point of trace as most everything runs through it.

from meteord.

I don't have better atm. Since it's not an error object indeed, and there's no trace, I did

try {
  throw new Error();
} catch (e) {
  console.log('stack is', e.stack);
}

which printed me

stack is Error
    at Object.Future.throw (/bundle/bundle/programs/server/node_modules/fibers/future.js:293:25)
    at Object.<anonymous> (/bundle/bundle/programs/server/node_modules/fibers/future.js:355:15)
    at runWithEnvironment (packages/meteor/dynamics_nodejs.js:115:1)

(lines in fiber don't match the public package exactly since I've been adding logs)

from meteord.

I'm afraid you're going to have to take a different approach of spitting out lots of log messages in your app leading up to when the error gets thrown. That's just async code and you're essentially just catching a tick of the processor. I doubt you'll find anything there.

from meteord.

ok, thanks. I'll remove chunk of codes in my app and see what breaks

from meteord.

Don't forget to do build again between changes. Do try a basic app before you gut yours though! :)

from meteord.

I've reduced my app to just including meteorhacks:[email protected] in the packages, that's it (see https://github.com/gsabran/deployement-failure-reproduction)

I was running

sudo rm -rf ../my-meteor-build && meteor build ../my-meteor-build && sudo docker run -i -t -e ROOT_URL=http://localhost:9090 -e MONGO_URL=mongodb://... -v /home/ubuntu/my-meteor-build:/bundle -p 9090:80 abernix/meteord:base

between each change to clear the builds.

so

git clone https://github.com/gsabran/deployement-failure-reproduction.git
cd deployement-failure-reproduction
meteor build ../my-meteor-build
sudo docker run -i -t \
  -e ROOT_URL=http://localhost:9090 \
  -e MONGO_URL=<some mongo url or anything that will prevent meteor from failing> \
  -v /home/ubuntu/my-meteor-build:/bundle \
  -p 9090:80 abernix/meteord:base

should reproduce the issue.

meteohacks:fast-render has not been updated in 2 months, so this is weird. Let me know if you are able to reproduce.

If that's helpful, I'm running locally node 4.6.1 I also tried specifying abernix/meteord:base-node-4.6.1 and that fails as well

I'll keep drilling down what's failing by using local packages.

from meteord.

coming from meteorhacks:[email protected] one step at a time...

from meteord.

If I comment that line out: https://github.com/meteorhacks/meteorx/blob/master/lib/server.js#L22 things are fine. I've no idea what aronuda is doing here. It's a very old package that's a dependency of meteorhacks:fast-render that's a dependency of meteorhacks:flow-router-ssr.

from meteord.

Is there any chance a recent change in the docker image made this obscure package to crash?

from meteord.

@abernix Works for me now - thanks!

from meteord.

I've restarted everything on a fresh EC2 instance, downloaded meteor, set node to version 4.4.7, created a new project, set the meteor version to 1.4.1.1 (haven't tested 1.4.1.2), added meteorhacks:[email protected] package, run sudo rm -rf ../my-meteor-build && meteor build ../my-meteor-build && sudo docker run -i -t -e ROOT_URL=http://localhost:9090 -e MONGO_URL=mongodb://... -v /home/ubuntu/my-meteor-build:/bundle -p 9090:80 abernix/meteord:base and that crashed.
Probably the best repro I can get.

from meteord.

This might be a mongo issue. If I change the mongo url, things don't crash. I've no idea why things are crashing with the first url since it's working fine on other instances...

from meteord.

After some more testing, I'm confident this is a meteor error that is not related to meteord. I've opened an issue on meteor: meteor/meteor#7954

The issue only happened with some real database url, so it only happened for me when deploying, and error logs made it really hard to understand.

from meteord.