segment-boneyard / nightmare Goto Github PK

Any plans to add support for SlimerJS and TrifleJS? I believe it has similar implementation.

Would be nice to see an engine method as an option (defaults to phantomjs).

var Nightmare = require('nightmare');
new Nightmare()
+  .engine('slimerjs')
  .goto('http://kayak.com')
  .evaluate(function () {
    return document.documentElement.innerHTML;
  }, function (res) {
    console.log(res);
  })
  .run();

Thoughts?

Thanks!

I got the following error when loading https://github.com/lgersman/pragmatic-angular:

phantom stdout: TypeError: 'undefined' is not a function (evaluating 'Array.prototype.forEach.call.bind(Array.prototype.forEach)')


phantom stdout:   https://assets-cdn.github.com/assets/frameworks-dabc650f8a51dffd1d4376a3522cbda5536e4807e01d2a86ff7e60d8d6ee3029.js:29
  https://assets-cdn.github.com/assets/frameworks-dabc650f8a51dffd1d4376a3522cbda5536e4807e01d2a86ff7e60d8d6ee3029.js:29

phantom stdout: ReferenceError: Can't find variable: $

  https://assets-cdn.github.com/assets/github-fa2f009761e3bc4750ed00845b9717b09646361cbbc3fa473ad64de9ca6ccf5b.js:1
  https://assets-cdn.github.com/assets/github-fa2f009761e3bc4750ed00845b9717b09646361cbbc3fa473ad64de9ca6ccf5b.js:1

Is it true that we can suppress these errors if we add onError handler on PhantomJS?
Currently we cannot add it through Nightmare.

because none of the lower layers will throw an error apparently :/

It's nearly impossible to debug automation scripts without seeing what's going on. So far I've been lamely scp-ing screenshots from .screen() calls back to my local machine and then inspecting.

The idea here would be to have a singleton server with a simple page showing a list of running Nightmare instances. Clicking an instance would let you observe the instance's behavior in close to real time, with the instance adding a screenshot step in between every action. When the viewer disconnected it would automatically stop screenshotting.

I'm just writing to ask if there is a simple way I can accomplish this.

I want to know if I can run a part of a nightmare, then stop and do whatever I like...say start a for loop, and then within that loop start up some more of the nightmare where the nightmare left off. As in, still logged in that is. Then I'd run a little more chunk of nightmare code, and then upon the next iteration begin some more nightmare code....all on the same login.

I'll try and write some psuedo-code to help explain it.

Nightmare = req (nightmare)

var myArray = [.., .., .., ..];
var newArray = [];
var myNM = new Nightmare

//Initial login with my nightmare.
myNM.goto('yahoo.com')
.type('#login', login)
.type('#pass', pass)
.click('#submit')
.run();

//Stop and begin a for loop.
for (i = 0; i < myArray.length; i++) {
    //Immediately invoke the previous nightmare and start some more code right where it had left off..
    myNM.wait()
    .type('#field1', myArray[i])
    .click('#checkBtn')
    .wait()
    //Give back some data...add it to the newArray for later.
    .evaluate(func() {
        newArray.push(document.querySelector('#result'));
        return document.querySelector('#result');
    }, func(res) { console.log(res); })
    .run()
}

//Print out the contents of the newArray we just built.
console.dir(newArray);

This is a great little nifty plugin!

I'm wondering if there's any way to get user input while Nightmare is running. I want to be able to pull my bank info automatically, but in order to log in to my bank account I have to:

1. Go to the login page
2. Type in the code generated by the website into my physical device (I can see the code by using .screenshot())
3. My device generates a new code that I type into the website to log in.

The problem is the third step. Is there a way for Nightmare to get some user input AFTER that it has been initiated? Either through stdin, or maybe just by using .wait() so that you have time to edit a textfile that it can later read from?

Thanks!

Let's say I have a simple search task:

new Nightmare()
  .use(tw.login('user', 'pw')
  .use(tw.search('503'))
  .use(function plugin () {
    return function (nightmare) {
      // function gets executed immediately, before tw.search() and tw.login() are done.
    }
  })

The return function get's executed immediately after nightmare is instantiated. If search() returns some properties that I want to use for plugin(), they would be undefined since the function was executed before the plugin.

Not sure if this can be done and if I'm missing something but the .use() function could be added to the queue so that all the other tasks before finishes first.

causing the bloat on hubeau

I grabbed the example code and modified it:

new Nightmare()
    .goto('http://kayak.com')
    .evaluate(function (page) {
        return page;
        // return document.documentElement.innerHTML;
    }, function (res) {
        console.log(page);
        // console.log(res);
    })
    .run();

The output in the terminal is:

>> node test.js

stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
ReferenceError: page is not defined

var Nightmare = require('nightmare');
var nightmare= new Nightmare();

nightmare
    .goto("http://www.google.com")
    .wait("input")
    .run(function( err ){
        if ( err ) throw err;
    });

results in

phantom stdout: SYNTAX_ERR: DOM Exception 12: An invalid or illegal string was specified.

phantom stdout:   phantomjs://webpage.evaluate():2
  phantomjs://webpage.evaluate():8
  phantomjs://webpage.evaluate():8

The selector used doesn't seem to matter - it always throws the error.

In Chrome on latest Android on a Nexus tablet:

And they're also not scrollable, so it's not possible to view the cut-off parts of the code at all.

Currently type just changes the value property of the element. This doesn't work with libraries that listen for key events. Therefore I suggest to use phantoms page.sendEvent("keypress", keys) for the implementation of type(). I unfortunately can't provide a pull request due to work restrictions.

Great work on the latest commits, much nicer now :) (Also I noticed some speed improvements since you changed to phantomjs-node)

There is one problem with the .run() method though:

new Nightmare()
    .goto('http://yahoo.com')
    .evaluate(function () {
      return document.title;
    }, function (title) {
      console.log('1')
    })
    .run(function (err, nightmare) {
      console.log('2');
    });

then the output would be:

  2
  1

which doesn't make sense, right? It should first report the 1 and then it should invoke the callback from .run(). Right now it's the opposite. Couldn't figure out why though.

Quick fix is to throw a .wait(500) in between. But this gets quite hard to guess the time it takes for .evaluate() to finish especially if it does not always have the same execution time.

Weak is a module that requires cpp compilation which is a terrible experience for Windows developers and it can be disabled through the phantom.create option: { dnodeOpts: { weak: false } }

Since phantom port is already an implemented option, this would be implemented in much of the same way.

1. one interface to get received resources
2. one interface to save special received resources to disk ?

The debug package at version ^0.7.4 is causing an error upon install. Changing it to ~0.7.4 solved the problem for me.

I'm trying to work with nightmare in an "after-login" scenario (already had nightmare do the logging in and that part works great). Only on the page that comes up after the login, it's utilizing some kind of client-side javascript to call back to the server where it checks the submitted form data and then updates the page with the results.

In an actual web browser such as Chrome the page actually does this; however in my nightmare script, the moment I tell it to click the submit button, it just resets the form or something...I end up with nothing but what I started with before entering the data. My screenshots literally show empty form, then typed-in(filled) form, then empty form again after the click.

Am I going about this all wrong? Or is there a real problem here?

Thanks.

http://phantomjs.org/api/webpage/method/open.html

• open(url, callback) {void}
• open(url, method, callback) {void}
• open(url, method, data, callback) {void}
• open(url, settings, callback) {void}

I have
phantomjs --version 1.4.0

When I try out any of the examples in the README, I always only get:
phantom stdout: XMLHttpRequest cannot load http://localhost:12301/dnode/info. Cannot make any requests from null.

What can I do?

eg:

nightmare.run();

if you dont care to wait for it to finish (like when running on a loop)

For example:

document.querySelectorAll('A[href="www.example.com"]').parentNode.querySelector('A.my-link').click();

in evaluate i can access to 'A.my-link' element, but .click() not work.
nightmare.click() - use selector, and not support ...parentNode.querySelector()

Would be awesome if we could add an option to change the userAgent.

Maybe we could add it as an optional parameter to the .goto function.

new Nightmare()
  .goto(agent, 'http://site.com')
  .goto('http://example.com')

But it might be better to set something like this global and add it to the options object for nightmare

new Nightmare({
    agent: 'Chrome ...'
  })
  .goto('http://example.com')

you can modify the user agent in phantomjs from page.settings.userAgent.

btw, great libary! So much easier to work with, especially without the callback hell and those unnecessary complex function names :p

Could you please add remote-debugger-port and remote-debugger-autorun?

I tried to hack this in myself,

index.js:
if (this.options.debuggerPort !== null) {
flags.push('--remote-debugger-port='+this.options.debuggerPort);
flags.push('--remote-debugger-autorun=yes');
}

But it didn't work, the console complained with the following:

phantom stdout: TypeError: 'undefined' is not a function (evaluating 'core_require('webpage')')

about:blank:5433
about:blank:5642
about:blank:179
about:blank:25
about:blank:5645
:1

There are some major problems with node-phantom:

• alexscheelmeyer/node-phantom#83
• alexscheelmeyer/node-phantom#89

The repository hasn't changed since 6 months ago and he hasn't replied to any of those issues. phantomjs-node is much more active and has the above issue fixed.

No need for those

this.page.setFn('EVIL_EVAL', ...)

hacks.

Love the tool, definetely better than casperjs.
Can we have more examples such as but not limited to :
--how to open and post to multiple urls
--dynamic loading,lazy-loading
--javascript injection

Thank you 👍

looks like there is no way of getting out of run method and continue the chain, see the code below..

var Nightmare = require( 'nightmare' );

new Nightmare()
  .goto( './pulse/jasmine/specrunner.html' )
  .run(function() {
    return $( '.alert' ).text();
  }, function( err, res ) {
    console.log( 'test status: ' + ( !!res.match( /passing/i ) ) );
    if( err ) { throw new Error( err ) };
    return;
  })
  .done(function( nightmare ) {
    // <<-- cannot reach here 
    console.log( 'done' );
    process.exit( 0 );
  });

Running into this when trying to run the examples:

  nightmare queueing action "evaluate" +0ms
  nightmare run +5ms
  nightmare .setup() creating phantom instance on port 12301 +1ms
phantom stdout: TypeError: undefined is not an object (evaluating 'phantom.args[0]')


phantom stdout:   /Users/matt/Playground/node_modules/nightmare/node_modules/phantom/shim.js:5421
  /Users/matt/Playground/node_modules/nightmare/node_modules/phantom/shim.js:5599
  /Users/matt/Playground/node_modules/nightmare/node_modules/phantom/shim.js:159
  /Users/matt/Playground/node_modules/nightmare/node_modules/phantom/shim.js:11 in require

This could definitely be an issue with phantomjs on 10.10, but any ideas?

Hi. Thanks for the neat tool.

I'm trying it out tonight and have hit a roadblock that I can't seem to make it around. I'm looking to add a conditional branch to my pipeline -- i.e. if a certain element is visible on the page, then click it and fill out some form, if that element is not visible then just proceed to whatever else you were going to do.

I can't seem to figure out a way to do this without dropping down to raw JS for that alternate flow in an evaluate -- which kind of defeats the whole purpose of using this abstraction. I thought about adding the control flow in the callback used in run(), but it seems that the nightmare instance returned as the second argument there has already been deconstructed, meaning I can't queue up any further interactions with it.

Are there any examples of this functionality?

Is it possible to set the language used by phantom in nightmare?

Disclaimer: I've never worked with PhantomJS before, only with nightwatch.
The README is missing an important line for newcomers-- how to execute the tests?
Also if only node testname.js needs to be executed (I don't know if I'm doing it correctly) no output is shown to track progress/debug.

I promise I'm not going to open a billion issues.

I'm tying to set viewport and take a screenshot with:

var Nightmare = require('nightmare');
new Nightmare()
  .goto('http://yahoo.com')
  .type('input[title="Search"]', 'github nightmare')
  .click('.searchsubmit')
  .viewport(320, 480)
  .screenshot('yahoo-320.png')
  .viewport(1000, 480)
  .screenshot('yahoo-1000.png')
  .run(function (err, nightmare) {
    if (err) return console.log(err);
    console.log('Done!');
  });

The resulting yahoo-320.png and yahoo-1000.png have no diff. I tried with viewport params as integers and strings. Same result.

I believe phantomjs allows to store cookies to an external file

I'm trying to hit 'npm install --save nightmare' on a fresh windows install. OS is windows 7 64-bit.

I've installed phantomjs via npm, and I have my PYTHON env variable pointing to python 3.4.

It looks like the 'weak' dependency is failing maybe?

Command line dump:

C:\dev\Hackathon\NightmareJS>npm install --save nightmare

[email protected] install C:\dev\Hackathon\NightmareJS\node_modules\nightmare\node_modules\phantom\node_modules\dnode\node_modules\weak
node-gyp rebuild

C:\dev\Hackathon\NightmareJS\node_modules\nightmare\node_modules\phantom\node_modules\dnode\node_modules\weak>node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild
gyp ERR! configure error
gyp ERR! stack Error: spawn ENOENT
gyp ERR! stack at errnoException (child_process.js:1001:11)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:792:34)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild"
gyp ERR! cwd C:\dev\Hackathon\NightmareJS\node_modules\nightmare\node_modules\phantom\node_modules\dnode\node_modules\weak
gyp ERR! node -v v0.10.32
gyp ERR! node-gyp -v v1.0.1
gyp ERR! not ok
npm WARN optional dep failed, continuing [email protected]
[email protected] node_modules\nightmare
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
└── [email protected] ([email protected], [email protected], [email protected], [email protected])

Are there any additional dependencies I'm missing perhaps?

PhantomJS allows setting the username and password to use when receiving a WWW-Authenticate request. It'd be very useful if this could be added to supported options pass through of nightmare as well.

Why would a wait timeout not result in an error considering it would not be yielding the expected result?

nightmarejs: command not found.

It must work a little differently than phantomjs on the command line. As far as I know I installed it properly.

I'm struggling with this one because it seems like it should just work.

When I've got things set up like so.... it should pass in arg1 and arg2 into the fn on the evaluate when it calls it right? Well, it doesn't seem to. In addition it seems to also throw errors that don't make sense sometimes.

The return statement is where I thought I could return whatever I wanted as a result, but when I have the callback called with res (for the result (what was returned from fn), sometimes it gives me some errors, and sometimes it just does nothing.

[pseudo-code...sort of]

global var1 // foo
global var2 // bar

new Nightmare()
    .blah-blah-blah
    .evaluate(fn(param1, param2){
        some code
        console.log(param1 + " " + param2);  // foo bar
        return "The result is pretty.";
    }, cb(res) { console.log(res); }, ["var1", "var2])
    .run();  // In theory produces the result of calling fn with var1 and var2, ...which would
    // be "foo bar" in the console, along with "The result is pretty."

I was wondering your thoughts on adding a method to break out of the nightmare chain, so that you can execute things in Node's context. I'm thinking of an action like this:

exports.node = function( callback, done ){
  callback( this, done );
};

which would allow something like this in the nightmare chain:

nightmare
    .goto("http://www.google.com")
    .node( function( n, done ){
        n.screenshot("out.png");
        ///image processing in the local filesystem
        request.post(/* send the processed file somewhere */, function(){
            done();
        });
    })
    //continue the chain
    .run();

You can accomplish nearly the same thing with .evaluate, but this is more explicit and gives you access to the nightmare instance.

A cool idea might be to add the ability for each plugin to append metadata to the nightmare instance. I know that metalsmith does the exact same thing. It would be incredibly helpful when you try to communicate with those different plugins and easier for testing.

Nightmare()
  .use(plugin())
  .run(function () {
    var val = nightmare.metadata.value
    assert(val, 'test')
  })

function plugin () {
  return function (nightmare) {
    nightmare.metadata.value = 'test'
  }
}

won't be too difficult to implement. I can make a PR if this make sense to add in.

is there an easy way to access the window object ?

Cloned project. cd to project. npm install. created a google.js with:

var Nightmare = require('nightmare');
new Nightmare()
  .goto('https://google.com')
    .type('input#gbqfq', 'github nightmare')
    .click('button#gbqfba')
    .run(function(err, nightmare) {
      if (err) return fn(err);
      fn();
    });

tried running node google.js

Error:

stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
ReferenceError: fn is not defined
    at /Users/dstanton/Sites/_projects/nightmarejs/nightmare/google.js:9:7
    at next (/Users/dstanton/Sites/node_modules/nightmare/lib/index.js:59:34)
    at f (/Users/dstanton/Sites/node_modules/nightmare/node_modules/once/once.js:16:25)
    at Proto.apply (/Users/dstanton/Sites/node_modules/nightmare/node_modules/phantom/node_modules/dnode/node_modules/dnode-protocol/index.js:123:13)
    at Proto.handle (/Users/dstanton/Sites/node_modules/nightmare/node_modules/phantom/node_modules/dnode/node_modules/dnode-protocol/index.js:99:19)
    at D.dnode.handle (/Users/dstanton/Sites/node_modules/nightmare/node_modules/phantom/node_modules/dnode/lib/dnode.js:140:21)
    at D.dnode.write (/Users/dstanton/Sites/node_modules/nightmare/node_modules/phantom/node_modules/dnode/lib/dnode.js:128:22)
    at SockJSConnection.ondata (stream.js:51:26)
    at SockJSConnection.emit (events.js:95:17)
    at Session.didMessage (/Users/dstanton/Sites/node_modules/nightmare/node_modules/phantom/node_modules/shoe/node_modules/sockjs/lib/transport.js:220:25)

'use strict';

var Nm = require('nightmare');
var browser = new Nm();
var events = [ 'initialized', 'loadStarted', 'loadFinished', 'urlChanged', 'navigationRequested', /*'resourceRequested', 'resourceReceived'*/, 'consoleMessage', 'alert', 'confirm', 'prompt' ];

events.forEach(function (event) {
  browser.on(event, function() {
    console.log('event %s: %j', event, arguments);
  });
});

browser
  .useragent('Cheddar Cheese Dip')
  .goto('http://twitter.com')
  .screenshot('twitter_com.jpg')
  .goto('http://httpbin.org')
  .screenshot('httpbin_org.jpg')
  .evaluate(function () {
    return document.title;
  }, function (res) {
    console.log('title: %j', res);
  })
  .goto('http://gfycat.com')
  .screenshot('gfycat_com.jpg')
  .run(function (err) {
    console.log('err:', err);
  });

twitter_com.jpg will become a black image.

Here's a log:

$ node app.js                                                                         
event navigationRequested: {"0":"http://twitter.com/","1":"Other","2":true,"3":true}  
event loadStarted: {}                                                                 
event navigationRequested: {"0":"https://twitter.com/","1":"Other","2":true,"3":true} 
event loadFinished: {"0":"fail"}                                                      
event initialized: {}                                                                 
event navigationRequested: {"0":"http://httpbin.org/","1":"Other","2":true,"3":true}  
event loadStarted: {}                                                                 
event urlChanged: {"0":"http://httpbin.org/"}                                         
event initialized: {}                                                                 
event loadFinished: {"0":"success"}                                                   
title: "httpbin(1): HTTP Client Testing Service"                                      
event navigationRequested: {"0":"http://gfycat.com/","1":"Other","2":true,"3":true}   
event loadStarted: {}                                                                 
event urlChanged: {"0":"http://gfycat.com/"}                                          
event initialized: {}                                                                 
event loadFinished: {"0":"success"}                                                   
err: null                                                                             

Is nightmare seems like it would be a suitable solution for web scraping. Do you think you could write an example?

Right now I have to do:

  var nm = new Nightmare()
  nm.setupInstance(function (instance) { instance.exit() })

So my test runner can exit and the Node process can terminate (I know Mocha does some trickery to close stuff for you but other frameworks do not -- for instance tap and tape).

Would it makes sense to either expose the PHANTOM_INSTANCE or add a method to exit?

I have a fork of phantomjs-node that supports PhantomJS 2, and I am trying to get it accepted upstream. The changes so far look pretty clean and are backwards compatible with PhantomJS 1.9.7, so hopefully that goes smoothly. In the meantime, people that want to give Phantom 2 a spin should be able to replace the nightmare version in their package.json with "https://github.com/bprodoehl/nightmare/archive/phantom2.tar.gz"

The PR for phantomjs-node is here: amir20/phantomjs-node#209

Brew is only for mac. You're not alone in this world. Some(lot) of us are on pc or just debian, ubuntu and other great OS. We don't need to know you've a Mac.

Sorry for saying this like this, but it's really annoying to see a lot of people thinking that all of developers are on mac. It's not true.

Here's an example of how we can abstract out Swiftly methods:

new Nightmare()
  .use(Swiftly.login(email, password))
  .use(Swiftly.task(description, paths))
  .done(function(err, nightmare){
    if (err) return fn(err);
    fn();
  });

exports.login = function(email, password){
  return function(nightmare) {
    nightmare
      .goto('https://swiftly.com/login')
        .type('#username', email)
        .type('#password', password)
        .click('.button--primary')
      .wait();
  };
}

exports.task = function(description, paths){
  return function(nightmare){
    nightmare
      .goto('https://swiftly.com/create')
      .wait(2000)
      .type('#body', description);

    paths.forEach(function (path) {
      nightmare.upload('input[name=qqfile]', path);
    });

    nightmare
      .wait(5000)
      .click('#task-pay-button')
      .wait(500)
      .click('#pay-button')
      .wait();
    }
  };
}

In the simple api section of the website, the code formatting(?) is converting the double-quotes to ” (ironically enough, not 201f for the opening quote) and the apostrophes to ‘ and ’ in the code snippets.

This stops users from copy-and-pasting the examples into their terminal or their editor, along with looking out of place with the rest of the fixed-width font.