puppeteer / puppeteer Goto Github PK
View Code? Open in Web Editor NEWNode.js API for Chrome
Home Page: https://pptr.dev
License: Apache License 2.0
Node.js API for Chrome
Home Page: https://pptr.dev
License: Apache License 2.0
.wait(selector) // wait until selector you specified appear in a DOM tree.
Example:
var Browser = require('puppeteer').Browser;
var browser = new Browser();
browser.newPage().then(async page => {
page.on('dialog', dialog => {
dialog.accept('test');
});
console.log(await page.evaluate(() => prompt('q?')));
browser.close();
});
Dialog accepting/dismissing throws on error on headless:
Error: Protocol error (Page.handleJavaScriptDialog): Could not handle JavaScript dialog
Upstream bug: crbug.com/718235
Currently we have the following api to emulate devices:
let names = page.emulatedDevices(); // to get list of emulatable devices
page.emulate('iphone 6'); // to emulate "iphone 6"
Problems with this api:
emulatedDevices()
as a getter and under a page makes me feel that the device list might change over time and from page to page.How about:
// Puppeteer defines the "Device" class
class Device {
constructor(name, userAgent, viewport, isMobile = true, hasTouch = true) {
this.name = name;
this.userAgent = userAgent;
this.viewport = viewport;
this.isMobile = isMobile;
this.hasTouch = hasTouch;
}
}
// The Device class and a list of all available devices are exposed
// in the top-level require
const {Browser, Device, devices} = require('puppeteer');
// `devices` list is an array with additional string properties. Devices
// could be both iterated and accessed via device names
devices[0];
devices.nexus5;
// If needed, a hand-made device could be done
let myDevice = new Device('handmade', 'handmade-webkit', {
width: 100,
height: 100,
dpi: 2,
});
// Devices would be convenient to import from npm
let exoticDevice = require('list-of-exotic-devices-for-puppeteer');
// Finally, page can start emulating a device
page.emulateDevice(device);
page.emulatedDevice(); // returns currently-emulated device
// And we can change page's orientation
page.setOrientation('landscape'); // also could be 'portrait'
This kind of API would make emulation a first-class citizen in Puppeteer - which makes a lot of sense since devtools device emulation is very comprehensive.
The screenshots should follow the following logic:
devicePixelRatio
devicePixelRatio = 1
so that screenshots are consistent across platformsUnit tests on mac are timeouting by the end of the session, no matter which test is the last.
This seems to be caused by https://bugs.chromium.org/p/chromium/issues/detail?id=741689
For now, the workaround is to re-start the browser during unit-testing once in a while. We should be able to remove the workaround once the upstream issue is fixed.
There should be a way to set a file for file input:
<input type="file" id="file">
The phantom's approach to handling javascript dialogs is convenient and familiar: instead of firing Alert
, Prompt
and Confirm
events, we should use onAlert
, onPrompt
and onConfirm
callbacks. It would be also convenient to have a complimentary Dialog
event which dispatches dialog
class.
Thanks to the emulation domain, we can emulate mobile devices.
Is there a need for such an api in puppeteer? If yes, what's the scenario and what would api look like?
While "puppeteer" is apt, there are a few downsides: 3 syllables (and kinda a mouthful), difficult spelling (surprisingly), kinda plain, the npm name is taken. There's been an interest amongst a few folks to retitle the project before shipping.
I have collected a few possibilities below.
I'm not explaining each name for simplicity, I'd rather them stand on their own. However, most have connotations and associations with things like: legend of sleepy hollow, headlessness, phantom of the opera, scientific method/testing, shadow puppets.
blacksmith
silhouette
christine
theremin
bradshaw
dudley
leroux
sobaka
irving
mombi
scrim
curie
Any other proposals? Comment and I will integrate new ones into the list above.
Page.evaluate
should handle unserializable values properly.
For example, the following returns false
:
let result = await page.evaluate(() => NaN);
console.log(Number.isNaN(result));
Whereas it should be true
.
There should be a capability to ignore certificate errors during navigation. See #66 for proposal and cyrus-and/chrome-remote-interface#183 for original discussion.
For example, we can add ignoreSSL
navigation option:
page.navigate('https://example.com', { ignoreSSL: true })
We should use standard request definition from https://developer.mozilla.org/en-US/docs/Web/API/Request, user should be able to intercept and fetch requests similarly to web service workers.
We should run chrome with the --remote-debugging-port=0 at all times and read the port from system.out
Consider the following example, which takes 10 screenshots, each of a different 50px
x50px
square.
browser.newPage().then(async page => {
await page.navigate('http://example.com');
var promises = [];
for (var i = 0; i < 10; ++i) {
promises.push(page.screenshot({
path: i + '.png',
clip: {x: 50 * i, y: 0, width: 50, height: 50},
});
}
await Promise.all(promises);
browser.close();
});
Unfortunately, this doesn't work - calls to page.screenshot
interfere with each other, failing to clip proper rect.
Mind if we switch to 2-space identation? That's more common in JS projects these days and we tend to stick with Google's style guide in the GoogleChrome org.
We can extend the google style guide in eslintrc.js to enforce it :)
(taken from gr2m/headless-chrome-test#1)
Puppeteer should have an offline mode to test PWA's. For example:
page.setOfflineMode(true);
Emulation of other network conditions might be useful as well, but we can wait with them until there are good scenarios.
The chrome-remote-interface
dependency served us good. However, our usage of chrome-remote-interface
is minimal: we use only .send()
method and event subscription. This is not hard to implement (e.g. there's a lighthouse implementation: connection.js).
The following should be implemented:
.click(selector, options)
.type(selector, text, options)
This library is sweet!
It would be awesome to add a convenience method to dump the innerHTML of the body
element.
There should be an API to access cookies.
Rationale: waitForSelector is rarely useful on its own. I have a DOM element, but it is off screen / display:none, etc.
We can make waitFor generic and execute it either upon every task (by default) or upon tasks that had specific activities in them (optional).
utilitites.waitForSelector = (selector) => {
return page.waitFor(() => document.querySelector(selector), ['style'], selector);
}
utilitites.waitForLayout = () => {
return page.waitFor(() => true, ['layout']);
}
utilitites.waitForFrame = () => {
return page.waitFor(() => true, ['frame']);
}
Rely upon async/promise-based evaluation instead. Use Runtime.inspectRequested as a signal to notify puppeteer.
The screenshots API could be improved:
page.screenshot
, accept single optional options
objecttype
optional and default it to either jpeg
or png
.filePath
option to the screenshot's options
object.saveScreenshot
is an alias to the screenshot
with filePath
optiontakeScreenshot
is an alias for the screenshot
Navigation is hard.
Make sure the following navigation scenarios work properly:
page.navigate('not-a-url')
should return false
page.navigate('https://expired.badssl.com/')
should return false
page.navigate('http://example.com/non-existing-page')
should return false
page.navigate('http://example.com')
with no internet should return false
page.navigate('data:text/html,hello')
should return true
window.location.href = 'http://example.com'
should be reported to puppeteer, probably via the Navigated
event.All of this should be also applicable to frame navigation in #4
.mouseMoved(x, y, options = {})
.mousePressed(x, y, options = {})
.mouseReleased(x, y, options = {})
.tap(x, y, options = {})
.touchmove()
.touchend()
The following script hangs for me waiting for "click" to happen on the second link.
However, if I make viewport width 1000 instead of 300, the script works just fine.
const {Browser} = require('puppeteer');
const browser = new Browser({headless: false});
browser.newPage().then(async page => {
page.on('load', () => {
console.log('LOADED: ' + page.url());
});
// Make width 1000 instead of 300 and the scripts works just fine
await page.setViewport({width: 300, height: 300});
await page.navigate('https://google.com');
await page.waitFor('input[name=q]');
await page.focus('input[name=q]');
await page.type('blin');
await page.press('Enter');
for (let i = 0; i < 10; ++i) {
let searchResult = `div.g:nth-child(${i + 1}) h3 a`;
await page.waitFor(searchResult, {visible: true});
page.click(searchResult);
await page.waitForNavigation();
await page.screenshot({path: `screenshot-${i + 1}.png`});
await page.goBack();
}
browser.close();
});
Running the script with DEBUG=*page node scrape.js
shows what's going on.
Looks like we're clicking outside of viewport and the click turns out to be a noop.
@JoelEinbinder, wdyt?
I attempted to create a following stress test:
This is pretty much impossible using existing API :) Dumping observations here...
page.waitFor
resolves even if element is not on the screen (display:none, etc). Many elements are in DOM too early, no way to click-via-screen them.page.waitFor
does not time timeoutpage.*
.timestamp
s. google.com does not like thatpage.waitFor(predicate)
page.sleep(time)
page.waitForNavigation
in case navigation is initiated via click [i'll fix]page.waitFor(selector)
in my code is always followed by either page.click
or page.focus
with that element. Handles would make it look like (await page.waitFor(selector)).click()
.Lots of things I think could be simpler
"override" "set" "get" ...
const {Browser} = new require('.');
const browser = new Browser({headless: true});
browser.newPage().then(async page => {
await page.evaluate(() => window);
browser.close();
});
This code throws an error:
Object reference chain is too long
which happens because window is non-serializable.
Otherwise _onRequestIntercepted
and alike are exposed.
Currently, all the tests are failing on Travis-Ci. These should be fixed!
@paulirish shared some insights how this could be debugged: debugging Travis-Ci
There should be a nice README which
There's a need to navigate to URL with a POST
method rather then GET
method, e.g. to simulate form submission.
For this, the page.navigate()
method should accept options
object with method
parameter.
There should be a frames inspection API with an ability to evaluate code in frame
Currently, the docs/api.md is written manually to resemble the node.js documentation. The motivation was to make the doc look familiar to an average node dev.
However, the markup is tiring to maintain manually. It would be nice to generate the doc somehow from a simplistic format which would be easy to edit.
@paulirish, @ebidel - do you guys have any suggestions? @ebidel mentioned it's easy to setup a documentation website somehow, could you please share a link?
I did a (very) quick survey of the issues submitted to chrome-remote-interface so far and selected some features that users seem to struggle with using the low-level Chrome Debugging Protocol API.
Security.handleCertificateError
). Self-signed certificates in testing environments are quite common.Target.createBrowserContext
) so to avoid launching multiple instances if one wants a pristine session (e.g., browser.newIsolatedPage()
or even expose the context concept through the API).These may or may not suit this project so I'm not I'm not filing multiple issues; this is mainly to have a place to discuss about them.
We need a way to run a test with --inspect-brk flag to make it debuggable!
We should lint jsdoc types.
I spent some time trying to make closure compiler work with node.js, but it didn't work due to issues with node module requires.
While using puppeteer there are still occasions where I want to use the raw protocol.
I'd certainly want to do this against the Page, and potentially the browser as well.
I want to send methods and get responses. Additionally, I want to listen for specific events.
Not sure of the right API and if it needs to be transactional as to not mixup state.
Every public API method should have a test. At least a sanity test.
Given the #5 is done, there should be a fullPage
option in the page.screenshot
method.
There should be also an alias page.takeFullScreenshot
or page.takeFullPageScreenshot
(or maybe both?)
Support browser contexts (Target.createBrowserContext) so to avoid launching multiple instances if one wants a pristine session. See proposal #66 and original discussion at cyrus-and/chrome-remote-interface#118.
A viable user scenario might be testing several users logged in simultaneously into the service.
We might expose browser contexts as a string literal option to the browser.newPage:
browser.newPage(); // creates a new page in a default browser context
browser.newPage({ context: 'default' }); // same as previous call
browser.newPage({ context: 'another-context' }); // creates a page in another browser context
The setBlockedURLs might be occasionally removed from the protocol - we should not rely on it.
Instead, the request interception API should be implemented
The network request interception is a huge part of phantom.js capabilities which puppeteer was lacking. With the new addition to the devtools protocol, it is now possible to implement.
e.g.
// Fill an element
await browser.fill('#username', 'myUser')
// set an element's `value` value
await browser.setValue('#comment', 'my feelings')
// Type in an element
await browser.type('#password', 'Yey!ImAPassword!')
Thanks for the awesome module 🏆
I realize the docs are a WIP.
From the README:
Puppeteer sets an initial page size to 400px x 300px, which defines the screenshot size. The page size can be changed with Page.setSize() method
setSize()
does not however appear to have any API docs over in https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md. Looking through the repo via code-search the README entry for this method appears to be the only one in there :)
Is there another issue tracking adding docs for methods like this?
I've been long thinking about scripting interactions using something similar to the webpagetest test scripts.
It'd be great to have some interoperability with that preexisting script syntax/format/language. While I'm not a huge fan of the API, it has been established for some people already.
I'm certain that Patrick would have some feels regarding this idea too.
eg
visit https://google.com
focus #searchInput
type 'where is the world‘s best cheesecake?'
press button[type='submit']
screenshot my-image.jpg
Is this in scope for puppeteer? Or perhaps something that could just use puppeteer under the covers?
Today, there are two methods in page's API:
page.evaluate
which evaluates code in the page and returns result.page.evaluateAync
which too evaluates code in the page and, if the evaluation result is promise, waits for promise to resolve and returns the promise value.The page.evaluate
is straightforward, but page.evaluateAsync
has cryptic name and hard to grasp. Should we remove the page.evaluateAsync
and always await promise in page.evaluate
?
If we want to keep them separate, is there a better name for the page.evaluateAsync
?
Illustrating with examples, the following snippet returns "42":
var Browser = require('puppeteer').Browser();
var browser = new Browser();
browser.newPage().then(async page => {
console.log(await page.evaluate(() => 6 * 7)); // prints '42'
browser.close();
});
However, if we try to return a promise, we will get an empty object as a result:
var Browser = require('puppeteer').Browser();
var browser = new Browser();
browser.newPage().then(async page => {
console.log(await page.evaluate(() => Promise.resolve(6 * 7))); // prints '{}'
browser.close();
});
In order to get the '42' as a returned value, one would need to use page.evaluateAsync
:
var Browser = require('puppeteer').Browser();
var browser = new Browser();
browser.newPage().then(async page => {
console.log(await page.evaluateAsync(() => Promise.resolve(6 * 7))); // prints '42'
browser.close();
});
This kind of situation could be avoided if we were to await promise in page.evaluate
.
For the record: in case of merging evaluate
with the evaluateAsync
methods, we should make sure that evaluating the code from-inside the inpagecallback works successfully (which is a bit tricky since page is on a pause)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.