Stress test: scrape search results user story about puppeteer HOT 7 CLOSED

Why do you want to press enter? Usually there is an equally clickable element.

from puppeteer.

let Browser = require('../lib/Browser.js');
let browser = new Browser({ args: ['--no-sandbox'] });
browser.newPage().then(async page => {
  await page.navigate('https://www.google.com');
  let query = 'input[title="Search"]';
  page.focus(query);
  page.type('blin');
  page.keyboard.press('Enter');
  await page.waitFor('h3 a');
  let urls = await page.$$('h3 a', a => a.href);
  let i = 0;
  for (let url of urls) {
    i++;
    await page.navigate(url);
    await page.screenshot({
      path: i + '.png'
    });
  }
  browser.close();
});

page.waitFor didn't work for me. Added text nodes crash it, and a new node could make the selector resolve anywhere in the document.

from puppeteer.

• Running your script on a Mac, it does not work. I get stuck with blink| in the search field.
• Also, this script does not do what I want - I want to be clicking search results, not navigating manually. That covers the back/forward use cases
• And to add stress to it, I want to be clicking the search button. It is in DOM all the time, but invisible half of the time which adds fun!

from puppeteer.

The page.waitFor was broken and is now fixed in 28265fc

A few observations:

• For some reason, google.com handles "Enter" in input field differently every other time. Sometimes it issues a navigation to the https://www.google.com/search?q=blin endpoint, and sometimes it behaves as SPA and navigates to https://www.google.com/#q=Blin.
• Opening multiple tabs and navigating them altogether doesn't work - half of the tabs timeout navigation
• Trying to take screenshot simultaneously from multiple tabs doesn't work good - screenshotting hangs

The following works for me, but I don't really click the links and don't really click the "search" button. But we'll fix it!

const {Browser} = new require('.');
const browser = new Browser({headless: false});

const VIEWPORT = {width: 1000, height: 600};

browser.newPage().then(async page => {
  await page.setViewport(VIEWPORT);
  await page.navigate('https://google.com');
  await page.focus('#lst-ib');
  await page.type('Blin');
  await page.keyboard.press('Enter');

  try {
    await page.waitFor('div.g');
  } catch(e) {
    // Page might issue a navigation, so the page.waitFor will throw.
    await page.waitFor('div.g');
  }

  await page.screenshot({path: 'ee.png'});
  let links = await page.$$('div.g h3 > a', link => link.href);
  // have to do this sequentially =/
  for (let i = 0; i < links.length; ++i)
    await screenshotURL(links[i], `${i}.png`);
  browser.close();
});

async function screenshotURL(url, name) {
  let page = await browser.newPage();
  await page.setViewport(VIEWPORT);
  try {
    await page.navigate(url, { maxTime: 5000 });
  } catch (e) {
    // we did our best.
  }
  await page.screenshot({path: name});
  page.close();
  console.log('Done: ' + name);
}

from puppeteer.

After the offline discussion of the stress-test, here are the bullets:

• make mouse to scroll element into view before clicking (otherwise it doesn't work!)
• introduce page.press as a shortcut for page.keyboard.press
• don't do any network throttling other than offline mode. We're bad at emulating it
• implement page.waitForVisible(selector) which comes handy
• implement page.waitForNavigation
• implement page.back / page.forward
• improve support for DEBUG module for public api methods
• make screenshot tasks per browser, not per page. Screenshotting doesn't work across multiple tabs
• screenshots on mac seem to be unstable
• rename the keyboard's hold into down, and release into up
• page.waitForSelector should survive navigation (currently it doesn't)
• page.waitForSelector should timeout after some time (e.g. 30s by default)

from puppeteer.

The resulting script looks like this:

const {Browser} = require('puppeteer');
const browser = new Browser({headless: false});

browser.newPage().then(async page => {
  page.on('load', () => console.log('LOADED: ' + page.url()));
  await page.navigate('https://google.com');
  await page.waitFor('input[name=q]');
  await page.focus('input[name=q]');
  await page.type('blin');
  await page.press('Enter');
  for (let i = 0; i < 10; ++i) {
    let searchResult = `div.g:nth-child(${i + 1}) h3 a`;
    await page.waitFor(searchResult, {visible: true});
    page.click(searchResult);
    await page.waitForNavigation();
    await page.screenshot({path: `screenshot-${i + 1}.png`});
    await page.goBack();
  }
  browser.close();
});

The two checkboxes from feedback which are not addressed are filed separately:

• "screenshots on mac seem to be unstable" - which is #100
• "don't do any network throttling other than offline mode. We're bad at emulating it" - which is #63

from puppeteer.

You might be in different buckets too. I'm sure google is doing A/B testing. There's a way to force a bucket, but I'm not sure how.

from puppeteer.