Git Product home page Git Product logo

chrome-aws-lambda's Introduction

chrome-aws-lambda

chrome-aws-lambda TypeScript Chromium Donate

Chromium Binary for AWS Lambda and Google Cloud Functions

Install

npm install chrome-aws-lambda --save-prod

This will ship with appropriate binary for the latest stable release of puppeteer (usually updated within a few days).

You also need to install the corresponding version of puppeteer-core (or puppeteer):

npm install puppeteer-core --save-prod

If you wish to install an older version of Chromium, take a look at Versioning.

Usage

This package works with all the currently supported AWS Lambda Node.js runtimes out of the box.

const chromium = require('chrome-aws-lambda');

exports.handler = async (event, context, callback) => {
  let result = null;
  let browser = null;

  try {
    browser = await chromium.puppeteer.launch({
      args: chromium.args,
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: chromium.headless,
      ignoreHTTPSErrors: true,
    });

    let page = await browser.newPage();

    await page.goto(event.url || 'https://example.com');

    result = await page.title();
  } catch (error) {
    return callback(error);
  } finally {
    if (browser !== null) {
      await browser.close();
    }
  }

  return callback(null, result);
};

Usage with Playwright

const chromium = require('chrome-aws-lambda');
const playwright = require('playwright-core');

(async () => {
  const browser = await playwright.chromium.launch({
    args: chromium.args,
    executablePath: await chromium.executablePath,
    headless: chromium.headless,
  });

  // ...

  await browser.close();
})();

You should allocate at least 512 MB of RAM to your Lambda, however 1600 MB (or more) is recommended.

Running Locally

Please refer to the Local Development Wiki page for instructions and troubleshooting.

API

Method / Property Returns Description
font(url) {?Promise<string>} Provisions a custom font and returns its basename.
args {!Array<string>} Provides a list of recommended additional Chromium flags.
defaultViewport {!Object} Returns more sensible default viewport settings.
executablePath {?Promise<string>} Returns the path the Chromium binary was extracted to.
headless {!boolean} Returns true if we are running on AWS Lambda or GCF.
puppeteer {!Object} Overloads puppeteer and returns the resolved package.

Fonts

The Amazon Linux 2 AWS Lambda runtime is no longer provisioned with any font faces.

Because of this, this package ships with Open Sans, which supports the following scripts:

  • Latin
  • Greek
  • Cyrillic

To provision additional fonts, simply call the font() method with an absolute path or URL:

await chromium.font('/var/task/fonts/NotoColorEmoji.ttf');
// or
await chromium.font('https://raw.githack.com/googlei18n/noto-emoji/master/fonts/NotoColorEmoji.ttf');

Noto Color Emoji (or similar) is needed if you want to render emojis.

For URLs, it's recommended that you use a CDN, like raw.githack.com or gitcdn.xyz.

This method should be invoked before launching Chromium.

On non-serverless environments, the font() method is a no-op to avoid polluting the user space.


Alternatively, it's also possible to provision fonts via AWS Lambda Layers.

Simply create a directory named .fonts and place any font faces you want there:

.fonts
├── NotoColorEmoji.ttf
└── Roboto.ttf

Afterwards, you just need to ZIP the directory and upload it as a AWS Lambda Layer:

zip -9 --filesync --move --recurse-paths .fonts.zip .fonts/

Overloading

Since version 8.0.0, it's possible to overload puppeteer with the following convenient API:

interface Browser {
  defaultPage(...hooks: ((page: Page) => Promise<Page>)[])
  newPage(...hooks: ((page: Page) => Promise<Page>)[])
}

interface BrowserContext {
  defaultPage(...hooks: ((page: Page) => Promise<Page>)[])
  newPage(...hooks: ((page: Page) => Promise<Page>)[])
}

interface Page {
  block(patterns: string[])
  clear(selector: string)
  clickAndWaitForNavigation(selector: string, options?: WaitForOptions)
  clickAndWaitForRequest(selector: string, predicate: string | RegExp, options?: WaitTimeoutOptions)
  clickAndWaitForRequest(selector: string, predicate: ((request: HTTPRequest) => boolean | Promise<boolean>), options?: WaitTimeoutOptions)
  clickAndWaitForResponse(selector: string, predicate: string | RegExp, options?: WaitTimeoutOptions)
  clickAndWaitForResponse(selector: string, predicate: ((request: HTTPResponse) => boolean | Promise<boolean>), options?: WaitTimeoutOptions)
  count(selector: string)
  exists(selector: string)
  fillFormByLabel(selector: string, data: Record<string, boolean | string | string[]>)
  fillFormByName(selector: string, data: Record<string, boolean | string | string[]>)
  fillFormBySelector(selector: string, data: Record<string, boolean | string | string[]>)
  fillFormByXPath(selector: string, data: Record<string, boolean | string | string[]>)
  number(selector: string, decimal?: string, property?: string)
  selectByLabel(selector: string, ...values: string[])
  string(selector: string, property?: string)
  waitForInflightRequests(requests?: number, alpha: number, omega: number, options?: WaitTimeoutOptions)
  waitForText(predicate: string, options?: WaitTimeoutOptions)
  waitUntilVisible(selector: string, options?: WaitTimeoutOptions)
  waitWhileVisible(selector: string, options?: WaitTimeoutOptions)
  withTracing(options: TracingOptions, callback: (page: Page) => Promise<any>)
}

interface Frame {
  clear(selector: string)
  clickAndWaitForNavigation(selector: string, options?: WaitForOptions)
  clickAndWaitForRequest(selector: string, predicate: string | RegExp, options?: WaitTimeoutOptions)
  clickAndWaitForRequest(selector: string, predicate: ((request: HTTPRequest) => boolean | Promise<boolean>), options?: WaitTimeoutOptions)
  clickAndWaitForResponse(selector: string, predicate: string | RegExp, options?: WaitTimeoutOptions)
  clickAndWaitForResponse(selector: string, predicate: ((request: HTTPResponse) => boolean | Promise<boolean>), options?: WaitTimeoutOptions)
  count(selector: string)
  exists(selector: string)
  fillFormByLabel(selector: string, data: Record<string, boolean | string | string[]>)
  fillFormByName(selector: string, data: Record<string, boolean | string | string[]>)
  fillFormBySelector(selector: string, data: Record<string, boolean | string | string[]>)
  fillFormByXPath(selector: string, data: Record<string, boolean | string | string[]>)
  number(selector: string, decimal?: string, property?: string)
  selectByLabel(selector: string, ...values: string[])
  string(selector: string, property?: string)
  waitForText(predicate: string, options?: WaitTimeoutOptions)
  waitUntilVisible(selector: string, options?: WaitTimeoutOptions)
  waitWhileVisible(selector: string, options?: WaitTimeoutOptions)
}

interface ElementHandle {
  clear()
  clickAndWaitForNavigation(options?: WaitForOptions)
  clickAndWaitForRequest(predicate: string | RegExp, options?: WaitTimeoutOptions)
  clickAndWaitForRequest(predicate: ((request: HTTPRequest) => boolean | Promise<boolean>), options?: WaitTimeoutOptions)
  clickAndWaitForResponse(predicate: string | RegExp, options?: WaitTimeoutOptions)
  clickAndWaitForResponse(predicate: ((request: HTTPResponse) => boolean | Promise<boolean>), options?: WaitTimeoutOptions)
  fillFormByLabel(data: Record<string, boolean | string | string[]>)
  fillFormByName(data: Record<string, boolean | string | string[]>)
  fillFormBySelector(data: Record<string, boolean | string | string[]>)
  fillFormByXPath(data: Record<string, boolean | string | string[]>)
  getInnerHTML()
  getInnerText()
  number(decimal?: string, property?: string)
  selectByLabel(...values: string[])
  string(property?: string)
}

To enable this behavior, simply call the puppeteer property exposed by this package.

Refer to the TypeScript typings for general documentation.

Page Hooks

When overloaded, you can specify a list of hooks to automatically apply to pages.

For instance, to remove the Headless substring from the user agent:

async function replaceUserAgent(page: Page): Promise<Page> {
  let value = await page.browser().userAgent();

  if (value.includes('Headless') === true) {
    await page.setUserAgent(value.replace('Headless', ''));
  }

  return page;
}

And then simply pass that page hook to defaultPage() or newPage():

let page = await browser.defaultPage(replaceUserAgent);

Additional bundled page hooks can be found on /build/hooks.

Versioning

This package is versioned based on the underlying puppeteer minor version:

puppeteer Version chrome-aws-lambda Version Chromium Revision
10.1.* npm i chrome-aws-lambda@~10.1.0 884014 (92.0.4512.0)
10.0.* npm i chrome-aws-lambda@~10.0.0 884014 (92.0.4512.0)
9.1.* npm i chrome-aws-lambda@~9.1.0 869685 (91.0.4469.0)
9.0.* npm i chrome-aws-lambda@~9.0.0 869685 (91.0.4469.0)
8.0.* npm i chrome-aws-lambda@~8.0.2 856583 (90.0.4427.0)
7.0.* npm i chrome-aws-lambda@~7.0.0 848005 (90.0.4403.0)
6.0.* npm i chrome-aws-lambda@~6.0.0 843427 (89.0.4389.0)
5.5.* npm i chrome-aws-lambda@~5.5.0 818858 (88.0.4298.0)
5.4.* npm i chrome-aws-lambda@~5.4.0 809590 (87.0.4272.0)
5.3.* npm i chrome-aws-lambda@~5.3.1 800071 (86.0.4240.0)
5.2.* npm i chrome-aws-lambda@~5.2.1 782078 (85.0.4182.0)
5.1.* npm i chrome-aws-lambda@~5.1.0 768783 (84.0.4147.0)
5.0.* npm i chrome-aws-lambda@~5.0.0 756035 (83.0.4103.0)
3.1.* npm i chrome-aws-lambda@~3.1.1 756035 (83.0.4103.0)
3.0.* npm i chrome-aws-lambda@~3.0.4 737027 (81.0.4044.0)
2.1.* npm i chrome-aws-lambda@~2.1.1 722234 (80.0.3987.0)
2.0.* npm i chrome-aws-lambda@~2.0.2 705776 (79.0.3945.0)
1.20.* npm i chrome-aws-lambda@~1.20.4 686378 (78.0.3882.0)
1.19.* npm i chrome-aws-lambda@~1.19.0 674921 (77.0.3844.0)
1.18.* npm i chrome-aws-lambda@~1.18.1 672088 (77.0.3835.0)
1.18.* npm i chrome-aws-lambda@~1.18.0 669486 (77.0.3827.0)
1.17.* npm i chrome-aws-lambda@~1.17.1 662092 (76.0.3803.0)
1.16.* npm i chrome-aws-lambda@~1.16.1 656675 (76.0.3786.0)
1.15.* npm i chrome-aws-lambda@~1.15.1 650583 (75.0.3765.0)
1.14.* npm i chrome-aws-lambda@~1.14.0 641577 (75.0.3738.0)
1.13.* npm i chrome-aws-lambda@~1.13.0 637110 (74.0.3723.0)
1.12.* npm i chrome-aws-lambda@~1.12.2 624492 (73.0.3679.0)
1.11.* npm i chrome-aws-lambda@~1.11.2 609904 (72.0.3618.0)
1.10.* npm i chrome-aws-lambda@~1.10.1 604907 (72.0.3582.0)
1.9.* npm i chrome-aws-lambda@~1.9.1 594312 (71.0.3563.0)
1.8.* npm i chrome-aws-lambda@~1.8.0 588429 (71.0.3542.0)
1.7.* npm i chrome-aws-lambda@~1.7.0 579032 (70.0.3508.0)
1.6.* npm i chrome-aws-lambda@~1.6.3 575458 (69.0.3494.0)
1.5.* npm i chrome-aws-lambda@~1.5.0 564778 (69.0.3452.0)
1.4.* npm i chrome-aws-lambda@~1.4.0 555668 (68.0.3419.0)
1.3.* npm i chrome-aws-lambda@~1.3.0 549031 (67.0.3391.0)
1.2.* npm i chrome-aws-lambda@~1.2.0 543305 (67.0.3372.0)
1.1.* npm i chrome-aws-lambda@~1.1.0 536395 (66.0.3347.0)
1.0.* npm i chrome-aws-lambda@~1.0.0 526987 (65.0.3312.0)
0.13.* npm i chrome-aws-lambda@~0.13.0 515411 (64.0.3264.0)

Patch versions are reserved for bug fixes in chrome-aws-lambda and general maintenance.

Compiling

To compile your own version of Chromium check the Ansible playbook instructions.

AWS Lambda Layer

Lambda Layers is a new convenient way to manage common dependencies between different Lambda Functions.

The following set of (Linux) commands will create a layer of this package alongside puppeteer-core:

git clone --depth=1 https://github.com/alixaxel/chrome-aws-lambda.git && \
cd chrome-aws-lambda && \
make chrome_aws_lambda.zip

The above will create a chrome-aws-lambda.zip file, which can be uploaded to your Layers console.

Alternatively, you can also download the layer artifact from one of our CI workflow runs.

Google Cloud Functions

Since version 1.11.2, it's also possible to use this package on Google/Firebase Cloud Functions.

According to our benchmarks, it's 40% to 50% faster than using the off-the-shelf puppeteer bundle.

Compression

The Chromium binary is compressed using the Brotli algorithm.

This allows us to get the best compression ratio and faster decompression times.

File Algorithm Level Bytes MiB % Inflation
chromium - - 136964856 130.62 - -
chromium.gz Gzip 1 51662087 49.27 62.28% 1.035s
chromium.gz Gzip 2 50438352 48.10 63.17% 1.016s
chromium.gz Gzip 3 49428459 47.14 63.91% 0.968s
chromium.gz Gzip 4 47873978 45.66 65.05% 0.950s
chromium.gz Gzip 5 46929422 44.76 65.74% 0.938s
chromium.gz Gzip 6 46522529 44.37 66.03% 0.919s
chromium.gz Gzip 7 46406406 44.26 66.12% 0.917s
chromium.gz Gzip 8 46297917 44.15 66.20% 0.916s
chromium.gz Gzip 9 46270972 44.13 66.22% 0.968s
chromium.gz Zopfli 10 45089161 43.00 67.08% 0.919s
chromium.gz Zopfli 20 45085868 43.00 67.08% 0.919s
chromium.gz Zopfli 30 45085003 43.00 67.08% 0.925s
chromium.gz Zopfli 40 45084328 43.00 67.08% 0.921s
chromium.gz Zopfli 50 45084098 43.00 67.08% 0.935s
chromium.br Brotli 0 55401211 52.83 59.55% 0.778s
chromium.br Brotli 1 54429523 51.91 60.26% 0.757s
chromium.br Brotli 2 46436126 44.28 66.10% 0.659s
chromium.br Brotli 3 46122033 43.99 66.33% 0.616s
chromium.br Brotli 4 45050239 42.96 67.11% 0.692s
chromium.br Brotli 5 40813510 38.92 70.20% 0.598s
chromium.br Brotli 6 40116951 38.26 70.71% 0.601s
chromium.br Brotli 7 39302281 37.48 71.30% 0.615s
chromium.br Brotli 8 39038303 37.23 71.50% 0.668s
chromium.br Brotli 9 38853994 37.05 71.63% 0.673s
chromium.br Brotli 10 36090087 34.42 73.65% 0.765s
chromium.br Brotli 11 34820408 33.21 74.58% 0.712s

License

MIT

chrome-aws-lambda's People

Contributors

abdulg avatar adriangmweb avatar alixaxel avatar diegoparrilla avatar g-rath avatar ik1ne avatar mxschmitt avatar samuraiseoul avatar screamz avatar tnolet avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

chrome-aws-lambda's Issues

Permissions issue on AWS, spawn EACCES

I know this is pretty new, but was just wondering if you've been successful running on AWS lambda yet. I hit permissions error when running on AWS lambda. Could it be that the binary is neither statically linked nor built for the matching version of Amazon Linux as specified here
I'm running the latest version of this package 67.0.3372 with puppeteer 1.2.0

Thank you.

Invoke locally - spawn EACCES error

any idea how I can invoke a function locally?

sls invoke local --function crawl --path mockInput.json leads to:


Serverless: Bundling with Webpack...
Time: 505ms
Built at: 29/04/2019 17:40:41
           Asset      Size     Chunks             Chunk Names
    src/crawl.js  8.84 KiB  src/crawl  [emitted]  src/crawl
src/crawl.js.map    13 KiB  src/crawl  [emitted]  src/crawl
Entrypoint src/crawl = src/crawl.js src/crawl.js.map
[./src/crawl.js] 1.29 KiB {src/crawl} [built]
[./src/helpers.js] 3.17 KiB {src/crawl} [built]
[chrome-aws-lambda] external "chrome-aws-lambda" 42 bytes {src/crawl} [built]
{
    "errorMessage": "Error: [500] spawn EACCES",
    "errorType": "Error",
    "stackTrace": [
        "Error: Error: [500] spawn EACCES",
        "    at ./src/crawl.js.module.exports.handler (/.webpack/service/src/crawl.js:139:11)",
        "    at <anonymous>"
    ]
}

I want to use that for tests...

Running from s3

I would like to run from s3 in order to minimize deployment package sizes. But I am not sure what's the recommended way to do so. Should I unpack it all? Thanks for all recommendations.

Better usage documentation

As a developer, I rely on clear and concise usage documentation when installing and (properly) using a package. The example in the readme is great, but, there should be clear documentation on setting up the serverless.yml file to properly utilize this package as a layer.

Also, unless the binaries shipped with this package are added to the package:exclude serverless.yml section, then the Lambda function still contains a massive binary. This should be made clear to other developers in the documentation, along with what else can be excluded from your package to optimize Lambda size.

As a starting point, here is what I did to get this package working for myself:

  1. Put built layer zip into a folder in your project:

image

  1. Setup Lambda layer in serverless.yml to use zip:
layers:
  chrome:
    path: layer
  1. Add new layer to the function you need it in:
functions:
  yourFunctionName:
    ...
    layers:
      - {Ref: ChromeLambdaLayer}
  1. Tell serverless not to include it in your actual Lambda function:
package:
  exclude:
    - node_modules/chrome-aws-lambda/bin/**

Emoji Support

Hello 👋 This is a really great tool you built! 😄

I found a related issue (#4) but the solution is to add a custom font in the .fonts directory.

Which font would I use to get emoji support? Have you had any success with rendering emoji? 🤔

waitForSelector always times out.

What I'm trying to do:

  • I'm trying to automate the login process of Amazon Connect CCP.

Code that reproduces the issue:

module.exports.agentLogin = async (event, context) => {
    let browser = null;
    try
    {

    browser = await puppeteer.launch({
      args: chromium.args,
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: chromium.headless,
    });

    const page = await browser.newPage();
    
    const CCP_URL = "https://ccalderon-reinvent.awsapps.com/connect/ccp#/";
    const CCP_USERNAME = process.env.CCP_USERNAME;
    const CCP_PASSWORD = process.env.CCP_PASSWORD;

    //logging in
    await page.goto(CCP_URL, { waitUntil: "networkidle0" });
    console.log("waiting for username input");
    await page.waitForSelector('#wdc_username');
    console.log('filling login form...');
    await page.type('input[type="username"]', CCP_USERNAME);
    await page.focus('input[type="password"]');
    await page.keyboard.type(CCP_PASSWORD);
    await page.keyboard.press('Enter');
    console.log('Submitted login form');
    }
  catch (error) {
    return context.fail(error);
  }
  finally {
    if (browser !== null) {
      await browser.close();
    }
  }
  return context.succeed("Fine!")
};

Error Message from Lambda:

{
"errorMessage": "waiting for selector "#wdc_username" failed: timeout 30000ms exceeded",
"errorType": "TimeoutError",
"stackTrace": [
"new WaitTask (/var/task/node_modules/puppeteer-core/lib/DOMWorld.js:554:28)",
"DOMWorld._waitForSelectorOrXPath (/var/task/node_modules/puppeteer-core/lib/DOMWorld.js:483:22)",
"DOMWorld.waitForSelector (/var/task/node_modules/puppeteer-core/lib/DOMWorld.js:437:17)",
"Frame.waitForSelector (/var/task/node_modules/puppeteer-core/lib/FrameManager.js:606:47)",
"Frame. (/var/task/node_modules/puppeteer-core/lib/helper.js:109:23)",
"Page.waitForSelector (/var/task/node_modules/puppeteer-core/lib/Page.js:1079:29)",
"module.exports.agentLogin (/var/task/handler.js:29:16)",
"",
"process._tickDomainCallback (internal/process/next_tick.js:228:7)"
]
}

Please note that if you inspect the page, you will find an input field with id #wdc_username.

Protocol error (Target.createTarget): Target closed

Hello, I'm getting the following error while trying to take a screenshot on Lambda:

  const page = await incognitoContext.newPage();
  // ...
  return page.screenshot();
2018-12-17T10:07:15.426Z 856540b8-01e3-11e9-b71f-a1bd004ededf Error: Protocol error (Target.createTarget): Target closed.
at Promise (/opt/nodejs/node_modules/puppeteer-core/lib/Connection.js:73:56)
at new Promise (<anonymous>)
at Connection.send (/opt/nodejs/node_modules/puppeteer-core/lib/Connection.js:72:12)
at Browser._createPageInContext (/opt/nodejs/node_modules/puppeteer-core/lib/Browser.js:174:47)
at BrowserContext.newPage (/opt/nodejs/node_modules/puppeteer-core/lib/Browser.js:368:26)
at takeScreenshot (/var/task/index.js:35:41)
at exports.handler (/var/task/index.js:100:28)
at <anonymous>
at process._tickDomainCallback (internal/process/next_tick.js:228:7)

If I try to run the binary locally (Ubuntu 18.04.1 LTS) and open https://example.com, it indeed closes almost immediately:

time ./chromium-72.0.3617.0 --disable-accelerated-2d-canvas --disable-background-timer-throttling --disable-breakpad --disable-client-side-phishing-detection --disable-cloud-import --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-gesture-typing --disable-gpu --disable-hang-monitor --disable-infobars --disable-notifications --disable-offer-store-unmasked-wallet-cards --disable-offer-upload-credit-cards --disable-popup-blocking --disable-print-preview --disable-prompt-on-repost --disable-setuid-sandbox --disable-software-rasterizer --disable-speech-api --disable-sync --disable-tab-for-desktop-share --disable-translate --disable-voice-input --disable-wake-on-wifi --enable-async-dns --enable-simple-cache-backend --enable-tcp-fast-open --hide-scrollbars --media-cache-size=33554432 --metrics-recording-only --mute-audio --no-default-browser-check --no-first-run --no-pings --no-sandbox --no-zygote --password-store=basic --prerender-from-omnibox=disabled --use-mock-keychain --single-process https://example.com
[1217/114728.139380:WARNING:resource_bundle.cc(346)] locale_file_path.empty() for locale 
./chromium-72.0.3617.0 --disable-accelerated-2d-canvas  --disable-breakpad     0,12s user 0,03s system 26% cpu 0,560 total

If I try to open https://twitter.com instead, the process segfaults:

Thread 3 "Chrome_IOThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4b0f700 (LWP 22900)]
0x000055555993e872 in ?? ()
(gdb) bt
#0  0x000055555993e872 in ?? ()
#1  0x0000555558cd9630 in ?? ()
#2  0x00005555589c5bad in ?? ()
#3  0x0000555558bd325c in ?? ()
#4  0x0000555558bd3033 in ?? ()
#5  0x0000555558bd76ad in ?? ()
#6  0x0000555558be137f in ?? ()
#7  0x000055555ab12897 in ?? ()
#8  0x0000555558359547 in ?? ()
#9  0x00005555583593e6 in ?? ()
#10 0x000055555a269684 in ?? ()
#11 0x000055555a268e88 in ?? ()
#12 0x000055555a2637c3 in ?? ()
#13 0x000055555a263f91 in ?? ()
#14 0x000055555a27633a in ?? ()
#15 0x0000555559913d78 in ?? ()
#16 0x000055555991337f in ?? ()
#17 0x00005555599138a2 in ?? ()
#18 0x00005555599a02f9 in ?? ()
#19 0x000055555992da45 in ?? ()
#20 0x00005555589e8354 in ?? ()
#21 0x00005555599668df in ?? ()
#22 0x000055555999b18e in ?? ()
#23 0x00007ffff79b96db in start_thread (arg=0x7ffff4b0f700) at pthread_create.c:463
#24 0x00007ffff633b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

If I remove --single-process, it doesn't segfault anymore but again closes with a slight delay:

time ./chromium-72.0.3617.0 --disable-accelerated-2d-canvas --disable-background-timer-throttling --disable-breakpad --disable-client-side-phishing-detection --disable-cloud-import --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-gesture-typing --disable-gpu --disable-hang-monitor --disable-infobars --disable-notifications --disable-offer-store-unmasked-wallet-cards --disable-offer-upload-credit-cards --disable-popup-blocking --disable-print-preview --disable-prompt-on-repost --disable-setuid-sandbox --disable-software-rasterizer --disable-speech-api --disable-sync --disable-tab-for-desktop-share --disable-translate --disable-voice-input --disable-wake-on-wifi --enable-async-dns --enable-simple-cache-backend --enable-tcp-fast-open --hide-scrollbars --media-cache-size=33554432 --metrics-recording-only --mute-audio --no-default-browser-check --no-first-run --no-pings --no-sandbox --no-zygote --password-store=basic --prerender-from-omnibox=disabled --use-mock-keychain https://twitter.com
[1217/115133.515117:WARNING:resource_bundle.cc(346)] locale_file_path.empty() for locale 
[1217/115133.540260:WARNING:resource_bundle.cc(346)] locale_file_path.empty() for locale 
./chromium-72.0.3617.0 --disable-accelerated-2d-canvas  --disable-breakpad     0,18s user 0,10s system 17% cpu 1,551 total

I've tried the previous release as well (chromium-71.0.3563.0) with the same results. I would expect the browser to stay open. I tried the release that comes with Puppeteer and it does stay open with the same flags. Any ideas?

The report looks similar to #3 but the error message is different so I thought I would open another one.

Build chrome with non-latin characters support

Hi @alixaxel , thanks a lot for your work, its been really useful and easy to use.

Although I have one request, could you give me some pointers to build chrome with support for japanase, korean, chinese, etc and other non-latin characters? While using puppeteer + chrome-aws-lambda on some japanase pages, when I take a screenshot, characters are not correctly rendering:

image

I'm guessing I'd need to tweak the ansible play, then run it to get the expected build. But it might be beyond my experience, I'll try tho, so if you have any pointers I'd appreciate it, if possible I'll try to submit the corresponding PR.

chrome.executablePath is returning null

Have problem with executablePath. It's returning null.

const browser = await puppeteer.launch({
      args: chrome.args,
      headless: chrome.headless,
      executablePath: await chrome.executablePath,
})

Usage with puppeteer-cluster

Hi & thanks for this library! It's looking cleaner than other implementations.

I'm wondering if it's better to spawn a browser on each implementation or to use something like puppeteer-cluster?

WIthout puppeteer-cluster, how do you manage bursts of traffic to the function? Is lambda really that smart to scale the server or would it run out of memory quickly?

Cannot find module 'puppeteer-core/lib/FrameManager'

I receive the following error when trying to run the usage example on AWS Lambda:

{
    "errorMessage": "Cannot find module 'puppeteer-core/lib/FrameManager'",
    "errorType": "Error",
    "stackTrace": [
        "Function.Module._resolveFilename (module.js:547:15)",
        "Function.Module._load (module.js:474:25)",
        "Module.require (module.js:596:17)",
        "require (internal/module.js:11:18)",
        "Object.<anonymous> (/var/task/node_modules/chrome-aws-lambda/source/puppeteer/lib/FrameManager.js:6:11)",
        "Module._compile (module.js:652:30)",
        "Object.Module._extensions..js (module.js:663:10)",
        "Module.load (module.js:565:32)",
        "tryModuleLoad (module.js:505:12)",
        "Function.Module._load (module.js:497:3)"
    ]
}

I verified that the node_modules/chrome-aws-lambda/source/puppeteer/lib/FrameManager.js file exists

Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"

While trying the example code in readme file I got following error

Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"

Update

Turns out the path to chrome is null for non lambda environment.
I am using docker image which is similar to aws lambda. Is there a way to get the path to the chrome installed by chrome-aws-lambda?

Navigation failed because browser has disconnected

Hello there, I am getting the below error when executing your sample code.
{
"errorMessage": "Navigation failed because browser has disconnected!",
"errorType": "Error",
"stackTrace": [
"Connection.NavigatorWatcher._eventListeners.helper.addEventListener (/var/task/node_modules/puppeteer-core/lib/FrameManager.js:1146:117)",
"emitNone (events.js:111:20)",
"Connection.emit (events.js:208:7)",
"Connection._onClose (/var/task/node_modules/puppeteer-core/lib/Connection.js:123:10)",
"WebSocketTransport._ws.addEventListener.event (/var/task/node_modules/puppeteer-core/lib/WebSocketTransport.js:45:22)",
"WebSocket.onClose (/var/task/node_modules/ws/lib/event-target.js:124:16)",
"emitTwo (events.js:126:13)",
"WebSocket.emit (events.js:214:7)",
"WebSocket.emitClose (/var/task/node_modules/ws/lib/websocket.js:172:10)",
"Socket.socketOnClose (/var/task/node_modules/ws/lib/websocket.js:781:15)",
" -- ASYNC --",
"Frame. (/var/task/node_modules/puppeteer-core/lib/helper.js:144:27)",
"Page.goto (/var/task/node_modules/puppeteer-core/lib/Page.js:579:49)",
"Page. (/var/task/node_modules/puppeteer-core/lib/helper.js:145:23)",
"exports.handler (/var/task/index.js:17:20)",
"",
"process._tickDomainCallback (internal/process/next_tick.js:228:7)"
]
}

Layer Deployment Issue

When attempting to run the sample function code I get an error, verified correct runtime environment of 8.10. When I look in the nodejs folder of the ZIP file created below I only see chrome-aws-lambda...is that possibly the issue?

Creating ZIP file and uploaded as a layer

ubuntu@ip-172-30-0-235:~/chrome-aws-lambda$ git clone https://github.com/alixaxel/chrome-aws-lambda.git && cd chrome-aws-lambda && npm pack && mkdir --parents nodejs/node_modules/chrome-aws-lambda/ && tar --directory nodejs/node_modules/chrome-aws-lambda/ --extract --file chrome-aws-lambda-*.tgz --strip-components=1 && rm chrome-aws-lambda-*.tgz && zip -9 --filesync --move --recurse-paths _/chrome-aws-lambda.zip nodejs/
Cloning into 'chrome-aws-lambda'...
remote: Enumerating objects: 74, done.
remote: Counting objects: 100% (74/74), done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 370 (delta 32), reused 55 (delta 19), pack-reused 296
Receiving objects: 100% (370/370), 417.87 MiB | 45.70 MiB/s, done.
Resolving deltas: 100% (163/163), done.
npm notice 
npm notice 📦  [email protected]
npm notice === Tarball Contents === 
npm notice 1.2kB   package.json                            
npm notice 1.1kB   LICENSE                                 
npm notice 10.0kB  README.md                               
npm notice 35.9MB  bin/chromium-72.0.3617.0.br             
npm notice 800.6kB source/iltorb/build/bindings/iltorb.node
npm notice 5.4kB   source/iltorb/index.js                  
npm notice 1.2kB   source/iltorb/LICENSE                   
npm notice 399B    source/index.d.ts                       
npm notice 4.5kB   source/index.js                         
npm notice 5.8kB   source/puppeteer/lib/FrameManager.js    
npm notice 6.6kB   source/puppeteer/lib/Page.js            
npm notice === Tarball Details === 
npm notice name:          chrome-aws-lambda                       
npm notice version:       1.11.2                                  
npm notice filename:      chrome-aws-lambda-1.11.2.tgz            
npm notice package size:  36.2 MB                                 
npm notice unpacked size: 36.7 MB                                 
npm notice shasum:        6614089bdd971555f0d3d35fad006ed7fadb8947
npm notice integrity:     sha512-DADk4SoH9Jpl6[...]tixDDGN0cOjdg==
npm notice total files:   11                                      
npm notice 
chrome-aws-lambda-1.11.2.tgz
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/package.json (deflated 56%)
  adding: nodejs/node_modules/chrome-aws-lambda/bin/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/bin/chromium-72.0.3617.0.br (deflated 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/index.js (deflated 63%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/iltorb/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/iltorb/build/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/iltorb/build/bindings/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/iltorb/build/bindings/iltorb.node (deflated 55%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/iltorb/index.js (deflated 81%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/iltorb/LICENSE (deflated 42%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/index.d.ts (deflated 46%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/puppeteer/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/puppeteer/lib/ (stored 0%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/puppeteer/lib/FrameManager.js (deflated 74%)
  adding: nodejs/node_modules/chrome-aws-lambda/source/puppeteer/lib/Page.js (deflated 73%)
  adding: nodejs/node_modules/chrome-aws-lambda/README.md (deflated 68%)
  adding: nodejs/node_modules/chrome-aws-lambda/LICENSE (deflated 41%)

Run sample function with layer in AWS Lambda
Response:
{
"errorMessage": "Cannot find module 'puppeteer-core'",
"errorType": "Error",
"stackTrace": [
"Function.Module._load (module.js:474:25)",
"Module.require (module.js:596:17)",
"require (internal/module.js:11:18)",
"Object. (/var/task/index.js:2:19)",
"Module._compile (module.js:652:30)",
"Object.Module._extensions..js (module.js:663:10)",
"Module.load (module.js:565:32)",
"tryModuleLoad (module.js:505:12)",
"Function.Module._load (module.js:497:3)"
]
}

node10.x

Currently does not support node10.x

slow as hell

the same function takes less than 4 seconds locally

yet even with using layers, it takes 12 seconds. 3X slowdown when deployed on AWS Lambda.

Bitbucket + lambda

So wondering how you all manage this when you run a bitbucket pipeline before deploying to lambda. We do our initial testing using bitbucket on a docker image and this package only works on lambda?

I installed this and then added the sample code but then I get:

Detected an unhandled promise rejection Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"

and the chromium-76.0.3803.0.br is there.. so I assume somehow this only runs on lambda?

any suggestions?


(async () => {
  console.log(`loading chrome `);
  const chromium = require('chrome-aws-lambda');
  browser = await chromium.puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath,
    headless: chromium.headless,
  });

  let page = await browser.newPage();

  await page.goto(event.url || 'https://example.com');

  result = await page.title();
  console.log(`result `, result);
})();


Issue running with webpack + typescript

I'm currently having trouble getting this running with webpack + typescript with serverless framework.

It seems to be having issues with the including iltorb binary file.

Example error on webpack build
Screenshot 2019-03-27 at 17 22 32

Let me know what additional info you need.

Dynamic Injection of Fonts

Explore the idea of providing an additional method to this package that will take a URL to a font and:

  • download it once into the appropriate directory
  • patch the environment variables so that Chromium knows where to look for it

Would address #4 and #16 in a consistent and easy way.

Included build has screenshot problems

I first noticed after I deployed to lambda (using the layer generated from the commands in the docs) that my screenshots were blank at the bottom. Here is what it should look like vs what it looked like using a different chromium build of the same version:

Included Chromium Rebuilt Chromium
image image

So I used this script at https://github.com/pahud/lambda-layer-headless-chromium to build the same version (linux dev 76.0.3786.0), compressed it, and overwrote the build included with this layer. Works okay now. Not sure what about the build is different or if it matters, but thought I would file an issue in case.

To reproduce, deploy the code attached at the bottom. Then attach the lambda layer generated from this repo.

To see the correct version, replace the chrome build in the layer with one built from https://github.com/pahud/lambda-layer-headless-chromium

serverless.yml

service: screenshot-problem-sample

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: us-east-1
  memorySize: 1600
  timeout: 30

package:
  exclude:
    - node_modules/puppeteer/.local-chromium/** # exclude puppeteer chrome if exists

functions:
  screenshot:
    handler: handler.screenshot
    events:
    - http:
        method: get
        path: screenshot

resources:
  Resources:
    ApiGatewayRestApi:
      Properties:
        BinaryMediaTypes:
          - "image/png"

handler.js

const chromium = require('chrome-aws-lambda');

module.exports.screenshot = async(event) => {
  const browser = await chromium.puppeteer.launch({
    args: chromium.args,
    executablePath: await chromium.executablePath,
    headless: chromium.headless,
    defaultViewport: { height: 900, width: 1440, }
  });

  const page = await browser.newPage();
  await page.goto('https://make.plated.com/home?utm_ad=50Px2', { waitUntil: 'networkidle0' });

  const $ele = await page.$('body');
  const { _, height } = await $ele.boundingBox();

  const capture = await page.screenshot({
    fullPage: false,
    encoding: 'base64',
    clip: {
      x: 0,
      y: 0,
      width: 1440,
      height: height
    },
  });

  await browser.close();

  return {
    statusCode: 200,
    body: capture,
    headers: {
      'Content-Type': 'image/png',
    },
    isBase64Encoded: true,
  };
};

package.json

{
  "name": "cal-repro",
  "version": "1.0.0",
  "description": "Reproduce screenshot issue.",
  "main": "handler.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Wes Painter",
  "license": "ISC",
  "devDependencies": {
    "chrome-aws-lambda": "^1.17.1"
  },
  "dependencies": {
    "serverless": "^1.45.1"
  }
}

puppeteer randomly fails after around 40 seconds

Thanks for providing the most recent binaries. Hugely appreciated. Located in Neukölln by the way ;)

I have an issue with the latest version of this module and puppeteer:

    "chrome-aws-lambda": "^1.12.1",
    "puppeteer": "^1.12.2",

After around 30 to 60 seconds in my puppeteer function, let page = await browser.newPage(); fails with an error:

2019-03-03T16:25:49.045Z	53f2d87e-153c-40b9-a64b-6502e7db59f7	{ Error: Protocol error (Page.createIsolatedWorld): Could not create isolated world
at Promise (/var/task/node_modules/puppeteer/lib/Connection.js:183:56)
at new Promise (<anonymous>)
at CDPSession.send (/var/task/node_modules/puppeteer/lib/Connection.js:182:12)
at Promise.all.frames.map.frame (/var/task/node_modules/puppeteer/lib/FrameManager.js:259:63)
at Array.map (<anonymous>)
at FrameManager._ensureIsolatedWorld (/var/task/node_modules/puppeteer/lib/FrameManager.js:259:37)
at <anonymous>
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
-- ASYNC --
at Target.<anonymous> (/var/task/node_modules/puppeteer/lib/helper.js:108:27)
at Browser._createPageInContext (/var/task/node_modules/puppeteer/lib/Browser.js:177:31)
at <anonymous>
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
-- ASYNC --
at Browser.<anonymous> (/var/task/node_modules/puppeteer/lib/helper.js:108:27)
at Object.somefunction (/var/task/node_modules/somemodule/index.js:68:38)
at <anonymous>
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
message: 'Protocol error (Page.createIsolatedWorld): Could not create isolated world' }

I use

memorySize: 1600
timeout: 200

for my AWS lambda settings. so memory and timeout cannot be an issue.

Any ideas? I think it must be puppeteer related or chrome flags related. I use the following flags to the puppeteer constructor (they are mostly what you use in this module):

            this.browser = await puppeteer.launch({
                args: pupArgs,
                defaultViewport: chromium.defaultViewport,
                executablePath: execPath,
                headless: chromium.headless,
                ignoreHTTPSErrors: true,
            });

            if (this.config.debug) {
                console.log(`browser ${execPath} successfully launched.`);
                console.log(pupArgs);
            }
let pupArgs=
[ '--disable-accelerated-2d-canvas',
     '--disable-background-timer-throttling',
     '--disable-breakpad',
     '--disable-client-side-phishing-detection',
     '--disable-cloud-import',
     '--disable-default-apps',
     '--disable-dev-shm-usage',
     '--disable-extensions',
     '--disable-gesture-typing',
     '--disable-gpu',
     '--disable-hang-monitor',
     '--disable-infobars',
     '--disable-notifications',
     '--disable-offer-store-unmasked-wallet-cards',
     '--disable-offer-upload-credit-cards',
     '--disable-popup-blocking',
     '--disable-print-preview',
     '--disable-prompt-on-repost',
     '--disable-setuid-sandbox',
     '--disable-software-rasterizer',
     '--disable-speech-api',
     '--disable-sync',
     '--disable-tab-for-desktop-share',
     '--disable-translate',
     '--disable-voice-input',
     '--disable-wake-on-wifi',
     '--enable-async-dns',
     '--enable-simple-cache-backend',
     '--enable-tcp-fast-open',
     '--hide-scrollbars',
     '--media-cache-size=33554432',
     '--metrics-recording-only',
     '--mute-audio',
     '--no-default-browser-check',
     '--no-first-run',
     '--no-pings',
     '--no-sandbox',
     '--no-zygote',
     '--password-store=basic',
     '--prerender-from-omnibox=disabled',
     '--use-mock-keychain',
     '--single-process',
     '--window-position=0,0',
     '--ignore-certifcate-errors',
     '--ignore-certifcate-errors-spki-list' ],

Flags for improving text rendering

Hello,

I noted some text rendering is inconsistent compared with doing the same but using a local process.

This is a screenshot done by my service running locally:

download (1)

and that's exactly the same but running at AWS Lambda using this project:

download

As you can appreciate, the text looks a bit spaced.

I saw this is a recurrent issue opened at puppeteer repository: puppeteer/puppeteer#2410

Based on the suggestions, I tried some multiple flags combinations that currently are not present as default args:

'--font-render-hinting=medium' // could be 'none', 'medium'
'--enable-font-antialiasing'

None of these combinations had a successful output yet, but happy if we can use to try different approaches in order to normalize how text is rendering 🙂

mkdir: illegal option -- -

I get the following error when running the npm command below:

juans-MacBook-Pro:lambda juan$ npm pack && mkdir --parents nodejs/node_modules/chrome-aws-lambda/ && tar --directory nodejs/node_modules/chrome-aws-lambda/ --extract --file chrome-aws-lambda-.tgz --strip-components=1 && rm chrome-aws-lambda-.tgz && zip -9 --filesync --move --recurse-paths _/chrome-aws-lambda.layer.zip nodejs/
npm notice
npm notice 📦 [email protected]
npm notice === Tarball Contents ===
npm notice 262B package.json
npm notice 598B lambda-1.0.0.tgz
npm notice === Tarball Details ===
npm notice name: lambda
npm notice version: 1.0.0
npm notice filename: lambda-1.0.0.tgz
npm notice package size: 947 B
npm notice unpacked size: 860 B
npm notice shasum: 29d89767c3965d5e7a6dab58e091adacab16544d
npm notice integrity: sha512-nPp0/FzFTECan[...]TAb3ALR/1roJA==
npm notice total files: 2
npm notice
lambda-1.0.0.tgz
mkdir: illegal option -- -
usage: mkdir [-pv] [-m mode] directory ...
juans-MacBook-Pro:lambda juan$

Consider providing shared AWS Lambda Layers please

Since you already provide instructions for creating a lambda layer (thank you!), how about uploading one public layer for everyone to use, and listing it at λ A curated list of awesome AWS Lambda Layers.

That would save everyone having to create and upload their own version (and especially Windows users).

I think one upload per release/tagged version should be enough (so ~once per month), anyone who needs daily updates could do it themselves.

GCP Cloud Functions Node10 environment does not anymore set FUNCTION_NAME env

GCP Cloud Functions Node10 environment does not anymore set FUNCTION_NAME environment variable (https://cloud.google.com/functions/docs/env-var)

This is used for setting correct launch parameters so the check should be replaced with FUNCTION_TARGET

Workaround for this is to set it by yourself when running gcloud functions deploy myfunc --runtime nodejs10 --memory 2048MB --trigger-http --set-env-vars=FUNCTION_NAME=myfunc

The current Chromium Binary doesn't work for AWS nodejs10.x runtime

First of all, a big thank you for your such amazing work for this project.

We've been using the Chromium Binary file from this project as AWS Lambda Layer for a while. It works flawlessly on runtime 8.10. However, since AWS just announced supports for Nodejs10.x. We didn't hesitate to upgrade our lambda functions and it ended up breaking our builds.

The error we are getting from local tests is error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory. And I also noticed that AWS changed to use Amazon Linux 2 instead of Amazon Linux as Nodejs 10 Lambda Operation System. That might change a lot of shared libs or dependencies. Please find more information with the link below:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

I just hope that this project can add supports for AWS nodejs10.x runtime in the near future.

Cheers,

Nick

ERR_CONNECTION_CLOSED when loading sample code

Here are the steps to reproduce, owe a drink if anyone can help me out.

  1. Create new node project with index file with sample code provided
    `const chromium = require('chrome-aws-lambda');
    const puppeteer = require('puppeteer-core');

exports.handler = async (event, context) => {
let result = null;
let browser = null;

try {
    browser = await puppeteer.launch({
        args: chromium.args,
        defaultViewport: chromium.defaultViewport,
        executablePath: await chromium.executablePath,
        headless: chromium.headless,
    });

    let page = await browser.newPage();

    await page.goto(event.url || 'https://example.com');

    result = await page.title();
} catch (error) {
    return context.fail(error);
} finally {
    if (browser !== null) {
        await browser.close();
    }
}

return context.succeed(result);

};
execute();`
2. run npm install for both "chrome-aws-lambda" and "puppeteer-core"
3. Zip the index.js and node_modules folder into s3 bucket
4. Create lambda function that references s3 bucket.
5. Create test event, get connection error

Here is the full error I am receiving
{ "errorMessage": "net::ERR_CONNECTION_CLOSED at https://example.com", "errorType": "Error", "stackTrace": [ "navigate (/var/task/node_modules/puppeteer-core/lib/FrameManager.js:101:37)", "<anonymous>", "process._tickDomainCallback (internal/process/next_tick.js:228:7)", " -- ASYNC --", "Frame.<anonymous> (/var/task/node_modules/puppeteer-core/lib/helper.js:108:27)", "Page.goto (/var/task/node_modules/puppeteer-core/lib/Page.js:662:49)", "Page.<anonymous> (/var/task/node_modules/puppeteer-core/lib/helper.js:109:23)", "exports.handler (/var/task/index.js:18:20)", "<anonymous>", "process._tickDomainCallback (internal/process/next_tick.js:228:7)" ] }

Error: Cannot find module 'iltorb' #33 [BUG]

When running on aws lambda through layer and using proxy I have this trace:

{
    "errorMessage": "Cannot find module 'iltorb'",
    "errorType": "Error",
    "stackTrace": [
        "Error: Cannot find module 'iltorb'",
        "    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:668:15)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:591:27)",
        "    at Module.require (internal/modules/cjs/loader.js:723:19)",
        "    at require (internal/modules/cjs/helpers.js:14:16)",
        "    at inflate (/Users/daolf/node_modules/chrome-aws-lambda/source/index.js:200:14)",
        "    at Function.get executablePath [as executablePath] (/Users/daolf/node_modules/chrome-aws-lambda/source/index.js:159:7)",
        "    at module.exports.hello (/Users/daolf/Code/ProjetPerso/scrapingninja/scrapingninja-lambda/lambda-js/handler_puppeteer.js:13:40)",
        "    at BbPromise (/usr/local/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:593:30)",
        "    at Promise._execute (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/debuggability.js:313:9)",
        "    at Promise._resolveFromExecutor (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:488:18)",
        "    at new Promise (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:79:10)",
        "    at AwsInvokeLocal.invokeLocalNodeJs (/usr/local/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:547:12)",
        "    at AwsInvokeLocal.invokeLocal (/usr/local/lib/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:151:19)",
        "    at AwsInvokeLocal.tryCatcher (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/util.js:16:23)",
        "    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:517:31)",
        "    at Promise._settlePromise (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:574:18)",
        "    at Promise._settlePromiseCtx (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/promise.js:611:10)",
        "    at _drainQueueStep (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:142:12)",
        "    at _drainQueue (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:131:9)",
        "    at Async._drainQueues (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:147:5)",
        "    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/serverless/node_modules/bluebird/js/release/async.js:17:14)",
        "    at processImmediate (internal/timers.js:443:21)",
        "    at process.topLevelDomainCallback (domain.js:136:23)"
    ]
}

I use proxy by adding --proxy-server=0.0.0.0:3128 to the args.

My version: 1.18.1

"Core" version

Would it be possible for you to create a "core" version of this package like puppeteer-core did or create a flag where the chromium binary isn't downloaded? We want to build our own AWS layer with the chromium version built in (extracting it from this package). This way we can include your package (without chromium pre-installed) in our package.json and still keep our bundle size small.

How do you generate these binaries?

Hi Alix!

Thanks a lot for your contributions to this project! I am able to use your binaries, but I am wondering how you are generating these binaries in the first place? Is there a chromium page where you can specify a configuration and it will generate the right chromium binary for you? I think this would be a helpful addition to the readme as it wont leave your users in the dark in the event it becomes hard for you to generate the binaries. Plus the community can help you out as well.

Also there is an opportunity for improvement in your readme documentation under the Lambda section where you can update the commands for uncompressing the .br binary and then zipping it up as a layer that can then be uploaded to lambda (currently it shows an example for tarballs).

Thanks,
Govind

Ansible version

Thanks for the great library, I was trying to build chromium myself and I'm having a few issues starting ansible due missing vpc, wrong values (e.g. in the Registering host) step.
I've seen latest ansible have different things like ec2_instance instead of ec2, other tasks are deprecated..
Which version of Ansible did you run this with? Just to hopefully make everything easier on my side

Blank Pdf

When attempting to capture a PDF using AWS lambda it is always coming back blank.
I have tried going to a url (page.goto) and also simply setting content (page.setContent).
I have also tried with and without: await page.emulateMedia('screen');

I can return the content of the page (page.content) but not create a PDF.

I am using nodejs8.10 runtime
Below is the code I am using and I have also attached the PDF returned, right now I am returning the PDF, later I will save this to S3 instead.

Any help would be greatly appreciated, thanks a lot

// index.js
const serverless = require('serverless-http');
const express = require('express');
const chromium = require('chrome-aws-lambda');
const puppeteer = require('puppeteer-core');
const app = express();

const main = async() => {
    const browser = await puppeteer.launch({
        args: chromium.args,
        defaultViewport: chromium.defaultViewport,
        executablePath: await chromium.executablePath,
        headless: true
    });

    const page = await browser.newPage();
    await page.goto('https://example.com', {waitUntil: 'networkidle2'});
    await page.emulateMedia('screen');
    const pdf = await page.pdf({format: 'A4', footerTemplate: '<div style="font-size:20px;">footer</div>'});
    // close page?
    return pdf;
};

app.get('/', function (req, res) {
    res.send('Hello World!');
});

app.post('/pdf', async function(req, res) {
    const pdf = await main();
    res.contentType("application/pdf");
    res.send(pdf);
});

module.exports.handler = serverless(app);

response.pdf

add a bin to create a layer?

the AWS layer instructions are convoluted and doesn't work on Windows, so what about creating a bin that could be executed from npm scripts section? like chrome-aws-lambda create-layer or just chrome-aws-lambda that puts the proper zip inside ./layer. this would automatically make it cross platform

Works on lambda but not SAM local

I'm able to convert some html to PDF and it works fine under lambda, but when I test on my machine using SAM local it gives:

Error:
Failed to load PDF document.

The code is:

try {
      browser = await Puppeteer.launch({
         args          : Chromium.args,
         executablePath: await Chromium.executablePath,
         headless      : Chromium.headless
      });

      const page = await browser.newPage();
      await page.goto(`data:text/html,${html}`, {waitUntil: 'networkidle0'});
      await page.pdf({path: "/tmp/file.pdf", format: 'A4'});
      await browser.close();

   } catch (error) {
      throw error;
   } finally {
      if (browser !== null) {
         await browser.close();
      }
   }

Any ideas why it might not be working? I don't see any obvious errors.

Guide to getting it to work

I've tried running:

npm pack &&
mkdir --parents nodejs/node_modules/chrome-aws-lambda/ &&
tar --directory nodejs/node_modules/chrome-aws-lambda/ --extract --file chrome-aws-lambda-.tgz --strip-components=1 &&
rm chrome-aws-lambda-
.tgz &&
zip -9 --filesync --move --recurse-paths _/chrome-aws-lambda.layer.zip nodejs/

on AWS linux AMI, Ubuntu, and Mac OS and cannot get this work. Likely due to my understanding of node but really love the idea of the project and would like to implement!

Not working on AWS Lambda with Node 8 + Layers

Hi, we're in the process of migrating from puppeteer-lambda to this framework, but cannot seem to get it to work. We followed the lambda layer instructions and uploaded the zip to the layer, then attached the layer to our lambda. When using the lib and attempting to spin up a browser we get

Error: Chromium revision is not downloaded. Run \"npm install\" or \"yarn install\"\n at Launcher.launch (/opt/nodejs/node_modules/puppeteer-core/lib/Launcher.js:120:15)\n at "} 

Not sure what we're missing. Commands followed to create layer zip:

git clone --depth=1 https://github.com/alixaxel/chrome-aws-lambda.git && \
cd chrome-aws-lambda && \
make chrome_aws_lambda.zip

Zip is then directly uploaded to layer and layer is attached to lambda. Nothing in the lambda codebase is different other than this:

import * as chromium from "chrome-aws-lambda";

Support local execution for testing by default

It's nice to be able to test your Lambda function in a local, Lambda-like environment as you're developing.

However, currently there's a check for whether or not you're running in Lambda (https://github.com/alixaxel/chrome-aws-lambda/blob/master/source/index.js#L91), and you get the following error when you try to run locally:

Error: Chromium revision is not downloaded. Run "npm install" or "yarn install"

Just faking out the check for headless (doing export AWS_LAMBDA_FUNCTION_NAME=myLambdaFunc or process.env.AWS_LAMBDA_FUNCTION_NAME = 'myLambdaFunc') makes everything work nicely, assuming you're running on a Lambda-like host environment.

Thoughts on just removing the check for headless?

Defaults chrome to USA time

Just want to apologise if I am missing any information.

Basically, I am trying to scrape a website that loads new data every day. The issue is that it seems to be giving me the data from the day before.

await page.evaluate(() => document.body.innerHTML);

The result of the command above gives me the data from the day before, instead of my current day.
I live in australia, but end up getting the previous day's data until its 12 am in USA.

To check that if I was wrong, I installed the full version of puppeteer and did a test. It got me the updated data for my current location. I was just curious to know if there is a way to fix the locale or region in the chrome bundled. I would be more than happy to help.

WebGL support

I'm attempting to use this binary to create PDFs from pages that include WebGL canvas elements created by Mapbox. Basically I'm wondering if this should this work out of the box with the default flags, or different set of flags and/or a custom built binary will be necessary(to enable something like swiftshader).

With version 1.14.0 and the default args exposed by this package, and visiting somewhere like:
https://docs.mapbox.com/mapbox-gl-js/example/simple-map/
or
http://mapbox.github.io/mapbox-gl-supported/diagnostics.html
and attempting to generate a PDF results in blank spots in the PDF.

If this is something you think should be supported, then I can provide a more fleshed out reproducer.

Is puppeteer-core / puppeteer installed with chrome-aws-lambda

Hello,

Thanks for your work on this, saved me a lot of time already :)

Im using chrome-aws-lambda in a layer, i was wondering if I have to install puppeteer-core separately?

If i dont install it separately on my lambda, it doesnt find puppeteer-core, but i can see it in chrome-aws-lambda dependencies, so i was just wondering if im doing something wrong.

My concern is that those dependencies can desynchronize and as you probably know, any puppeteer version if guaranteed to work only with certain chromium version (more on that here: https://github.com/GoogleChrome/puppeteer#q-why-doesnt-puppeteer-vxxx-work-with-chromium-vyyy ) - thats why im trying to invent a way to avoid breaking it when i update one and forget about the other.

Thanks,
Pawel

v1.17.0 throws 'Printing is not enabled' rendering PDFs

On v1.17.0, a call to page.pdf fails with Protocol error (Page.printToPDF): Printing is not enabled, which I think is due to d5b969b#diff-a562799028e672851e8b32af90205952R588

Works fine on v1.16.0.

Code that reproduces it:

browser = await puppeteer.launch({
        args: [
          '--headless',
          '--disable-gpu',
          '--unlimited-storage',
          '--no-sandbox',
          '--disable-setuid-sandbox',
          '--disable-dev-shm-usage',
          '--disable-extensions',
          '--single-process'
        ],
        defaultViewport: chromium.defaultViewport,
        executablePath: await chromium.executablePath,
        headless: true,
      });
const page = await browser.newPage();
await page.goto("http://example.com", {
        waitUntil: [
          'load'
        ],
        timeout: 0
      });
await page.pdf({
        printBackground: false,
        format: 'A4',
        displayHeaderFooter: false,
      });

Error: spawn EACCES

Hey there. Thanks for building this out...obviously it has been very helpful to a lot people :)

I am having an issue launching a browser both when attempting to launch locally and when deployed to Lambda.

My setup:
Node: v8.10

const puppeteer = require('puppeteer-core');
const chromium = require('chrome-aws-lambda');

const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath,
    headless: chromium.headless,
  });

The error message from Lambda console:

(node:1) UnhandledPromiseRejectionWarning: Error: spawn EACCES
    at _errnoException (util.js:1022:11)
    at ChildProcess.spawn (internal/child_process.js:323:11)
    at Object.exports.spawn (child_process.js:502:9)
    at Launcher.launch (/var/task/node_modules/puppeteer-core/lib/Launcher.js:126:40)
    at <anonymous>

I also got this error the first time I ran it locally but wasn't able to recreate:

Error --------------------------------------------------
 
  Cannot find module 'iltorb'

I've maxed out my memory allocation but still doesn't work. Any ideas?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.