Also add some real tests.

The following currently dumps a binary PNG to the user's terminal:

shot-scraper https://datasette.io/

It is vanishingly rare for users to want this to happen!

Instead, the default should imitate wget - if no -o filename is specified, make one up based on the URL - and avoid overwriting an existing file with the same name by adding a .1.

If the user really does want it to write binary to standard output they can specify that with -o -.

https://playwright.dev/python/docs/intro may be a better option than mucking around with npx, see:

• #1 (comment)

Selecting elements as the target for screenshots based on CSS selectors does not (currently) allow for the selection of elements based on tag content, or relative to DOM elements selected based on tag content.

However, elements can be selected based on tag content using Javascript. It would be useful to allow for the selection of elements via Javascript as well as CSS.

Alternatively, support a method that can be called from a javascript scraper call that will apply a screen shot to a Javascript selected element.

At the moment, selector based screenshots seem to be focused in _selector_javascript(selectors) by:

As well as passing --selector(s), s, one approach might be to pass element(s) el returned from a --js-selector script?

A repository template that helps users create a repo that takes screenshots of a page.

Create your own template from the repo and it will give you a YAML file that you can then edit - it writes screenshots to the same repository.

On my Retina MacBook Pro this image looks pretty fuzzy:

It looks like https://playwright.dev/python/docs/api/class-browser#browser-new-page supports doing this:

page = browser.new_page(device_scale_factor=2)

While building this I realized that for ongoing git scraping projects file size matters, and so an option for lower quality JPEGs would be good:

• #14

Here's a recent call that hung for 360 minutes before being killed by GitHub. I'm unclear of the cause.

From https://datasette.io/tutorials/explore

This is a really tough one: need to execute JavaScript to open that menu, then add the pink arrows to highlight specific elements!

Refs:

• #9

This is a bit of an out-there idea: what if you could execute custom JavaScript that returned a result, and then write that result to disk?

You could even skip the screenshot entirely and use this as a generic scraping tool at that point.

Bonus: if it can affect the exit code in some way it could be used as part of a CI flow to test something.

A repository template that helps users create a repo that takes screenshots of a page.

Create your own template from the repo and it will give you a YAML file that you can then edit - it writes screenshots to the same repository.

Feature suggestion from https://writethedocs.slack.com/archives/C72NZ18FR/p1646878775398999?thread_ts=1646877544.991549&cid=C72NZ18FR

Tried this, works great! One issue I found is that if the CSS selector is used and the container is taller than the viewport, it seems to cut it.

https://twitter.com/journeyer/status/1501772311715717126

For an .html file stored locally I use python -m http.server 80 and then shot-scraper http://localhost/file but it would be handy to be able to just point shot-scraper to the file directly.

This is a bit of an unconventional need, but I think it's worth exploring.

When taking screenshots for tutorials, I often want to grab an area of the screen that incorporates more than one element - where there's no convenient wrapper element that I can use to get the shot that I want.

Imagine if you could specify multiple selectors and get back a screenshot of the smallest area of the screen that incorporates all of those elements.

The implement would look at the founding box of all of those elements, generate a new box that wraps all of those, inject an absolutely positioned box of that size and take the screenshot of that area.

It could even optionally add some padding to that box before taking the shot.

Refs:

• #4

If not provided it can derive a filename using the same approach as:

• #32

Suggestion from Ben Pickles: https://twitter.com/benpickles/status/1507391958343471111

@simonw I’m using headless Chrome in an HTTP screenshot service, my favourite feature is being able to force prefers-reduced-motion=reduce mode which can be used as a standards-based hint to prevent animations (useful when taking a screenshot)

Playwright documentation: https://playwright.dev/python/docs/api/class-browser#browser-new-context-option-reduced-motion

browser.new_context(..., reduced_motion="reduce")

Since PDF will accept a different set of options from regular screenshots I think this will be the point at which I add much more robust validation of the YAML mini-language, probably using https://pydantic-docs.helpmanual.io/

Originally posted by @simonw in #27 (comment)

I missed this when I implemented:

• #35

So that you can comment out everything in that file and not see this error:

% shot-scraper multi empty.yml
...
    for shot in shots:
TypeError: 'NoneType' object is not iterable

Currently all shots that don't specify a selector are the full page, with no way to take a partial shot - that needs to change.

Also, the browser defaults to 1280px wide - this should be customizable.

Goal here is to be able to take a screenshot of a specific component of the page, identified by a CSS selector.

Should work for both multi and single shot mode.

Missed this when I implemented:

• #38
  In 2cd70b0

I want to be able to annotate pages before applying screenshots - I often add pink arrows to things when I screenshot them manually, for example in https://datasette.io/tutorials/explore

These annotations should be defined in the YAML or as command line options to the shot command.

• --format - one of Letter, Legal, Tabloid, Ledger, A0, A1, A2, A3, A4, A5, A6
• --print-background - include backgrounds
• --scale - from 0.1 to 2
• --width - measurement with units e.g. 10cm
• --height - ditto

Split from:

• #24

I'd like it to be able to do a few different things, so I'm going to break it up into sub-commands. Need to design those first.

Cool project, thanks

I was able to get DrudgeReport and ZeroHedge, captured both

Then I added LA Times using the sample code provided in the readme, however it hung and needed to be killed (would have been 3 shots)

I rearranged the order and it would still hang after two shots

Removed the first two and it worked, getting 3 shots. But being able to get a long list of shots seems not consistent.

Not sure how to debug, but feel free to review any logs
https://github.com/coding-to-music/page-snapshot-github-actions-shot-scraper/actions

A live demo running in a GitHub repo to show what a daily scrape could look like.

This is just a harebrained idea at the moment: what if the YAML could define JavaScript that could be applied to all of the shots in the file, with customizations for each shot?

Something like this:

macros:
- javascript: |
    Array.from(
      document.querySelectorAll(
        ShotScraper.inputs.toRemove
      )
    ).forEach(el => el.style.display='none')
shots:
- url: https://site-one.com/
  output: one.png
  inputs:
    toRemove: "#ad,#footer"
- url: https://site-two.com/
  output: two.png
  inputs:
    toRemove: "#adbanner"

Refs:

• #4

The auth command could take the standard --javascript option.

It could also take a --headless option - which causes it to open the page in regular headless mode, execute that JavaScript and then close the page again.

This would allow you to automate logins on pages that support it:

shot-scraper auth https://site.local/ auth.json --headless --javascript "
document.getElementById('#username').value = 'username';
document.getElementById('#password').value = 'password';
document.forms[0].submit();
"

Originally posted by @simonw in #18 (comment)

For consistency with how shot-scraper shot works.

Related:

• #23

This is the starting point for fancy features like the arrow annotations described in these two issues:

• #9
• #10

I added PDF support here:

• #24

But there's currently no way to script PDFs in the YAML syntax with shot-scraper multi. This would be really useful, especially as the PDF generation grows even more options:

• #26

Hello,

I got this error:

Traceback (most recent call last): File "/usr/bin/shot-scraper", line 33, in <module> sys.exit(load_entry_point('shot-scraper==0.10', 'console_scripts', 'shot-scraper')()) File "/usr/bin/shot-scraper", line 22, in importlib_load_entry_point for entry_point in distribution(dist_name).entry_points File "/usr/lib/python3.9/importlib/metadata.py", line 524, in distribution return Distribution.from_name(distribution_name) File "/usr/lib/python3.9/importlib/metadata.py", line 187, in from_name raise PackageNotFoundError(name) importlib.metadata.PackageNotFoundError: shot-scraper

is it not working for the pi?

thanks
Carsten

The initial prototype of this will involve a YAML file that defines multiple screenshots to be taken, and a GitHub Actions workflow that takes those screenshots.

Playwright can do this, so it's not too hard to get shot-scraper to do this too.

Via Twitter I found out that Playwright has a utility function for dumping out the accessibility tree for a page: https://playwright.dev/python/docs/api/class-accessibility

I'm going to add this as a feature of shot-scraper.

It looks more actively maintained:

• https://www.npmjs.com/package/playwright

Refs:

• #1

Describe the bug
When running shot-scraper multi shots.yml if one of the sites times out, the cli stops at that site instead of continuing on

shots.yml

- output: w3c.org.png
  url: https://www.w3.org/
- output: cnn.com.png
  url: http://www.cnn.com/
- output: example.com.png
  url: http://www.example.com/

❯ shot-scraper multi shots.yml
Screenshot of 'https://www.w3.org/' written to 'w3c.org.png'
Traceback (most recent call last):
  File "/Users/ryan/.local/bin/shot-scraper", line 8, in <module>
    sys.exit(cli())
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/shot_scraper/cli.py", line 208, in multi
    take_shot(context, shot)
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/shot_scraper/cli.py", line 469, in take_shot
    page.goto(url)
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 7345, in goto
    self._sync(
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 88, in _sync
    return task.result()
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_page.py", line 487, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_frame.py", line 122, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 39, in send
    return await self.inner_send(method, params, False)
  File "/Users/ryan/.local/pipx/venvs/shot-scraper/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 63, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
navigating to "http://www.cnn.com/", waiting until "load"
============================================================

To Reproduce
Steps to reproduce the behavior:

1. Create a file shots.yml

- output: w3c.org.png
  url: https://www.w3.org/
- output: cnn.com.png
  url: http://www.cnn.com/
- output: example.com.png
  url: http://www.example.com/

2. run shot-scraper multi shots.yml
3. See that the second site causes an error and does not move onto the third site

Expected behavior
If a timeout occurs, the tool should output an error has occurred and move onto the next site to process, something like this:

Screenshot of 'https://www.w3.org/' written to 'w3c.org.png'
Timeout 30000ms exceeded.
=========================== logs ===========================
navigating to "http://www.cnn.com/", waiting until "load"
============================================================
Screenshot of 'http://www.cnn.com/' written to 'cnn.com.png'
Screenshot of 'http://www.example.com/' written to 'example.com.png'

Additional context
This may be a duplicate of #48, or perhaps just related

Following #47:

 % shot-scraper https://www.cnn.com/ -h 800 --timeout 5
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/bin/shot-scraper", line 33, in <module>
    sys.exit(load_entry_point('shot-scraper', 'console_scripts', 'shot-scraper')())
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/shot-scraper/shot_scraper/cli.py", line 190, in shot
    shot = take_shot(context, shot, use_existing_page=use_existing_page)
  File "/Users/simon/Dropbox/Development/shot-scraper/shot_scraper/cli.py", line 576, in take_shot
    page.screenshot(**screenshot_args)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/sync_api/_generated.py", line 7886, in screenshot
    self._sync(
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_sync_base.py", line 111, in _sync
    return task.result()
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_page.py", line 611, in screenshot
    encoded_binary = await self._channel.send("screenshot", params)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 39, in send
    return await self.inner_send(method, params, False)
  File "/Users/simon/.local/share/virtualenvs/shot-scraper-sQHOtKI2/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 63, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 5ms exceeded.

Could be useful here:

• palewire/news-homepages#5 (comment)

Options:

• --landscape to switch to landscape mode
• --media-screen for page.emulate_media(media="screen")

Had the idea in this Tweet

Yeah, if you want a high frequency you should absolutely run this on its own box

Might be value in supporting that directly in the tool, since it would save it having to launch a brand new Chromium instance each time if it stayed running

If you're scraping a frequently updating resource, it would be neat if you didn't have to instantiate an entirely new Chromium instance every time you ran the scraper.

The --interactive option is implied if you pass --devtools.

This pattern here: https://github.com/simonw/shot-scraper-demo/blob/e72b6b5960b9b9cd184c3c569fa04a0edcd63a59/shots.yml#L12-L70

It's a way to run custom JavaScript code and then wait a few seconds before taking the shot - to allow things like fading animations to finish running.

It's very useful but non obvious, so it should go in the documentation.

To make these more convenient to run locally:

If you could run JavaScript before dumping the snapshot you could do things like stripping out parts of the pages that aren't relevant, or clicking on menu items to open them before running the dump.

Related:

• #22
• #12

While working on #14 I found that all images hadn't yet loaded when a screenshot was taken, so I hacked around it like this:

shot-scraper 'https://www.owlsnearme.com/?place=127871' \
  --selector 'section.secondary' \
  --javascript 'new Promise(resolve => setTimeout(resolve, 2000))' \
  -o owls.jpg

Inspired by how auth works:

• #18

It would be cool if you could take a shot where the browser window and suchlike are set on the command-line, but the browser then opens such that you can further interact with the site before the image is taken. Then hit <enter> to take and save the image.