contentmine / scraperjson Goto Github PK

It would be nice to have real-world example-json files
together with directory/file-collection, which are created by
running a scraper with a certain scraperJSON-json file.

That would be helpful to implement a scraper that follows the scraperJSON scheme/policy.

A *.zip or *.tgz file for results (or json-file and results) would make sense as examples, IMHO.

See ContentMine/thresher#2

See ContentMine/thresher#3

A common problem when scraping scientific journal articles is if access to the PDF or other file downloads is not granted, but the server returns a 200 OK status and sends an HTML document telling the user they don't have access. In this case, a scraperJSON client will simply download the HTML page and may rename it to the user's specified filename, which leads to a confusing situation where an HTML document might be mislabelled as some other filetype.

A solution is to allow a download to specify one or more content-types that are permitted, or perhaps a regex that should match the content-type. If the content-type does not match, the download is skipped.

The client would implement this by performing a HEAD request to the download URL initially, then evaluating the Content-Type HTTP header, then deciding whether to proceed to full download.

We should use the model PeerLibrary has developed, in which they store essentially all the information available about the location of an annotation

Defined here: https://github.com/peerlibrary/peerlibrary/blob/1e27cc32ee93d4322afce356738c45f24205a729/client/lib/annotator/domanchors.coffee#L31-L50

Add ability to specify regex post-extraction and a way to decide how the captures are handled.

See ContentMine/quickscrape#12 for discussion.

Need to document this feature and update the spec

simple name key

this could be a part of the rename option, or could be its own option