Comments (2)
I like the idea.
Some suggestions:
- we should use the last available artifacts, not the one from last task (which might have produced no artifacts since already running from previous artifacts)
- we should find a way to place downloaded artifacts in a location which will not be considered for archiving at the end of the task, so that we do not archive again and again the same artifacts
- I would rather rename the boolean "Download latest artifacts before task execution" ("if available" is implicit but quite obvious)
A corollary of this feature is #940
from zimfarm.
This is in particular true for Zimit, where we have many scenarios were we want to relaunch the recipe without the burden (for both us and the upstream Web site) of the spider part.
That's an interesting question because zimit is not a regular scraper. It plays the role of a scraper but it's two tools run one after the other: a scraper (albeit for the WARC format) and a converter from WARC to ZIM.
It's because WARC is a standard format and it provides value to convert from it that we didn't plug into browsertrix to write ZIM content directly.
All scrapers purposely (performance) run in one pass and directly store transformed data in ZIM. Specific data are sent to our S3 Optimization cache to save on both spiding and processing time when there are good chances this data is static.
I don't see how this can be useful for non-zimit scrapers.
For zimit, I see how this feature can be useful but only for development: when would we want to re-create a ZIM from the same WARCs? When our converter code failed. Are there other use cases?
We've refused to use Zimfarm as a dev tool for now but adding the artifacts was a clear step towards itβ¦
from zimfarm.
Related Issues (20)
- Again \u0000 cannot be converted to text HOT 1
- Add capability to recreate a ZIM with updated metadata
- Add concept of scraper release channels HOT 1
- Simplify warehouse list HOT 4
- Zimfarm worker: add support for multiple outgoing IPs HOT 4
- Fix sotoki scraper configuration
- Watcher is not detecting updated files when SE is fixing issues HOT 3
- Seems impossible to specify a tag HOT 5
- Upload logs and artifacts even when task is cancelled
- Name resolution errors in StackExchange watcher
- Configuration of Zimfarm local instance HOT 1
- Installing monitoring of a task is failing HOT 2
- Workers: report real disk size (and CPU / RAM) HOT 7
- StackExchanger watcher regularly restarts after connection reset by peer errors HOT 1
- Is the worker incorrectly checking the space already consumed by the Zimfarm? HOT 8
- Collect zimfarm usage statistics
- Some Zimfarm task are never finishing while the process is obviously finished HOT 5
- Adapt Zimfarm to MWoffliner 1.14 HOT 3
- Ensure we report MiB/GiB/... values everywhere
- Migrate to MWoffliner 1.14 HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zimfarm.