Git Product home page Git Product logo

archiveteam-megawarc-factory's People

Contributors

alard avatar arkiver2 avatar chfoo avatar falconkirtaran avatar fusl avatar ghostofapacket avatar ivan avatar km09 avatar

Stargazers

avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

pombredanne hannahwhy falconkirtaran fusl tomglass kiska3

archiveteam-megawarc-factory's Issues

Exponential backoff when uploads fail

Rather than trying to jam the WARCs into IA, it should do something like exponential backoff to give IA time to recover and to reduce bandwidth usage.

Chunker fails with fast drives (and small megawarcs)

Chunker makes directories based on timestamp (accuracy of 1 second). With fast enough drives and enough backlog, it'll try to move multiple chunks to the same directory and fail due to it already existing.
For this project, we're using 10gb megawarcs

root@ns535835:/home/rsync/archiveteam-megawarc-factory# ./chunk-multiple
Wed Dec 13 21:52:03 UTC 2017
Moving /home/rsync/target/vidme/PoorHomie/vidme-video_16354827-20171213-211138.warc.gz
Moving /home/rsync/target/vidme/PoorHomie/vidme-video_16749903-20171213-210802.warc.gz
Moving /home/rsync/target/vidme/PoorHomie/vidme-video_16208604-20171213-210412.warc.gz
Current archive is full, moving to 20171213215204.
Moving /home/rsync/target/vidme/cloudfunnel/vidme-video_16918974-20171213-212127.warc.gz
Moving /home/rsync/target/vidme/BnAboyZ/vidme-video_16967625-20171213-213044.warc.gz
Moving /home/rsync/target/vidme/Kaz/vidme-video_16090968-20171213-213031.warc.gz
Moving /home/rsync/target/vidme/Aoede/vidme-video_16299663-20171213-221233.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16158030-20171213-210118.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16774371-20171213-212356.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16562241-20171213-210714.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16431138-20171213-211714.warc.gz
Current archive is full, moving to 20171213215204.
Moving /home/rsync/target/vidme/odemg/vidme-video_16612260-20171213-211355.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16500207-20171213-210850.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16983396-20171213-212036.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16698984-20171213-205807.warc.gz
Moving /home/rsync/target/vidme/odemg/vidme-video_16022853-20171213-210045.warc.gz
Moving /home/rsync/target/vidme/josho493/vidme-video_16005048-20171213-211747.warc.gz
Current archive is full, moving to 20171213215204.
mv: cannot move '/home/rsync/vidme-data/chunker-work/current' to '/home/rsync/vidme-data/packing-queue/20171213215204/current': Directory not empty
Moving /home/rsync/target/vidme/MIABIL/vidme-video_16914327-20171213-125653.warc.gz
Moving /home/rsync/target/vidme/CoolCanuk/vidme-video_16516200-20171213-205725.warc.gz
Moving /home/rsync/target/vidme/HCross/vidme-video_16436868-20171213-223135.warc.gz
Moving /home/rsync/target/vidme/bitspill/vidme-video_16928310-20171213-221416.warc.gz
Moving /home/rsync/target/vidme/bitspill/vidme-video_16851003-20171213-215947.warc.gz
Moving /home/rsync/target/vidme/vantec/vidme-video_16164156-20171213-223126.warc.gz
Current archive is full, moving to 20171213215204.
mv: cannot move '/home/rsync/vidme-data/chunker-work/current' to '/home/rsync/vidme-data/packing-queue/20171213215204/current': Directory not empty
Sleeping...

Name collisions in collections due to using timestamp on megawarc

The current chunker uses date +'%Y%m%d%H%M%S' to give each megawarc a unique identifier, which is ok as long as there is only one chunker running on one rsync target. Since the megawarc name is based on the current time, multiple rsync targets are proving to generate identically named megawarcs.

I'd propose we move to using uuid to create unique identifiers for each megawarc. The resulting megawarc names would therefore look like, i.e.:
googleplus_2545040a-447b-11e9-8f1f-2b4a7312c1da.megawarc.warc.gz

Thoughts? Is there any risk in using the dash in an object name?

This does add the dependency of the uuid tool, since it may not be installed on all distros by default. Edit: It does look like Linux makes version 4 uuids available via /proc/sys/kernel/random/uuid - I'm not aware of how portable that is.

This will also close #2 and #6.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.