allow specification of multiple downloads at once about datalad HOT 5 CLOSED

I am not sure what change this needs on the git-annex side. You can already run multiple git annex get commands in parallel.

from datalad.

sorry for not being clear... idea is not to request downloads in parallel independently (e.g. parallel runs of wget requesting different files) but rather e.g. to provide full list of files to be 'get'ed at once which would then serve nicely requests like 'get files X Y Z from archive blah.tar.gz'. This would differ from running 3 independent processes in parallel.
Also I have no clue (yet) on how such request should be specified since those files all might need specification of the destination path/filename, e.g. I do not see anything in tar cmdline to allow extraction of multiple files into arbitrary destination locations.

from datalad.

Ok, sounds more like caching resources so they can be reused for multiple transfers.

So, there's potentially some overlap with the resource management I recently added to git-annex to allow reusing of eg, http connections when downloading multiple chunks of a chunked key.

Expanding that to support reusing connections (or reusing a downloaded tarball in your example) when downloading multiple keys needs a solution to the question: How long should the cached resource be kept around? Certianly only until the end of the git annex get command, but ideally less time than that; if a lot of files are being transferred we want to be able to examine the set of transfers and reorder ones that can reuse the same resources etc. There's a tension here with wanting git annex get to still start the first transfer promptly as it does now, and not need to buffer a great many transfers in memory.

from datalad.

well:

• git annex supports parallel downloads now with -J switch, so kinda "solved" on annex side (removing git-annex label)
• there is an outstanding issue to debug/fix for requesting multiple files from the same archive (#451)
• our install commands allows for multiple targets for installation ATM, and the rest of the logic on analysis of what should be the most efficient 'annex get' operations would be is TODO. See https://git-annex.branchable.com/todo/wishlist__58___--dry-run_option_for_all_commands/ and the particular command would be git annex find --not --in here -j [paths] which would return in json records also the keys in question

from datalad.

I think it was largely solved, not clear what else we should possibly do here, thus closing

from datalad.