Downloads fail if connection is broken about radiant-mlhub HOT 2 CLOSED

@karimmamer @michaeltrs I believe you both were experiencing this issue. I'll add a comment with the proposed solution below. Let me know if you have any additional detail on the problem or feedback on the solution.

from radiant-mlhub.

Proposed solution is 2-fold:

1. Allow users to resume downloads instead of always starting from scratch

   We could change the signature of the client.download_archive function to the following:

   def download_archive(
       archive_id: str,
       output_dir: Path = None,
       *,
       if_exists: Literal['skip', 'overwrite', 'resume'] = 'resume',
       **session_kwargs
   ) -> Path:

   The default behavior would be to resume the download, but users would have the option to overwrite the existing file or skip it if it exists. If resuming a download, the client would check the size of the existing file against the Content-Length heading for the download. If they are the same, then it would skip the download. If the file size is less than the Content-Length of the download then the client would make a Range request for just the remaining bytes and append them to the existing file. If the existing file size is greater than the Content-Length it would raise an exception (this shouldn't happen unless the remote archive changed).

2. Automatically retry failed requests from broken connections

   The client would automatically retry any requests that fail due to connection errors up to a specified maximum number of retries. If a retry is successful, then the attempts would reset to 0. This would ensure that larger files wouldn't ultimately fail just because they have a better chance of having more failures over the life of the entire download.

   I'm thinking we will implement this at the requests.Session level so we can take advantage of it for all requests.

from radiant-mlhub.