Git Product home page Git Product logo

download's Introduction

download

Build Status hexdocs badge

Simply downloads remote file and stores it in the filesystem.

Download.from(url, options)

Documentation

Features

  • Small RAM consumption
  • Ability to limit downloaded file size
  • Uses httpoison

Installation

def deps do
  [{:download, "~> x.x.x"}]
end

Into mix.exs

def application do
  [applications: [:download]]
end

download's People

Contributors

asiniy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

download's Issues

Add a flag to follow_redirect for downloading certain AWS S3 files

Would it be possible to add a flag to follow_redirect for downloading certain AWS S3 files.

curl s3-url does a 302 on quite a few AWS URLs, so at this time, I have to make an HTTP request and then use the location value in the response header as the actual download URL with Download

Gracefully handle closed connections

I'm seeing this error if the upstream closes the connection (or times out?)

17:11:16.935 [error] Process #PID<0.4030.2> raised an exception
** (FunctionClauseError) no function clause matching in Download.handle_async_response_chunk/2
    (download) lib/download.ex:92: Download.handle_async_response_chunk(%HTTPoison.Error{id: #Reference<0.2630262677.3125280769.67801>, reason: {:closed, :timeout}}, %{controlling_pid: #PID<0.4028.2>, downloaded_content_length: 170004, file: #PID<0.4029.2>, max_file_size: 1048576000, path: "/tmp/foo"})
** (EXIT from #PID<0.91.0>) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Download.handle_async_response_chunk/2
        (download) lib/download.ex:92: Download.handle_async_response_chunk(%HTTPoison.Error{id: #Reference<0.2630262677.3125280769.67801>, reason: {:closed, :timeout}}, %{controlling_pid: #PID<0.4028.2>, downloaded_content_length: 170004, file: #PID<0.4029.2>, max_file_size: 1048576000, path: "/tmp/foo"})

About HTTPoison options

Is it possible to add an HTTPoison's options to the Download.from opts argument? E.g

options = [
  path: "./file1",
  http_opts: [
    headers: [
      "user-agent":  "XXX",
      "referer": "YYY"
    ]
  ]
]
Download.from(address, options)

The options[:http_opts] is HTTPoison's options.

Genserver implementation

I was using this with a genserver, and it didn't handle calls well while the downloader was active so I put together my own genserver implementation of your library. We could probably add a couple calls to make the API compatible with yours, if you were interested in entertaining a PR

defmodule Local.Downloader do
  use GenServer
  alias HTTPoison.{AsyncHeaders, AsyncStatus, AsyncChunk, AsyncEnd}

  def start_link(default) when is_map(default) do
    GenServer.start_link(__MODULE__, default)
  end

  @impl true
  def init(state) do
    {:ok, state}
  end

  def start_download(pid), do: GenServer.cast(pid, :start_download)

  @impl true
  def handle_cast(:start_download, %{url: url, path: path} = state) do
    with {:ok, file} <- do_start_download(url, path) do
      {:noreply, Map.merge(state, %{file: file, downloaded: 0})}
    end
  end

  @impl true
  def handle_cast(:status, %{downloaded: size, content_length: total} = state) do
    {:reply, size / total, state}
  end

  @impl true
  def handle_info(%AsyncHeaders{headers: headers}, state) do
    {_, content_length} = Enum.find(headers, fn({ header_name, _value }) ->
      header_name == "content-length" || header_name == "Content-Length"
    end)

    {:noreply, Map.put(state, :content_length, content_length)}
  end
  def handle_info(%AsyncStatus{code: 200}, state) do
    {:noreply, state}
  end
  def handle_info(%AsyncStatus{code: _}, state) do
    finish_download({:error, :unexpected_status_code}, state)
    {:noreply, state}
  end
  def handle_info(%AsyncChunk{chunk: data}, %{downloaded: size, file: file} = state) do
    IO.binwrite(file, data)
    {:noreply, Map.put(state, :downloaded, size + byte_size(data))}
  end
  def handle_info(%AsyncEnd{}, state) do
    finish_download({:ok, state})
    {:stop, :normal, state}
  end

  defp do_start_download(url, path) do
    with {:ok, file} <- create_file(path),
         {:ok, _result} <- HTTPoison.get(url, %{}, stream_to: self()) do
      {:ok, file}
    else
      {:error, _reason} -> File.rm!(path)
    end
  end

  defp finish_download({:error, _}, %{path: path}), do: File.rm!(path)
  defp finish_download({:ok, %{notify: pid}}) do
    send(pid, {:download_complete})
  end

  defp create_file(path) do
    if File.exists?(path), do: File.rm(path)

    with {:ok, file} <- File.open(path, [:write, :exclusive]) do
      {:ok, file}
    end
  end
end

Option to specify user agent

Thanks for the helpful mod!
Is it possible to have some options passed to HTTPoison, for example headers like user agent name.

Feature request: add `from!` function returning just the path of the downloaded file

Thanks for writing this module. I was considering writing such mechanism myself but I was delighted to find out I don't need to shave that yak :)

Currently I have file download as a fist stage in a pipe, passing the downloaded file path to the next stage. I suspect it could be a common pattern much like HTTPoison.request/5 vs HTTPoison.request!/5, so it may be worth adding to your module.

I would happily supply a PR but I'm about to leave for computer-free holidays :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.