julialang / downloads.jl Goto Github PK

View Code? Open in Web Editor NEW

89.0 89.0 35.0 296 KB

License: MIT License

Julia 100.00%

downloads.jl's People

Contributors

Stargazers

Watchers

downloads.jl's Issues

Julia 1.0

Can a stub version of this package be released for Julia 1.0 that just calls Base.download? Currently, for package tests it's not possible to make Downloads an [extra] dependency without errorring on Julia 1.0, so currently I just choose to live with the depwarn.

turn TCP_FASTOPEN back on if libcurl is new enough?

See #76. For libcurl ≥ 7.74 it should be safe to turn on TCP_FASTOPEN even for FTP requests.

Download hang

I think JuliaLang/Pkg.jl#2337 and JuliaLang/Pkg.jl#2287 is still happening and I think the problem was that the server didn't send data (e.g. #85). I can reliably reproduce the hang with this snippet:

#!/usr/bin/env julia

import Downloads

n_concurrent_dls = 4

const jobs = Channel{String}(n_concurrent_dls)

const urls = "https://github.com/JuliaBinaryWrappers/gmsh_jll.jl/releases/download/gmsh-v4.7.1+2/" .* [
    "gmsh.v4.7.1.aarch64-linux-gnu-cxx03.tar.gz",
    "gmsh.v4.7.1.aarch64-linux-gnu-cxx11.tar.gz",
    "gmsh.v4.7.1.aarch64-linux-musl-cxx03.tar.gz",
    "gmsh.v4.7.1.aarch64-linux-musl-cxx11.tar.gz",
    "gmsh.v4.7.1.armv7l-linux-gnueabihf-cxx03.tar.gz",
    "gmsh.v4.7.1.armv7l-linux-gnueabihf-cxx11.tar.gz",
    "gmsh.v4.7.1.armv7l-linux-musleabihf-cxx03.tar.gz",
    "gmsh.v4.7.1.armv7l-linux-musleabihf-cxx11.tar.gz",
    "gmsh.v4.7.1.i686-linux-gnu-cxx03.tar.gz",
    "gmsh.v4.7.1.i686-linux-gnu-cxx11.tar.gz",
    "gmsh.v4.7.1.i686-linux-musl-cxx03.tar.gz",
    "gmsh.v4.7.1.i686-linux-musl-cxx11.tar.gz",
    "gmsh.v4.7.1.i686-w64-mingw32-cxx03.tar.gz",
    "gmsh.v4.7.1.i686-w64-mingw32-cxx11.tar.gz",
    "gmsh.v4.7.1.powerpc64le-linux-gnu-cxx03.tar.gz",
    "gmsh.v4.7.1.powerpc64le-linux-gnu-cxx11.tar.gz",
    "gmsh.v4.7.1.x86_64-apple-darwin.tar.gz",
    "gmsh.v4.7.1.x86_64-linux-gnu-cxx03.tar.gz",
    "gmsh.v4.7.1.x86_64-linux-gnu-cxx11.tar.gz",
    "gmsh.v4.7.1.x86_64-linux-musl-cxx03.tar.gz",
    "gmsh.v4.7.1.x86_64-linux-musl-cxx11.tar.gz",
    "gmsh.v4.7.1.x86_64-unknown-freebsd.tar.gz",
    "gmsh.v4.7.1.x86_64-w64-mingw32-cxx03.tar.gz",
    "gmsh.v4.7.1.x86_64-w64-mingw32-cxx11.tar.gz",
]

producer = @async begin
    for i = 1:5
        for url in urls
            put!(jobs, url)
            put!(jobs, url)
            # put!(jobs, url)
            # put!(jobs, url)
        end
    end
    close(jobs)
end

@sync begin
    for i in 1:n_concurrent_dls
        @async begin
            for j in jobs
                @info "Starting downloading..." file=basename(j)
                Downloads.request(j; output=abspath(basename(j)), verbose=true)
                @info "Done downloading..." file=basename(j)
            end
        end
    end
end

which uses 4 concurrent downloads. Note that this is just hitting Amazon servers so this is unrelated to any PkgServer issues.

After a while it just hangs, and from curl verbose mode I see

* stopped the pause stream!
* stopped the pause stream!
* stopped the pause stream!
* stopped the pause stream!
* Connection #4 to host github-releases.githubusercontent.com left intact

and it hangs forever.

Interupting this gives:

* Connection #4 to host github-releases.githubusercontent.com left intact
^C
signal (2): Interrupt
in expression starting at /home/fredrik/dev/Downloads/run.jl:48
epoll_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
uv__io_poll at /workspace/srcdir/libuv/src/unix/linux-core.c:309
uv_run at /workspace/srcdir/libuv/src/unix/core.c:379
jl_task_get_next at /home/fredrik/julia16/src/partr.c:472
poptask at ./task.jl:755
wait at ./task.jl:763 [inlined]
wait at ./condition.jl:106
take_buffered at ./channels.jl:389
take! at ./channels.jl:383 [inlined]
iterate at ./channels.jl:465
#12 at ./task.jl:406
unknown function (ip: 0x7f31dd5b1d8c)
_jl_invoke at /home/fredrik/julia16/src/gf.c:2238 [inlined]
jl_apply_generic at /home/fredrik/julia16/src/gf.c:2420
jl_apply at /home/fredrik/julia16/src/julia.h:1703 [inlined]
start_task at /home/fredrik/julia16/src/task.c:839
unknown function (ip: (nil))
unknown function (ip: (nil))

as in the other issues linked above.

Might be related to #94, I created the snippet above in an attempt to reproduce the error there, but I can't. Elliot could though (with the same snippet) for some reason.

Another download hang

The following consistently hangs during the second download at around 35% (progress callback just there for visualizing the hang):

import Downloads

function download(url)
    println("Downloading $url")
    Downloads.download(url; progress=(total,now)->print("$(round(100*now/total, digits=1))% ($(Base.format_bytes(now))/$(Base.format_bytes(total)))          \r"))
    println()
end

download("https://julia-bb-buildcache.s3.amazonaws.com/832f0d52e696354cdd22425aa70db1f209d62c9d84626a7e48cf4ca0190cfd50/8ae9d6affc9d3c0045f6d90f7c972db11b17d2babf4cfb3a204af49dd5afcaee/x86_64-linux-gnu.tar.gz")
download("https://julia-bb-buildcache.s3.amazonaws.com/832f0d52e696354cdd22425aa70db1f209d62c9d84626a7e48cf4ca0190cfd50/8ae9d6affc9d3c0045f6d90f7c972db11b17d2babf4cfb3a204af49dd5afcaee/powerpc64le-linux-gnu.tar.gz")

It does not happen when only downloading a single file, and does not require concurrent downloads like #95. Tested on the latest backports-1.6, so it includes #96.

Downloads.curl(::String) or @to_request "..."

I posted in #helpdesk but might be better seen here.

I think it'd be useful to have a way to give Downloads a string that you'd normally give the shell.
Ie Downloads.curl("curl -X GET \"https://reactome.org/ContentService/data/pathway/R-HSA-5673001/containedEvents\" -H "accept: application/json" -o .data.json")

It'd be nice to not have to reorganize all the options and flags myself. Additionally a macro like @to_request to build the Downloads.request expr for me would be super useful.

Getting "ERROR: Operation too slow." and timeout=Inf does not help

I'm trying to download a .xls file from a slow-to-respond server (not mine).

https://utslappisiffror.naturvardsverket.se/Excel/Excel?pageType=SubstancePage&sid=999

When pasting in the url in a browser it takes a while ~30s but then I'm able to download the file.

When using Downloads.download(url, "foo.xls") I get the following

ERROR: Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds while requesting https://utslappisiffror.naturvardsverket.se/Excel/Excel?pageType=SubstancePage&sid=999
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:369
  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Curl/Curl.jl:64
  [3] #8
    @ ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:311 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)
    @ ArgTools ~/julia-1.7.1/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:112
  [5] #7

    @ ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:310 [inlined]
  [6] arg_read
    @ ~/julia-1.7.1/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:61 [inlined]
  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Nothing, verbose::Bool, throw::Bool, downloader::Nothing)
    @ Downloads ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:309
  [8] #3
    @ ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:222 [inlined]
  [9] open(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, String}, args::String; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:write,), Tuple{Bool}}})
    @ Base ./io.jl:330
 [10] arg_write(f::Function, arg::String)
    @ ArgTools ~/julia-1.7.1/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:86
 [11] #download#2
    @ ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:221 [inlined]
 [12] download(url::String, output::String)
    @ Downloads ~/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:221
 [13] top-level scope
    @ REPL[3]:1

I tried Downloads.download(url, "foo.xls"; timeout=Inf) but got the same result.

Expected behaviour

I would expect setting timeout=Inf would make it so that it would not fail because of slow response, but rather keep waiting...

System info

Ubuntu 20.04 (running under Win10pro WSL2)
julia-1.7.1

ctr_drbg_init failed

Not sure how to expect this to be handled, but saw it on CI:

Error in testset Downloads:
Error During Test at /Users/julia/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:184
  Got exception outside of a @test
  TaskFailedException
  
      nested task error: Failed - mbedTLS: ctr_drbg_init returned (-0x0034) CTR_DRBG - The entropy source failed while requesting https://httpbingo.julialang.org/delay/2?id=37
      Stacktrace:
        [1] (::Downloads.var"#9#17"{IOBuffer, Base.DevNull, Nothing, Vector{Pair{String, String}}, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
          @ Downloads ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:313
        [2] with_handle(f::Downloads.var"#9#17"{IOBuffer, Base.DevNull, Nothing, Vector{Pair{String, String}}, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
          @ Downloads.Curl ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/Curl.jl:43
        [3] #8
          @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:267 [inlined]
        [4] arg_write(f::Downloads.var"#8#16"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Nothing, Bool, Bool, String, Int64, Bool, Bool}, arg::IOBuffer)
          @ ArgTools ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:112
        [5] #7
          @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:266 [inlined]
        [6] arg_read
          @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:61 [inlined]
        [7] request(url::String; input::Nothing, output::IOBuffer, method::Nothing, headers::Vector{Pair{String, String}}, progress::Nothing, verbose::Bool, throw::Bool, downloader::Downloads.Downloader)
          @ Downloads ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:265
        [8] #3
          @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:182 [inlined]
        [9] arg_write(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Nothing, Bool, Downloads.Downloader, String}, arg::IOBuffer)
          @ ArgTools ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:112
       [10] #download#2
          @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:181 [inlined]
       [11] (::Main.Test99Main_Downloads.var"#2#3"{Base.Iterators.Pairs{Symbol, Downloads.Downloader, Tuple{Symbol}, NamedTuple{(:downloader,), Tuple{Downloads.Downloader}}}, String})(output::IOBuffer)
          @ Main.Test99Main_Downloads ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/setup.jl:12
       [12] sprint(::Function; context::Nothing, sizehint::Int64)
          @ Base ./strings/io.jl:105
       [13] sprint
          @ ./strings/io.jl:101 [inlined]
       [14] #download_body#1
          @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/setup.jl:11 [inlined]
       [15] #download_json#4
          @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/setup.jl:17 [inlined]
       [16] macro expansion
          @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:197 [inlined]
       [17] (::Main.Test99Main_Downloads.var"#15#29"{Downloads.Downloader, String, Int64})()
          @ Main.Test99Main_Downloads ./task.jl:395
  
  ...and 37 more exceptions.
  
  Stacktrace:
    [1] sync_end(c::Channel{Any})
      @ Base ./task.jl:353
    [2] macro expansion
      @ task.jl:372 [inlined]
    [3] macro expansion
      @ timing.jl:276 [inlined]
    [4] macro expansion
      @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:195 [inlined]
    [5] macro expansion
      @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1144 [inlined]
    [6] macro expansion
      @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:185 [inlined]
    [7] macro expansion
      @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1144 [inlined]
    [8] top-level scope
      @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:4
    [9] include
      @ ./Base.jl:386 [inlined]
   [10] macro expansion
      @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/test/testdefs.jl:24 [inlined]
   [11] macro expansion
      @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1144 [inlined]
   [12] macro expansion
      @ ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/test/testdefs.jl:23 [inlined]
   [13] macro expansion
      @ ./timing.jl:353 [inlined]
   [14] runtests(name::String, path::String, isolate::Bool; seed::UInt128)
      @ Main ~/buildbot/worker-tabularasa/tester_macos64/build/share/julia/test/testdefs.jl:21
   [15] (::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278
   [16] run_work_thunk(thunk::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:63
   [17] macro expansion
      @ ~/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Distributed/src/process_messages.jl:278 [inlined]
   [18] (::Distributed.var"#105#107"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:395

https://build.julialang.org/#/builders/79/builds/5443

Downloads.jl keeps file descriptors open after download

MWE is:

using Downloads 
path = mktempdir()
delay = 2
n = 30
const server = "https://httpbingo.julialang.org"
url = "$server/delay/$delay"
t = @elapsed @sync for id = 1:n
    @async begin
        file = joinpath(path, string(id))
        data = Downloads.download("$url?id=$id", file)
    end
end

count(contains("amazon"), split(read(`lsof -p $(getpid())`, String), '\n'))

showing that connections to amazon are still open. Increasing n increasing the number of open connections.

With the increase in concurrency in #25 this likely pushed things over the edge, causing other tests in the julia repo to fail. Tests for Downloads.jl is now disabled on Julia to allow CI to be green.

make sure that `put!(ch, buf)` is callback-safe

This line here is the potential concern:

https://github.com/StefanKarpinski/Downloader.jl/blob/37718937228e77ccea6e27fa132724a94d84e357/src/Curl/Easy.jl#L60

If this yields, then everything could crash. I haven't observed this, but @JeffBezanson indicated that it might do so due to locking (even though the channel has an infinite buffer). The consumer of buffers is here:

https://github.com/StefanKarpinski/Downloader.jl/blob/37718937228e77ccea6e27fa132724a94d84e357/src/Downloader.jl#L20-L22

I'm not set on using a channel, but it seemed to me like the way to do it. Jeff, do you have a safer alternative that I could use here?

Set `CURLOPT_NOBODY` automatically for `HEAD`s?

Downloads.jl/src/Downloads.jl

Line 336 in 99165c6

set_body(easy, have_output)

we decide whether or not to set CURLOPT_NOBODY based on whether or not the user specifies an output. On the one hand, that makes sense, and lets the caller decide whether or not to set CURLOPT_NOBODY in a simple and intuitive way. On the other hand, HEAD responses should reply with a Content-Length if the GET would but should not return a body; if additionally CURLOPT_NOBODY is not set, this this will cause hangs. Thus, I wonder if Downloads.jl should set CURLOPT_NOBODY for HEADs automatically.

xref JuliaCloud/AWS.jl@a481a46

Add .netrc & cookie support

Netrc files can store logins (username, password) for external websites, so you don't need to provide them in the url when accessing these websites. If used for an central authentication service, reading cookies in the current session is also required.

This setup is pretty common for accessing scientific datasets. For example: https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+cURL+And+Wget

I've made #98 to set these options as default. If this is unwarranted, can we discuss the design to support these as keyword arguments to the download function?

Curl.SYSTEM_SSL compares Curl.SSL_VERSION against a typo

The parameter Curl.SYSTEM_SSL will always evaluate to false on macOS.

It is defined as:

const SYSTEM_SSL =
    Sys.isapple() && startswith(SSL_VERSION, "SecureTranspart") ||
    Sys.iswindows() && startswith(SSL_VERSION, "Schannel")

Note that macOS compares against "SecureTranspart", when Curl.SSL_VERSION == "SecureTransport" (transpArt vs. transpOrt).

Is this intentional?

Minimal Reproducible Example

julia> versioninfo()
Julia Version 1.6.0
Commit f9720dc2eb (2021-03-24 12:55 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.6.0)
  CPU: Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

shell> pwd
/Users/livingston/playground/tmp/curl_issue/Downloads.jl/src

julia> using Base.Experimental: @sync

julia> using NetworkOptions; using ArgTools; include("Curl/Curl.jl"); using .Curl

julia> Curl.SYSTEM_SSL
false

julia> Sys.isapple()
true

julia> startswith(Curl.SSL_VERSION, "SecureTranspart")
false

julia> Curl.SSL_VERSION
"SecureTransport"

Possible typo

I'm out of my element here but shouldn't this line read

setopt(easy, CURLOPT_KEYPASSWD, key_pass)

instead?

add default connect timeout

We currently default to having speed limit timeouts, which keeps very slow connections from hanging forever, but if the server never accepts or rejects the connection, the download hangs, potentially forever. We don't want to add a fixed overall timeout because if you are downloading a large request, it might legitimately take a very long time. If we add a connect timeout of 30s or so, however, that should cover this situation.

failure detected on CI

https://build.julialang.org/#/builders/61/builds/3775/steps/5/logs/stdio

Error in testset Downloads:
Error During Test at D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\runtests.jl:71
  Got exception outside of a @test
  TaskFailedException
  
      nested task error: HTTP/2 502
      Stacktrace:
       [1] error(s::SubString{String})
         @ Base .\error.jl:33
       [2] download(url::String, io::IOBuffer; headers::Vector{Union{}}, downloader::Downloads.Downloader)
         @ Downloads D:\buildbot\worker\package_win32\build\usr\share\julia\stdlib\v1.6\Downloads\src\Downloads.jl:67
       [3] (::Main.Test65Main_Downloads.var"#1#2"{String, Vector{Union{}}})(io::IOBuffer)
         @ Main.Test65Main_Downloads D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\setup.jl:13
       [4] sprint(::Function; context::Nothing, sizehint::Int32)
         @ Base .\strings\io.jl:105
       [5] sprint
         @ .\strings\io.jl:101 [inlined]
       [6] download_body
         @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\setup.jl:12 [inlined]
       [7] download_json
         @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\setup.jl:18 [inlined]
       [8] macro expansion
         @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\runtests.jl:77 [inlined]
       [9] (::Main.Test65Main_Downloads.var"#7#12"{String, Int32})()
         @ Main.Test65Main_Downloads .\task.jl:392
  Stacktrace:
    [1] sync_end(c::Channel{Any})
      @ Base .\task.jl:350
    [2] top-level scope
      @ task.jl:369
    [3] top-level scope
      @ timing.jl:233
    [4] top-level scope
      @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\runtests.jl:75
    [5] top-level scope
      @ D:\buildbot\worker\package_win32\build\usr\share\julia\stdlib\v1.6\Test\src\Test.jl:1113
    [6] top-level scope
      @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\runtests.jl:72
    [7] top-level scope
      @ D:\buildbot\worker\package_win32\build\usr\share\julia\stdlib\v1.6\Test\src\Test.jl:1113
    [8] top-level scope
      @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.6\Downloads\test\runtests.jl:4
    [9] include(mod::Module, _path::String)
      @ Base .\Base.jl:389
   [10] include
      @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\test\testdefs.jl:13 [inlined]
   [11] macro expansion
      @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\test\testdefs.jl:25 [inlined]
   [12] macro expansion
      @ D:\buildbot\worker\package_win32\build\usr\share\julia\stdlib\v1.6\Test\src\Test.jl:1113 [inlined]
   [13] macro expansion
      @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\test\testdefs.jl:24 [inlined]
   [14] macro expansion
      @ .\timing.jl:310 [inlined]
   [15] top-level scope
      @ D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\test\testdefs.jl:22
   [16] eval
      @ .\boot.jl:360 [inlined]
   [17] runtests(name::String, path::String, isolate::Bool; seed::UInt128)
      @ Main D:\buildbot\worker-tabularasa\tester_win32\build\share\julia\test\testdefs.jl:28
   [18] (::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed D:\buildbot\worker\package_win32\build\usr\share\julia\stdlib\v1.6\Distributed\src\process_messages.jl:294
   [19] run_work_thunk(thunk::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed D:\buildbot\worker\package_win32\build\usr\share\julia\stdlib\v1.6\Distributed\src\process_messages.jl:79
   [20] macro expansion
      @ D:\buildbot\worker\package_win32\build\usr\share\julia\stdlib\v1.6\Distributed\src\process_messages.jl:294 [inlined]
   [21] (::Distributed.var"#105#107"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed .\task.jl:392

not clear from readme how to actually use headers kwarg

an example is definitely needed, one with AbstractVec and one with Dict

drip test failed

https://build.julialang.org/#/builders/7/builds/2723/steps/5/logs/stdio

Error During Test at /buildworker/worker/tester_linuxaarch64/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:410
  Got exception outside of a @test
  Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds while requesting https://httpbingo.org/drip
  Stacktrace:
    [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Main.Test39Main_Downloads.var"#22#40"{Vector{Any}}, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
      @ Downloads /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Downloads/src/Downloads.jl:369
    [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Main.Test39Main_Downloads.var"#22#40"{Vector{Any}}, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
      @ Downloads.Curl /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/Curl.jl:64
    [3] #8
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Downloads/src/Downloads.jl:311 [inlined]
    [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Main.Test39Main_Downloads.var"#22#40"{Vector{Any}}, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)
      @ ArgTools /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/ArgTools/src/ArgTools.jl:112
    [5] #7
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Downloads/src/Downloads.jl:310 [inlined]
    [6] arg_read(f::Downloads.var"#7#16"{IOStream, Nothing, Vector{Pair{String, String}}, Float64, Main.Test39Main_Downloads.var"#22#40"{Vector{Any}}, Bool, Bool, String, Int64, Bool, Bool}, arg::Base.DevNull)
      @ ArgTools /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/ArgTools/src/ArgTools.jl:61
    [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Main.Test39Main_Downloads.var"#22#40"{Vector{Any}}, verbose::Bool, throw::Bool, downloader::Nothing)
      @ Downloads /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Downloads/src/Downloads.jl:309
    [8] #3
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Downloads/src/Downloads.jl:222 [inlined]
    [9] arg_write(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Main.Test39Main_Downloads.var"#22#40"{Vector{Any}}, Bool, Nothing, String}, arg::Nothing)
      @ ArgTools /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/ArgTools/src/ArgTools.jl:101
   [10] #download#2
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Downloads/src/Downloads.jl:221 [inlined]
   [11] macro expansion
      @ /buildworker/worker/tester_linuxaarch64/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:412 [inlined]
   [12] macro expansion
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1321 [inlined]
   [13] macro expansion
      @ /buildworker/worker/tester_linuxaarch64/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:411 [inlined]
   [14] macro expansion
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1321 [inlined]
   [15] macro expansion
      @ /buildworker/worker/tester_linuxaarch64/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:397 [inlined]
   [16] macro expansion
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1321 [inlined]
   [17] macro expansion
      @ /buildworker/worker/tester_linuxaarch64/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:337 [inlined]
   [18] macro expansion
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1321 [inlined]
   [19] top-level scope
      @ /buildworker/worker/tester_linuxaarch64/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:4
   [20] include
      @ ./Base.jl:416 [inlined]
   [21] macro expansion
      @ /buildworker/worker/tester_linuxaarch64/build/share/julia/test/testdefs.jl:24 [inlined]
   [22] macro expansion
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Test/src/Test.jl:1321 [inlined]
   [23] macro expansion
      @ /buildworker/worker/tester_linuxaarch64/build/share/julia/test/testdefs.jl:23 [inlined]
   [24] macro expansion
      @ ./timing.jl:368 [inlined]
   [25] runtests(name::String, path::String, isolate::Bool; seed::UInt128)
      @ Main /buildworker/worker/tester_linuxaarch64/build/share/julia/test/testdefs.jl:21
   [26] (::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278
   [27] run_work_thunk(thunk::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:63
   [28] macro expansion
      @ /buildworker/worker/package_linuxaarch64/build/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:278 [inlined]
   [29] (::Distributed.var"#105#107"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ./task.jl:411

`download()` multi-thread error

on:

Julia Version 1.6.0
Commit f9720dc2eb (2021-03-24 12:55 UTC)

julia> imgs = [
       "https://images.unsplash.com/photo-1554266183-2696fdafe3ff?ixlib=rb-1.2.1&auto=format&fit=crop&w=564&q=80"
       "https://images.unsplash.com/photo-1527026950045-9e066846bae4?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1419&q=80"
       "https://images.unsplash.com/photo-1549287748-f095932c9f81?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=606&q=80"
       "https://images.unsplash.com/photo-1437448317784-3a480be9571e?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=634&q=80"
       ]

julia> imgs = repeat(imgs, 4)

julia> Threads.nthreads()
4

julia> Threads.@threads for url in imgs
           download(url)
       end

results in

ERROR: TaskFailedException
Stacktrace:
 [1] wait
   @ ./task.jl:317 [inlined]
 [2] threading_run(func::Function)
   @ Base.Threads ./threadingconstructs.jl:34
 [3] top-level scope
   @ ./threadingconstructs.jl:93

    nested task error: TaskFailedException
    Stacktrace:
      [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))

also seen in another occation:

Error: curl_multi_socket_action: 8

fix/remove wrong host test, failing with OpenSSL

See #114 (comment). Synopsis:

conda-forge builds Julia against OpenSSL instead of MbedTLS
when build against OpenSSL, leaving CURLOPT_SSL_VERIFYHOST on as we do now causes the wrong host test to fail — this is what we would expect since we have told libcurl to allow hosts that it cannot verify the identity of with a root cert but not to allow connections to hosts that claim to be the wrong host entirely
this test doesn't fail when Julia is built against MbedTLS, which seems like a libcurl bug in that build configuration
so: Downloads tests pass (they shouldn't) for the standard Julia build using MbedTLS
and: Downloads tests fail (they should) for the conda-forge Julia build

Simplest fix seems to be to disable that test, but it would also be good to file an upstream issue with libcurl.

todo: more complete API tests

The test coverage is ok, but we could use some more thorough tests.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

timer_callback is causing segfaults with curl 7.81.0

Several unsupported builds of Julia 1.7.1 are reporting segmentation faults when trying to build with curl 7.81.0:

The segmentation fault can be replicated as follows:

$ julia -E 'download("https://curl.se")'

signal (11): Segmentation fault
in expression starting at none:1
prune_dead_connections at /home/mkitti/src/curl/lib/url.c:1092
create_conn at /home/mkitti/src/curl/lib/url.c:3869
Curl_connect at /home/mkitti/src/curl/lib/url.c:4129
multi_runsingle at /home/mkitti/src/curl/lib/multi.c:1860
multi_socket at /home/mkitti/src/curl/lib/multi.c:3131
curl_multi_socket_action at /home/mkitti/src/curl/lib/multi.c:3252
curl_multi_socket_action at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/LibCURL/src/lC_curl_h.jl:230 [inlined]
curl_multi_socket_action at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/utils.jl:91 [inlined]
macro expansion at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/utils.jl:35 [inlined]
#47 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/Multi.jl:147
lock at ./lock.jl:190
timer_callback at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/Multi.jl:146
jfptr_timer_callback_19000.clone_1 at /home/mkitti/anaconda3/envs/curltest/lib/julia/sys.so (unknown line)
_jl_invoke at /usr/local/src/conda/julia-1.7.1/src/gf.c:2247 [inlined]
jl_apply_generic at /usr/local/src/conda/julia-1.7.1/src/gf.c:2429
jlcapi_timer_callback_18861.clone_1 at /home/mkitti/anaconda3/envs/curltest/lib/julia/sys.so (unknown line)
Curl_update_timer at /home/mkitti/src/curl/lib/multi.c:3365
curl_multi_add_handle at /home/mkitti/src/curl/lib/multi.c:504
curl_multi_add_handle at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/LibCURL/src/lC_curl_h.jl:194 [inlined]
macro expansion at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/utils.jl:35 [inlined]
#27 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/Multi.jl:51
lock at ./lock.jl:190
add_handle at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/Multi.jl:44 [inlined]
#9 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:345
with_handle at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Curl/Curl.jl:64
#8 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:311 [inlined]
arg_write at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:112
#7 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:310 [inlined]
arg_read at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:61 [inlined]
#request#5 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:309
request##kw at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:293 [inlined]
#3 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:222 [inlined]
arg_write at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:101
#download#2 at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:221 [inlined]
download at /home/conda/feedstock_root/build_artifacts/julia_1641439672340/work/usr/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:221
_jl_invoke at /usr/local/src/conda/julia-1.7.1/src/gf.c:2247 [inlined]
jl_apply_generic at /usr/local/src/conda/julia-1.7.1/src/gf.c:2429
jl_apply at /usr/local/src/conda/julia-1.7.1/src/julia.h:1788 [inlined]
jl_f__call_latest at /usr/local/src/conda/julia-1.7.1/src/builtins.c:757
#invokelatest#2 at ./essentials.jl:716 [inlined]
invokelatest at ./essentials.jl:714 [inlined]
do_download at ./download.jl:24 [inlined]
download at ./download.jl:20
_jl_invoke at /usr/local/src/conda/julia-1.7.1/src/gf.c:2247 [inlined]
jl_apply_generic at /usr/local/src/conda/julia-1.7.1/src/gf.c:2429
jl_apply at /usr/local/src/conda/julia-1.7.1/src/julia.h:1788 [inlined]
do_call at /usr/local/src/conda/julia-1.7.1/src/interpreter.c:126
eval_value at /usr/local/src/conda/julia-1.7.1/src/interpreter.c:215
eval_stmt_value at /usr/local/src/conda/julia-1.7.1/src/interpreter.c:166 [inlined]
eval_body at /usr/local/src/conda/julia-1.7.1/src/interpreter.c:601
jl_interpret_toplevel_thunk at /usr/local/src/conda/julia-1.7.1/src/interpreter.c:731
jl_toplevel_eval_flex at /usr/local/src/conda/julia-1.7.1/src/toplevel.c:885
jl_toplevel_eval_flex at /usr/local/src/conda/julia-1.7.1/src/toplevel.c:830
jl_toplevel_eval_in at /usr/local/src/conda/julia-1.7.1/src/toplevel.c:944
eval at ./boot.jl:373 [inlined]
exec_options at ./client.jl:270
_start at ./client.jl:495
jfptr__start_40346.clone_1 at /home/mkitti/anaconda3/envs/curltest/lib/julia/sys.so (unknown line)
_jl_invoke at /usr/local/src/conda/julia-1.7.1/src/gf.c:2247 [inlined]
jl_apply_generic at /usr/local/src/conda/julia-1.7.1/src/gf.c:2429
jl_apply at /usr/local/src/conda/julia-1.7.1/src/julia.h:1788 [inlined]
true_main at /usr/local/src/conda/julia-1.7.1/src/jlapi.c:559
jl_repl_entrypoint at /usr/local/src/conda/julia-1.7.1/src/jlapi.c:701
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x5626d2001098)
Allocations: 2720 (Pool: 2711; Big: 9); GC: 0
Segmentation fault (core dumped)

In this case curl is configured according to
https://github.com/conda-forge/curl-feedstock/blob/master/recipe/build.sh#L8-L17
and is built with openssl:

$ julia -E "using LibCURL; unsafe_string(LibCURL.curl_version())"
"libcurl/7.81.0-DEV OpenSSL/3.0.0 zlib/1.2.11 libssh2/1.10.0 nghttp2/1.43.0"

I bisected the issue to the following commit and line:
curl/curl@2b3dd01#diff-f25cb725c8f9fcc9a6532bed11a8f9c22b60c17edd2d4a3615bd7ceaa6ad8c18R489

Error: curl_easy_setopt: 48

I am trying to install some packages in the cluster but I am getting an error. For example, installing DataFrames:

(v1.6) pkg> add DataFrames
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_getinfo: 48
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
...

I tried deleting LD_LIBRARY_PATH but the error persists. In the case of DataFrames the installation proceeds but in other cases I get errors such as:

Downloaded artifact: Xorg_libXext
ERROR: Unable to automatically install 'Xorg_libXext' from '/home/pmarg/.julia/packages/Xorg_libXext_jll/2vwNw/Artifacts.toml'

Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:443
  [3] ensure_all_artifacts_installed(artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, include_lazy::Bool, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:508
  [4] download_artifacts(ctx::Pkg.Types.Context, pkg_roots::Vector{String}; platform::Base.BinaryPlatforms.Platform, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:706
  [5] download_artifacts(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:683
  [6] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_autoprecomp::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1391
  [7] instantiate
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1304 [inlined]
  [8] instantiate(; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1300
  [9] instantiate()
    @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1300
 [10] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:405
 [11] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:386
 [12] do_cmd
    @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]
 [13] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:550
 [14] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [15] invokelatest
    @ ./essentials.jl:706 [inlined]
 [16] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2441
 [17] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1126
 [18] (::REPL.var"#44#49"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:411

versioninfo()
Julia Version 1.6.1
Commit 6aaedecc44 (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, broadwell)

problems with SSL host verification with proxy servers

Multiple people in the last few hours have reported problems downloading artifacts on Windows systems using proxy servers: here and here. In both cases, a workaround was ENV["JULIA_SSL_NO_VERIFY_HOSTS"] = "github.com".

Given how quickly this popped up after the 1.6 release, I'm guessing that this will be a common problem, so I'm filing this issue to keep track of progress on tracking down the source of the problem and fixing it.

FR: Support Data URIs

Increasingly common in webworld is the use of Data URIs, which contain the data in the URL itself.

For example (from wikipedia), this URL points to an image of a red dot (copy and open it in a new tab):

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==

Currently, the data: protocol is not supported in Downloads.jl, nor the old built-in download, but it would be great to have it added!

StackOverflowError with curl_multi_socket_action

This is a pretty low quality bug report since a lot of stacktraces were suppressed by try-catch, but I think the origin is a StackOverflow in Downloads. My logs only show

┌ Error: [2021-02-10T14:48:09] Error downloading
│   exception = StackOverflowError()

after a Downloads.request(...) call and then the logs were filled with 1.3 million lines of

┌ Error: [2021-02-10T14:48:09] curl_multi_socket_action: 8                                   
└ @ Downloads.Curl /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36

I have now put in some more debugging statements so hopefully it happens again and I will have more information...

`Downloads.download` prints `Unhandled Task ERROR: StackOverflowError:` and then hangs indefinitely

When running the generate_historical_stdlibs.jl script, I get Unhandled Task ERROR: StackOverflowError:, and then it hangs indefinitely.

Here are the steps to reproduce under rr:

git clone https://github.com/JuliaLang/Pkg.jl.git
cd Pkg.jl
wget https://raw.githubusercontent.com/JuliaLang/julia/master/.buildkite/utilities/rr/rr_capture.jl
export JULIA_DEPOT_PATH=$(mktemp -d)
export JULIA_PROJECT="ext/HistoricaStdlibGenerator"
julia -e 'import Pkg; Pkg.instantiate()'
export JULIA_TEST_RR_TIMEOUT=10 # 10 minutes
while julia rr_capture.jl julia ext/HistoricaStdlibGenerator/generate_historical_stdlibs.jl; do :; done

Here's the error:

rr: Saving execution to trace directory `/home/dilum/Desktop/Pkg.jl/temp_for_rr/jl_ebxdQF/rr_traces/julia-0'.
[ Info: Downloading versions.json...
[ Info: Identified 25 versions to try...
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.0-linux-x86_64.tar.gz
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.1-linux-x86_64.tar.gz
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.2-linux-x86_64.tar.gz
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.3-linux-x86_64.tar.gz
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.4-linux-x86_64.tar.gz
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.5-linux-x86_64.tar.gz
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.1/julia-1.1.0-linux-x86_64.tar.gz
[ Info: Downloading https://julialang-s3.julialang.org/bin/linux/x64/1.1/julia-1.1.1-linux-x86_64.tar.gz
Unhandled Task ERROR: StackOverflowError:


Process timed out (with a timeout of 10 minutes). Signalling `rr` for force-cleanup!
`rr` returned 0, packing and uploading traces...
rr: Packed trace directory `/home/dilum/Desktop/Pkg.jl/temp_for_rr/jl_ebxdQF/rr_traces/julia-0'.
Read : 443 MB ==> 38%[ Info: The `rr` trace file has been saved to: /home/dilum/Desktop/Pkg.jl/dumps/rr--build_----commit_--2021_11_05_11_26_00.tar.zst
/*stdin*\            : 39.97%   (482458112 => 192818618 bytes, /home/dilum/Desktop/Pkg.jl/dumps/rr--build_----commit_--2021_11_05_11_26_00.tar.zst)
[ Info: Finished running the command under rr
┌ Info:
│   process.exitcode = 0
└   process.termsignal = 15

I have the rr trace. Let me know if you want me to send it to you.

Using `easy_hook` for per-request parameters

Currently the easy_hook function gets passed an info parameter with URL, method and headers:

Downloads.jl/src/Downloads.jl

Lines 341 to 342 in e22219f

 info = (url = url, method = method, headers = headers) 

 easy_hook(downloader, easy, info)

But I can't see how to use this to set options on a per-request basis as HTTP.jl currently allows for things like the connection timeout (https://curl.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html). Ideally we'd add connection timeouts and some other obviously useful parameters (will post a separate issue about those), but to cover more unforeseen settings it would be helpful if there was some way to get per-request items into the info struct.

I'm not sure what's best for the API. A couple of possible ideas:

Add an easy_hook_opts::Any=nothing keyword to request, and just pass this through in the info tuple.
Add an easy_hook::Union{Function,Nothing}=nothing keyword to request and invoke this on the easy handle (maybe in addition to the easy hook which is contained in the Downloader).

I kind of favor the second of these ideas. Perhaps not as a public part of the API, but as an escape hatch it sure would be handy.

`Error: curl_easy_setopt: 48` since upgrading to MacOS 12

On Julia 1.7.0-beta4 and Julia 1.6.1, I started seeing this error message when calling download after upgrading to MacOS 12. (on Apple M1 if relevant). download seems to work fine despite the log message.

Here is the result of running test Downloads on Julia 1.7.0-beta4 (it failed): https://gist.github.com/fonsp/dcfc4bd0d3de845a59a95f1b5e027d52

➜  PlutoSliderServer.jl julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.1 (2021-04-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> download("http://asdf.com/")
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
"/var/folders/v_/fhpj9jn151d4p9c2fdw2gv780000gn/T/jl_nIm6GL"

(@v1.6) pkg> registry up
    Updating registry at `~/.julia/registries/General`
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36
    Updating registry at `~/.julia/registries/PlutoPackages`
    Updating git-repo `https://pluto-packages.dral.eu/PlutoPackages.git`

(@v1.6) pkg>

is it too much to ask to move the helper functions in test/setup.jl into the module?

specifically, it was kind of odd to see the response body get thrown away. I can see why you might not want the body as part of the Response struct, but to then not provide a download_json and request_json seems weird.

I can see this issue getting closed, since its so easy for someone to write themselves

feature request: add an option to `download` to preserve suffix

Sometimes certain file loading function uses suffix to decide what to do, for example, image files with FileIO's load. Right now one has to manually do PNGFile.load(download(...)). I'm wondering if we can add an option (default false) to preserve the suffix. If people like the idea I can try to submit a PR

download("ftp://...") does not work if server sends "229 Entering Extended Passive Mode"

Another regression when using ftp://

julia> using Downloads

julia> Downloads.download("ftp://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/psadmit/1138_bus.mtx.gz", "/tmp/xxx", verbose=true)
*   Trying 129.6.24.27:21...
* Connected to math.nist.gov () port 21 (#0)
< 220 NIST Applied and Computational Mathematics Division
> USER anonymous
< 331 Please specify the password.
> PASS [email protected]
< 230 Login successful.
> PWD
< 257 "/"
* Entry path is '/'
> CWD pub
* ftp_perform ends with SECONDARY: 0
< 250 Directory successfully changed.
> CWD MatrixMarket2
< 250 Directory successfully changed.
> CWD Harwell-Boeing
< 250 Directory successfully changed.
> CWD psadmit
< 250 Directory successfully changed.
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||21845|).
* Could not resolve host: 
* Can't resolve new host :21845
* Closing connection 0
ERROR: 229 Entering Extended Passive Mode (|||21845|) (Could not resolve host: ) while requesting ftp://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/psadmit/1138_bus.mtx.gz
Stacktrace:
 [1] (::Downloads.var"#9#17"{IOStream,Base.DevNull,Nothing,Array{Pair{String,String},1},Bool,Bool,String,Bool,Bool,Int64})(::Downloads.Curl.Easy) at /home/crusius/.julia/packages/Downloads/jW5ZZ/src/Downloads.jl:333
 [2] with_handle(::Downloads.var"#9#17"{IOStream,Base.DevNull,Nothing,Array{Pair{String,String},1},Bool,Bool,String,Bool,Bool,Int64}, ::Downloads.Curl.Easy) at /home/crusius/.julia/packages/Downloads/jW5ZZ/src/Curl/Curl.jl:51
 [3] #8 at /home/crusius/.julia/packages/Downloads/jW5ZZ/src/Downloads.jl:284 [inlined]
 [4] arg_write(::Downloads.var"#8#16"{Base.DevNull,Nothing,Array{Pair{String,String},1},Bool,Bool,String,Bool,Bool,Int64}, ::IOStream) at /home/crusius/.julia/packages/ArgTools/4vlk9/src/ArgTools.jl:94
 [5] (::Downloads.var"#7#15"{Nothing,Array{Pair{String,String},1},Bool,Bool,String,Bool,Bool,Int64})(::Base.DevNull) at /home/crusius/.julia/packages/Downloads/jW5ZZ/src/Downloads.jl:283
 [6] arg_read at /home/crusius/.julia/packages/ArgTools/4vlk9/src/ArgTools.jl:43 [inlined]
 [7] request(::String; input::Nothing, output::IOStream, method::Nothing, headers::Array{Pair{String,String},1}, progress::Nothing, verbose::Bool, throw::Bool, downloader::Nothing) at /home/crusius/.julia/packages/Downloads/jW5ZZ/src/Downloads.jl:282
 [8] #3 at /home/crusius/.julia/packages/Downloads/jW5ZZ/src/Downloads.jl:199 [inlined]
 [9] open(::Downloads.var"#3#4"{Nothing,Array{Pair{String,String},1},Nothing,Bool,Nothing,String}, ::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:write,),Tuple{Bool}}}) at ./io.jl:325
 [10] arg_write(::Function, ::String) at /home/crusius/.julia/packages/ArgTools/4vlk9/src/ArgTools.jl:68
 [11] #download#2 at /home/crusius/.julia/packages/Downloads/jW5ZZ/src/Downloads.jl:198 [inlined]
 [12] top-level scope at REPL[7]:1

Expose setting CURLOPT_LOW_SPEED_TIME

Hello, I noticed that CURLOPT_LOW_SPEED_TIME gets set to 20 seconds, so if there is no response from the server in 20 seconds, the connection will time out. However, I have a use case where I am requesting content from an AWS API Gateway, and I would like to allow at least 30 seconds. Is it possible to set this value to 30 seconds? Thank you.

I am currently running into the following error:

Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds

`arg_read_size` doesn't understand `GenericIOBuffer`

Through a weird series of events, I've discovered that

Downloads.arg_read_size(IOBuffer(codeunits("hi"))) === nothing

(This causes us not to call set_upload_size(), which in turn causes libcurl to set Transfer-Encoding: chunked, which in turn causes PutObject to S3 to fail when input is of type Base.GenericIOBuffer{Base.CodeUnits{UInt8, String}}.

I guess we should define arg_read_size(::GenericIOBuffer)?

Using `SSL_NO_VERIFY` stops sending the SNI header

When setting JULIA_SSL_NO_VERIFY_HOSTS, Downloads.jl does not send the SNI header. Some hosts fail to properly set up an SSL session when the SNI header is not present. Command line curl seems to present the header when when verification is switched off.

Tested on :
system curl : curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2

Downloads.jl: libcurl/7.73.0 SecureTransport zlib/1.2.11 libssh2/1.9.0 nghttp2/1.41.0

Tested with AWS Cloudfront, which shows this behaviour. Error shown from Pkg is:

err = SSL peer handshake failed, the server most likely requires a client certificate to connect while requesting https://.......

cc: @vdayanand

Downloading files with FTP protocol fails with `Could not resolve host`

Julia v1.5:

julia> download("ftp://xmlsoft.org/libxslt/libxslt-1.1.33.tar.gz")
"/tmp/jl_PhxwJQ"

with Downloads:

julia> Downloads.download("ftp://xmlsoft.org/libxslt/libxslt-1.1.33.tar.gz")
ERROR: Could not resolve host: 
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] (::Downloads.var"#2#3"{Vector{Pair{String, String}}, Downloads.Downloader, String})(io::IOStream)
   @ Downloads ~/repo/julia/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:69
 [3] arg_write(f::Downloads.var"#2#3"{Vector{Pair{String, String}}, Downloads.Downloader, String}, arg::Nothing)
   @ ArgTools ~/repo/julia/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:101
 [4] #download#1
   @ ~/repo/julia/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:49 [inlined]
 [5] download
   @ ~/repo/julia/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:48 [inlined]
 [6] top-level scope
   @ REPL[8]:1

tests insufficient to catch Julia CI errors

The test suite is apparently insufficient to catch errors like these:

JuliaLang/julia#42898

It would be good to add enough test coverage to catch all of that.

Error: curl_easy_setopt: 48

Hi,
I posted this problem on Slack originally and was asked to post here as well after we failed to find a fix.

Every time I upgrade my Julia packages, Pkg.update() gives me this same error multiple times:

┌ Error: curl_easy_setopt: 48
└ @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/utils.jl:36

Note that #123 concerns the same problem, but seeing as I'm using an official build of Julia, the discussion there doesn't help I think.

julia> versioninfo()
Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 12

This doesn't affect functionality AFAICT, has been happening for a couple of weeks now and persists even when I try with a nightly build, for example.
LD_LIBRARY_PATH , DYLD_FALLBACK_LIBRARY_PATH and DYLD_LIBRARY_PATH all aren't set.

Adding feature to verify SHA along with download function

A project called ArchGDAL.jl used the download_verify function of Pkg.PlatformEngines but @KristofferC informed that it is going to be deprecated from 1.6.

After going through the project discussion board, I found that Base.download will be replaced by download() of Downloads.jl.

I think an inbuilt functionality to verify the SHA signature of the file will be very helpful here when our users wish to download artifacts.

missing nss_compat.so.1

Looks like the build is missing a runtime dependency? https://build.julialang.org/#/builders/33/builds/5395/steps/5/logs/stdio

      From worker 2:	Error During Test at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/download_exec.jl:25
      From worker 2:	  Test threw exception
      From worker 2:	  Expression: download("https://httpbin.julialang.org/ip", file) == file
      From worker 2:	  could not load symbol "jl_task_get_next":
      From worker 2:	  Shared object "nss_compat.so.1" not found, required by "julia"
      From worker 2:	  Stacktrace:
      From worker 2:	    [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
      From worker 2:	      @ Base ./task.jl:694
      From worker 2:	    [2] wait
      From worker 2:	      @ ./task.jl:753 [inlined]
      From worker 2:	    [3] wait(c::Base.GenericCondition{ReentrantLock})
      From worker 2:	      @ Base ./condition.jl:106
      From worker 2:	    [4] take_buffered(c::Channel{Any})
      From worker 2:	      @ Base ./channels.jl:389
      From worker 2:	    [5] take!
      From worker 2:	      @ ./channels.jl:383 [inlined]
      From worker 2:	    [6] sync_end(c::Channel{Any})
      From worker 2:	      @ Base.Experimental ./experimental.jl:63
      From worker 2:	    [7] macro expansion
      From worker 2:	      @ ./experimental.jl:101 [inlined]
      From worker 2:	    [8] (::Downloads.var"#9#17"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
      From worker 2:	      @ Downloads /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:301
      From worker 2:	    [9] with_handle(f::Downloads.var"#9#17"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
      From worker 2:	      @ Downloads.Curl /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/Curl.jl:44
      From worker 2:	   [10] #8
      From worker 2:	      @ /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:275 [inlined]
      From worker 2:	   [11] arg_write(f::Downloads.var"#8#16"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Nothing, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)
      From worker 2:	      @ ArgTools /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:112
      From worker 2:	   [12] #7
      From worker 2:	      @ /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:274 [inlined]
      From worker 2:	   [13] arg_read
      From worker 2:	      @ /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:61 [inlined]
      From worker 2:	   [14] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, progress::Nothing, verbose::Bool, throw::Bool, downloader::Nothing)
      From worker 2:	      @ Downloads /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:273
      From worker 2:	   [15] #3
      From worker 2:	      @ /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:190 [inlined]
      From worker 2:	   [16] open(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Nothing, Bool, Nothing, String}, args::String; kwargs::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:write,), Tuple{Bool}}})
      From worker 2:	      @ Base ./io.jl:330
      From worker 2:	   [17] arg_write(f::Function, arg::String)
      From worker 2:	      @ ArgTools /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:86
      From worker 2:	   [18] #download#2
      From worker 2:	      @ /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:189 [inlined]
      From worker 2:	   [19] download(url::String, output::String)
      From worker 2:	      @ Downloads /usr~/buildbot/worker/package_freebsd64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:189
      From worker 2:	   [20] #invokelatest#2
      From worker 2:	      @ ./essentials.jl:707 [inlined]
      From worker 2:	   [21] invokelatest
      From worker 2:	      @ ./essentials.jl:706 [inlined]
      From worker 2:	   [22] do_download
      From worker 2:	      @ ./download.jl:33 [inlined]
      From worker 2:	   [23] download
      From worker 2:	      @ ./download.jl:28 [inlined]
      From worker 2:	   [24] (::Main.TestDownload.var"#3#5")(temp_dir::String)
      From worker 2:	      @ Main.TestDownload /usr~/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/download_exec.jl:25
      From worker 2:	   [25] mktempdir(fn::Main.TestDownload.var"#3#5", parent::String; prefix::String)
      From worker 2:	      @ Base.Filesystem ./file.jl:713
      From worker 2:	   [26] mktempdir(fn::Function, parent::String)
      From worker 2:	      @ Base.Filesystem ./file.jl:711
      From worker 2:	   [27] top-level scope
      From worker 2:	      @ /usr~/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/download_exec.jl:22
      From worker 2:	   [28] include(mod::Module, _path::String)
      From worker 2:	      @ Base ./Base.jl:386
      From worker 2:	   [29] exec_options(opts::Base.JLOptions)
      From worker 2:	      @ Base ./client.jl:285
      From worker 2:	   [30] _start()
      From worker 2:	      @ Base ./client.jl:485
      From worker 2:	ERROR: LoadError: There was an error during testing
      From worker 2:	in expression starting at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/test/download_exec.jl:3

end-to-end testing for #167

The tests added for #167 test the helper function content_length but don't do any kind of end-to-end testing that the new functionality works. The way to test would be to make a request to httpbingo that would have been chunked previously but is no longer chunked. However, I haven't figured out a way to tell if an upload request to httpbingo was chunked or not, which is necessary for that test.

Downloading files from SourceForge gets an HTML error page

Julia v1.5

julia> dir = mktempdir()
"/tmp/jl_f42CqU"

julia> url = "https://sourceforge.net/projects/libpng/files/libpng16/1.6.37/libpng-1.6.37.tar.gz"
"https://sourceforge.net/projects/libpng/files/libpng16/1.6.37/libpng-1.6.37.tar.gz"

julia> download(url, joinpath(dir, basename(url)))
"/tmp/jl_f42CqU/libpng-1.6.37.tar.gz"

correctly downloads the archive, while

julia> Downloads.download(url, joinpath(dir, basename(url)))
"/tmp/jl_f42CqU/libpng-1.6.37.tar.gz"

gets an HTML error page from SourceForge. Maybe it's a matter of setting the header correctly?

tests crashing with memory / stack corruption

https://build.julialang.org/#/builders/55/builds/2685/steps/5/logs/stdio

Downloads                          (6) |        started at 2021-08-25T20:56:57.493
      From worker 6:	Internal error: encountered unexpected error in runtime:
      From worker 6:	StackOverflowError()
      From worker 6:	jfptr_MethodCallResult_7509.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:511
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_edge at ./compiler/typeinfer.jl:823 [inlined]
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:507
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_ext at ./compiler/typeinfer.jl:909
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:942
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:938
      From worker 6:	jfptr_typeinf_ext_toplevel_12789.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	jl_type_infer at /buildworker/worker/package_linux32/build/src/gf.c:295
      From worker 6:	jl_generate_fptr at /buildworker/worker/package_linux32/build/src/jitlayers.cpp:338
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:1978
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:2428 [inlined]
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2237 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	#1 at ./experimental.jl:77
      From worker 6:	jfptr_YY.1_44291.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	start_task at /buildworker/worker/package_linux32/build/src/task.c:880
      From worker 6:	Internal error: encountered unexpected error in runtime:
      From worker 6:	StackOverflowError()
      From worker 6:	jfptr_MethodCallResult_7509.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	abstract_call_method at ./compiler/abstractinterpretation.jl:511
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:105
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1895
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:556
      From worker 6:	abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:113
      From worker 6:	abstract_call_known at ./compiler/abstractinterpretation.jl:1309
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1364
      From worker 6:	abstract_call at ./compiler/abstractinterpretation.jl:1349
      From worker 6:	abstract_eval_statement at ./compiler/abstractinterpretation.jl:1503
      From worker 6:	typeinf_local at ./compiler/abstractinterpretation.jl:1877
      From worker 6:	typeinf_nocycle at ./compiler/abstractinterpretation.jl:1985
      From worker 6:	_typeinf at ./compiler/typeinfer.jl:226
      From worker 6:	typeinf at ./compiler/typeinfer.jl:209
      From worker 6:	typeinf_ext at ./compiler/typeinfer.jl:909
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:942
      From worker 6:	typeinf_ext_toplevel at ./compiler/typeinfer.jl:938
      From worker 6:	jfptr_typeinf_ext_toplevel_12789.clone_1 at /buildworker/worker/tester_linux32/build/lib/julia/sys.so (unknown line)
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2226 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	jl_type_infer at /buildworker/worker/package_linux32/build/src/gf.c:295
      From worker 6:	jl_generate_fptr at /buildworker/worker/package_linux32/build/src/jitlayers.cpp:338
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:1978
      From worker 6:	jl_compile_method_internal at /buildworker/worker/package_linux32/build/src/gf.c:2428 [inlined]
      From worker 6:	_jl_invoke at /buildworker/worker/package_linux32/build/src/gf.c:2237 [inlined]
      From worker 6:	jl_apply_generic at /buildworker/worker/package_linux32/build/src/gf.c:2427
      From worker 6:	jl_apply at /buildworker/worker/package_linux32/build/src/julia.h:1771 [inlined]
      From worker 6:	start_task at /buildworker/worker/package_linux32/build/src/task.c:880
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.8/Downloads/src/Curl/utils.jl:36
      From worker 6:	��� Error: curl_multi_socket_action: 8
      From worker 6:	��� @ Downloads.Curl 
...
      From worker 6:	
      From worker 6:	signal (11): Segmentation fault
      From worker 6:	in expression starting at /buildworker/worker/tester_linux32/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:3
      From worker 6:	terminate called after throwing an instance of 'std::bad_alloc'
      From worker 6:	  what():  std::bad_alloc
      From worker 6:	
      From worker 6:	signal (6): Aborted
      From worker 6:	in expression starting at /buildworker/worker/tester_linux32/build/share/julia/stdlib/v1.8/Downloads/test/runtests.jl:3
Downloads                          (6) |         failed at 2021-08-25T20:57:49.740

9 bytes stray data read before trying h2 connection

This is sometimes printed in verbose mode when doing multiple upload requests. For example:

julia> request("https://httpbin.org/put", verbose=true, method="PUT")

*   Trying 34.194.129.11:443...
* Connected to httpbin.org (34.194.129.11) port 443 (#0)
* mbedTLS: Connecting to httpbin.org:443
* mbedTLS: Set min SSL version to TLS 1.0
* ALPN, offering h2
* ALPN, offering http/1.1
* mbedTLS: Handshake complete, cipher is TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
* Dumping cert info:
* cert. version     : 3
* serial number     : 0B:AB:56:F5:2F:C9:F7:21:C8:C3:5B:FC:58:E9:CC:3C
* issuer name       : C=US, O=Amazon, OU=Server CA 1B, CN=Amazon
* subject name      : CN=httpbin.org
* issued  on        : 2020-01-18 00:00:00
* expires on        : 2021-02-18 12:00:00
* signed using      : RSA with SHA-256
* RSA key size      : 2048 bits
* basic constraints : CA=false
* subject alt name  : httpbin.org, *.httpbin.org
* key usage         : Digital Signature, Key Encipherment
* ext key usage     : TLS Web Server Authentication, TLS Web Client Authentication

* ALPN, server accepted to use h2
* SSL connected
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb87c054400)

> PUT /put HTTP/2
Host: httpbin.org
user-agent: libcurl/7.71.1 mbedTLS/2.16.8 zlib/1.2.11 libssh2/1.9.0 nghttp2/1.40.0 julia/1.5.2
accept: */*

* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Wed, 28 Oct 2020 22:28:18 GMT
< content-type: application/json
< content-length: 390
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
* Excess found: excess = 390 url = /put (zero-length body)
* Connection #0 to host httpbin.org left intact

Response("https://httpbin.org/put", 200, "HTTP/2 200", [
    "date" => "Wed, 28 Oct 2020 22:28:18 GMT",
    "content-type" => "application/json",
    "content-length" => "390",
    "server" => "gunicorn/19.9.0",
    "access-control-allow-origin" => "*",
    "access-control-allow-credentials" => "true",
])

julia> request("https://httpbin.org/put", verbose=true, method="PUT")

* 9 bytes stray data read before trying h2 connection
* Found bundle for host httpbin.org: 0x7fb87b63fcd0 [can multiplex]
* Re-using existing connection! (#0) with host httpbin.org
* Connected to httpbin.org (34.194.129.11) port 443 (#0)
* Using Stream ID: 3 (easy handle 0x7fb87b932e00)

> PUT /put HTTP/2
Host: httpbin.org
user-agent: libcurl/7.71.1 mbedTLS/2.16.8 zlib/1.2.11 libssh2/1.9.0 nghttp2/1.40.0 julia/1.5.2
accept: */*

< HTTP/2 200
< date: Wed, 28 Oct 2020 22:28:20 GMT
< content-type: application/json
< content-length: 390
< server: gunicorn/19.9.0
< access-control-allow-origin: *
< access-control-allow-credentials: true
<
* Excess found: excess = 390 url = /put (zero-length body)
* Connection #0 to host httpbin.org left intact

Response("https://httpbin.org/put", 200, "HTTP/2 200", [
    "date" => "Wed, 28 Oct 2020 22:28:20 GMT",
    "content-type" => "application/json",
    "content-length" => "390",
    "server" => "gunicorn/19.9.0",
    "access-control-allow-origin" => "*",
    "access-control-allow-credentials" => "true",
])

I added some extra blank lines for clarity. As you can see, the requests succeed, so this isn't a dire problem, but presumably there should not be "stray data" on the HTTP/2 connection.

Downloads.download does not support local urls

Base.download(url, out) where `url = "file:///localfilename" used to work fine.
Now I get this error:

shell> sh
sh-5.0$ cd /tmp
sh-5.0$ echo hallo >xxx
sh-5.0$ exit

julia> Downloads.download("file:///tmp/xxx", "/tmp/yyy")
ERROR: Error status 0 while requesting file:///tmp/xxx
Stacktrace:
 [1] JuliaLang/julia#3
   @ /home/julia/julia/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:209 [inlined]
 [2] open(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Nothing, Bool, Nothing, String}, args::String; kwargs::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:write,), Tuple{Bool}}})
   @ Base /home/julia/julia/base/io.jl:330
 [3] arg_write(f::Function, arg::String)
   @ ArgTools /home/julia/julia/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:86
 [4] #download#2
   @ /home/julia/julia/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:198 [inlined]
 [5] download(url::String, output::String)
   @ Downloads /home/julia/julia/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:198
 [6] top-level scope
   @ REPL[32]:1

New failures on CI?

Downloads                          (2) |         failed at 2020-10-21T16:40:46.531
Test Failed at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:198
  Expression: 11 ��� length(progress) ��� 12
   Evaluated: 11 ��� 10 ��� 12
Test Failed at /usr/home/julia/buildbot/worker-tabularasa/tester_freebsd64/build/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:201
  Expression: all((p.dl_now == max(0, i - shift) for (i, p) = enumerate(progress)))

https://build.julialang.org/#/builders/33/builds/4986/steps/5/logs/stdio

Local test errors

Running the tests locally I get:

errors: Test Failed at /home/kc/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:93
  Expression: startswith(err.msg, "Could not resolve host")
   Evaluated: startswith("ssl_handshake returned - mbedTLS: (-0x7780) SSL - A fatal alert message was received from our peer", "Could not resolve host")
Stacktrace:
 [1] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:93
 [2] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1144
 [3] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:84

progress: Test Failed at /home/kc/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:173
  Expression: 11 ≤ length(progress) ≤ 12
   Evaluated: 11 ≤ 9 ≤ 12
Stacktrace:
 [1] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:173
 [2] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1144
 [3] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:168

progress: Test Failed at /home/kc/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:176
  Expression: all((p.dl_now == max(0, i - shift) for (i, p) = enumerate(progress)))
Stacktrace:
 [1] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:176
 [2] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1144
 [3] top-level scope
   @ ~/julia/usr/share/julia/stdlib/v1.6/Downloads/test/runtests.jl:168

Feature request: specify a timeout for `Downloads.request`

When using Downloads.request, it would be nice to specify a timeout period. If the timeout period elapses but the request has not yet completed, Downloads.request would throw an exception.

HTTPS - Cert verify failed: BADCERT_NOT_TRUSTED

I found this issue but no documentation to handle that. I believe that we need to add some parameter for verify_ssl=fase

Cert verify failed: BADCERT_NOT_TRUSTED

*   Trying 216.58.199.46:443...
* Connected to docs.google.com () port 443 (#0)
* mbedTLS: Connecting to docs.google.com:443
* mbedTLS: Set min SSL version to TLS 1.0
* ALPN, offering h2
* ALPN, offering http/1.1
* mbedTLS: Handshake complete, cipher is TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384
* Cert verify failed: BADCERT_NOT_TRUSTED
* Closing connection 0
Cert verify failed: BADCERT_NOT_TRUSTED while requesting https://docs.google.com/spreadsheets/d/...

why can't we provide a request body directly

You see,the input arg in request requires a stream, in which AbstractString gives the path of the file.
So how should I pass the string directly?

	info = (url = url, method = method, headers = headers)
	easy_hook(downloader, easy, info)

julialang / downloads.jl Goto Github PK

downloads.jl's People

Contributors

Stargazers

Watchers

Forkers

downloads.jl's Issues

Expected behaviour

System info

Minimal Reproducible Example

Recommend Projects

Recommend Topics

Recommend Org