<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clip

Thanks for the examples 👍 For <a href="https://code.visualstudio.co

Another site for test : Original URL : <a href="https://code.visuals

Fix for the first case implemented in <a class="commit-link" data-hovercard-type="comm

Another test case : <a href="https://download.mozilla.org/?product=firefox-latest-ssl&

[Bug] dlm doesn't use the original file name of redirected URL about dlm HOT 13 CLOSED

agourlay commented on May 27, 2024

[Bug] dlm doesn't use the original file name of redirected URL

from dlm.

Comments (13)

agourlay commented on May 27, 2024 2

Thanks for the examples 👍

For https://code.visualstudio.com/sha/download?build=stable&os=linux-x64 I am able to retrieve the proper file name via a HEAD query which contains the following header

"content-disposition": "attachment; filename=\"code-stable-x64-1639562789.tar.gz\""

So I can work this out.

However for https://atom.io/download/deb there is no info in the header, I don't know where the filename comes from.

Those are the headers retrieved:

{"connection": "keep-alive", "content-length": "137468500", "content-type": "application/octet-stream", "content-md5": "MjEM8AbNi9zFmSmtOmtYKw==", "last-modified": "Tue, 27 Jul 2021 05:51:40 GMT", "etag": "0x8D950C29B079281", "server": "Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0", "x-ms-request-id": "bc5c0b65-901e-0009-6a02-1967b6000000", "x-ms-version": "2009-09-19", "x-ms-lease-status": "unlocked", "x-ms-blob-type": "BlockBlob", "accept-ranges": "bytes", "age": "833", "date": "Thu, 03 Feb 2022 19:35:57 GMT", "via": "1.1 varnish", "x-served-by": "cache-hhn4028-HHN", "x-cache": "HIT", "x-cache-hits": "0", "x-timer": "S1643899557.434528,VS0,VE1"}

I will at least implement a fix for the first case.

from dlm.

agourlay commented on May 27, 2024 2

Great to hear 👍

The case for atom-amd64.deb?s=1627025597&ext=.part would not be super clean to handle and it does not look standard.
Happy to leave it as it is.

from dlm.

agourlay commented on May 27, 2024 1

Was the original URL in the input file something like http://domain.com/get_video?id=Jw9Bym8mVafjmoX&expire?
That is, the actual filename is not part of the URL.

from dlm.

chromer030 commented on May 27, 2024 1

Another site for test :

Original URL : https://code.visualstudio.com/sha/download?build=stable&os=linux-x64

Redirected URL : https://az764295.vo.msecnd.net/stable/899d46d82c4c95423fb7e10e68eba52050e30ba3/code-stable-x64-1639562789.tar.gz

I tested others tools like aria2c , They save the file based on redirected URL and with proper filename : code-stable-x64-1639562789.tar.gz

Or explicitly in browsers like Chrome, clicking on Original URL will give us the file with proper filename.

but dlm saves it with : download?build=stable&os=linux-x64part which is part of Original URL , in RAW mode, even without file extension.

Another site :

https://atom.io/download/deb

💐Thanks a Million for all your efforts to improving this useful tool.💐

from dlm.

agourlay commented on May 27, 2024 1

Fix for the first case implemented in ec74c4d
It is released as part of 0.2.4

I do not know yet how to handle the second redirect case.

from dlm.

agourlay commented on May 27, 2024 1

Thanks for the investigation, I will have to investigate the source code of aria2c.

I suspect it uses the Location header before redirection.

curl -i https://atom.io/download/deb 
HTTP/1.1 302 Found
Status: 302 Found
Location: https://atom-installer.github.com/v1.58.0/atom-amd64.deb?s=1627025597&ext=.deb

from dlm.

chromer030 commented on May 27, 2024 1

Another test case : https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US

Result :

user@user ~/Desktop/ % ./dlm -i dlm.txt -M 1 -o . --proxy http://127.0.0.1:10809 -U 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36'
[2022-02-04 14:29:13] Starting dlm with at most 1 concurrent downloads
[2022-02-04 14:29:13] Found 1 URLs in input file d.txt
[2022-02-04 14:29:17] Could not determine file extension for https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0/1
?product=firefox-latest-ssl&os=osx& [00:00:11] [>---------------------------------------] 814.95KiB/120.90MiB (speed:63.31KiB/s) (eta:32m)

Current output name : ?product=firefox-latest-ssl&os=osx&lang=en-US.part

True desired name : Firefox 96.0.3.dmg

aria2c output :

user@user ~ % aria2c 'https://download.mozilla.org/?product=firefox-latest-ssl&os=osx&lang=en-US'

02/04 14:33:58 [NOTICE] Downloading 1 item(s)
[#fe3133 0B/0B CN:1 DL:0B]                                                                                                                                  
02/04 14:33:59 [NOTICE] CUID#7 - Redirecting to https://download-installer.cdn.mozilla.net/pub/firefox/releases/96.0.3/mac/en-US/Firefox%2096.0.3.dmg                                                                                                             
[#fe3133 16KiB/120MiB(0%) CN:3 DL:60KiB ETA:34m10s]                                                                                                         
                                                                                            
02/04 14:34:03 [NOTICE] Download GID#fe313386b4ecf94a not complete: /home/user/Firefox 96.0.3.dmg

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
fe3133|ERR |   369KiB/s|/home/user/Firefox 96.0.3.dmg

from dlm.

agourlay commented on May 27, 2024 1

I pushed a fix for using the Location header if possible in case of redirect.
You can give it a try in 0.2.5.

from dlm.