cartmanishere / zippyshare-scraper Goto Github PK

View Code? Open in Web Editor NEW

107.0 15.0 28.0 46 KB

A module to get direct downloadable links from zippyshare download page.

License: MIT License

Python 99.52% JavaScript 0.48%

scraping-websites zippyshare download-links multithreading zippyshare-scraper

zippyshare-scraper's Introduction

Check the CHANGELOG for updates.

Zippyshare Scraper:

This is a script to get direct download links to files from zippyshare. If you've used zippyshare to download anything then you know that you have to go to their page and click on the download now button to get the download started.

This script extracts the real download link from the page. You can directly feed that link to a downloader to get your download started.

This script is useful when working on remote servers where you don't have access to gui software.

Reset Zippyshare Uploads:

If you're a zippyshare uploader, you know that zippyshare uploads are taken down if they're not downloaded in last 30 days.

You can use this script to reset the last download date without actually downloading the complete upload.

This script initiates the download of the file to test whether the link is working. Because of this, the last downloaded date for the file is also updated.

In this way, you can very easily extend the lifetime of your upload without wasting valuable time and bandwidth.

Dependencies :

You need Python 3 environment to execute the script. You can easily install it from here.
Install the python dependencies:

	pip install requirements.txt

Options:

Arg	Value	Description
`--in-file`	filepath	Path of the file containing zippyshare links to parse.
`--out-file`	filepath	Path of the file in which generated links will be stored.
`--dlc`	filepath	Path of a `.dlc` file. Takes precendence over `--in-file`.
`--filecrypt`	link	Link of a filecrypt container page. Note: It should not have a password or captcha.
`--engine`	`js`/`text`	Which engine to use for generating links. `js` by default. See `Engines` below for explanation.

Engines:

History

This library used to work by scraping the zippyshare webpage.
Parsing the javascript code to generate the link by regex matching.
Whenever the source code of the site changed even slightly, this broke the regex matchers.
Hence we ended up multiple different patterns that the site source code can have.

Update

Instead of parsing the javascript using regex matching, the library has switched to executing the javascript code.
The pure python implementation of javascript engine js2py is used for this.
This should make the library more robust.

For now, I am keeping both the different approaches for getting the download links. These are the two engines --

JsEngine
TextEngine

Usage :

Input links using an input file --

python zippyshare.py --in-file input_links.txt --out-file links.txt

input links using dlc file --

python zippyshare.py --dlc filename.dlc --out-file links.txt

Input links from terminal --

python zippyshare.py

Examples :

Example of unprocessed link (this type of link will be input): http://www120.zippyshare.com/v/7DpZTYfi/file.html

Example of Direct Downloadable link: http://www120.zippyshare.com/d/7DpZTYfi/4656/Ghost.In.The.Shell.S2.x265.7z.003

(Optional) :

You can then download from links.txt as follows:

aria2c -i links.txt --file-allocation=none -c --auto-file-renaming=false

wget -nc -i links.txt

Or you can download using any other downloader you prefer.

Known Issues :

You have to run the script from the same network using which you are downloading files otherwise links may not work.
The direct download links stop working after a few hours ( About 3-4 hrs, maybe). Don't know the exact time period. At that point, you can rerun the script to get new download links to the same files which will work without problem.
The script runs into an error when the "File not exist" zippyshare page loads.

License :

This project is licensed under the terms of the MIT license.

zippyshare-scraper's People

Contributors

Stargazers

Watchers

zippyshare-scraper's Issues

June 25/19 - I think they changed their code again

All 4 scripts not working as of June 25/19. Please check.

Script generates invalid download link

Something has changed recently on Zippyshare's side causing the script to generate invalid links.

$ python zippyshare.py
Enter URLs (leave blank to stop): https://www37.zippyshare.com/v/cBO2J0K5/file.html
Enter URLs (leave blank to stop): 
[*] Using JSEngine for generating links
[*] 1/1 links parsed https://www37.zippyshare.com/d/123/asd.xml
[*] Verifying download links...
[*] 1/1 links verified
Time taken: 3.223s
[*] All download links saved at links.txt

Alternative approaches to extracting links

Zippyshare keeps changing their link construction logic.

And I am relying on the fact that they will not make it hard enough that I can't solve it using some regex.

The day they decide to crack down on this, we will lose the ability to extract zippy links.

I am creating this issue so that if anyone has an alternative solution/idea on how something like what this repo does can be implemented in a reliable way, I would love to have a discussion around that.

i cannot install loaders in python3

please help

"redirected more than 3 times" during download link verification

This tool seems to have stopped working for me recently. Links can still be parsed, but when verifying download links, it says that it redirected more than 3 times. The links are not saved in links.txt. Additionally, the 'Last download' date on Zippyshare is not updated.

Here is my terminal output when running a single link:

Enter URLs (leave blank to stop): https://www25.zippyshare.com/v/dCDCcbJk/file.html
Enter URLs (leave blank to stop): 
[*] pattern_1 has failed for link: https://www25.zippyshare.com/v/dCDCcbJk/file.html
[*] Trying next pattern
[*] pattern_2 has failed for link: https://www25.zippyshare.com/v/dCDCcbJk/file.html
[*] Trying next pattern
[*] pattern_3 has failed for link: https://www25.zippyshare.com/v/dCDCcbJk/file.html
[*] Trying next pattern
[*] pattern_4 has failed for link: https://www25.zippyshare.com/v/dCDCcbJk/file.html
[*] Trying next pattern
[*] 1/1 links parsed https://www25.zippyshare.com/d/dCDCcbJk/5/VPN-White-Paper.pdf
[*] Verifying download links...
[*] https://www25.zippyshare.com/d/dCDCcbJk/4/VPN-White-Paper.pdf redirected more than 3 times
[*] All download links saved at links.txt

Missing dependency lxml

I tried this script on an almost fresh Ubuntu 20.04 install but got this error:

Traceback (most recent call last):
  File "zippyshare.py", line 398, in <module>
    links, failed = zippy.parse_links(urls)
  File "zippyshare.py", line 128, in parse_links
    extract, link = f.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "zippyshare.py", line 64, in get_download_link
    soup = BeautifulSoup(html.content, "lxml")
  File "/home/shivam/.local/lib/python3.8/site-packages/bs4/__init__.py", line 242, in __init__
    raise FeatureNotFound(
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?

This was fixed by:

pip3 install lxml

I think lxml should be added to the dependencies in the README

Please add extra opption

Hello.
Thanks for great scrapper, 👍
Is possible add extra extraction from filecrypt.cc

Thanks before.

None of the patterns are working

Hi, I tried to get the DDLs from a dlc but was not able to. It tries the patterns and outputs "All Patterns Failed" for all the links. I've pasted the output for one link as below

python zippyshare.py
Enter URLs (leave blank to stop): https://www61.zippyshare.com/v/t84fjr8d/file.html
Enter URLs (leave blank to stop):
[!] REGEX_1 failed for pattern #1
[*] pattern_1 has failed for link: https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #2
[*] pattern_2 has failed for link: https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #3
[*] pattern_3 has failed for link: https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #4
[*] pattern_4 has failed for link: https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #5
[*] pattern_5 has failed for link: https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #6
[*] pattern_6 has failed for link: https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #7
[*] pattern_7 has failed for link: https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Trying next pattern
[*] All patterns have failed
[*] Failed to parse - https://www61.zippyshare.com/v/t84fjr8d/file.html
[*] Verifying download links...
[*] All download links saved at links.txt
[*] All failed links saved at failed.log

I tried it with a .dlc file first but it has around 200 links all with the same result.

Additionally, the program for some reason is not identifying --dlc-file as a valid input

python zippyshare.py --dlc-file "C:\Users\username\Downloads\f91ddc57a78414c162378996348be2a2457f5a88.dlc"
usage: zippyshare.py [-h] [--in-file INFILE] [--out-file OUTFILE] [--dlc DLCFILE]
zippyshare.py: error: unrecognized arguments: --dlc-file C:\Users\adilk\Downloads\f91ddc57a78414c162378996348be2a2457f5a88.dlc

--dlc is working to get the links from a dlc container.

Thanks for your work though. :)

Regex failing

I think they changed their code again, yes? Tried all 3 scripts.

All patterns have failed

I'm getting below error

Enter URLs (leave blank to stop): https://www72.zippyshare.com/v/5X58ey6E/file.html
Enter URLs (leave blank to stop):
[!] REGEX_1 failed for pattern #1
[*] pattern_1 has failed for link: https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #2
[*] pattern_2 has failed for link: https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #3
[*] pattern_3 has failed for link: https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #4
[*] pattern_4 has failed for link: https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #5
[*] pattern_5 has failed for link: https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #6
[*] pattern_6 has failed for link: https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Trying next pattern
[!] REGEX_1 failed for pattern #7
[*] pattern_7 has failed for link: https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Trying next pattern
[*] All patterns have failed
[*] Failed to parse - https://www72.zippyshare.com/v/5X58ey6E/file.html
[*] Verifying download links...
[*] All download links saved at links.txt
[*] All failed links saved at failed.log

filecrypt not working again?

Hey man,

thanks for this script. But it seems it doesn't work. Maybe they changed the site again? Tried several attempts.

I've seen I can hit you up, if this happens, so there it is haha.

Cheers.

No links working

For example, https://www115.zippyshare.com/v/QPL29BGy/file.html doesn't work for any pattern and ends up in failed.log
same with http version of link and administrator cmd

All patterns now do not work

[] pattern_1 has failed
[] Trying next pattern
[] pattern_2 has failed
[] Trying next pattern
[] pattern_3 has failed
[] Trying next pattern
[] pattern_4 has failed
[] Trying next pattern
[*] All patterns have failed

zippyshare changed somehow?

Hey, since today i can't process zippy links, it's testing them before output and all are getting redirected to hell and outputting this error:

if count > 3:
logging.error('{} redirected more than {} times'.format(link, count))
return None

i've tried to increase count but no dice...

All patterns have failed

Hi guys, I was trying to use your program but I got this error:

Python version

[root@server]# python --version
Python 3.6.9

links.txt

https://www41.zippyshare.com/v/dcJOkMVs/file.html

[root@server]# python zippyshare.py --in-file links.txt
[*] pattern_1 has failed for link: https://www41.zippyshare.com/v/dcJOkMVs/file.html
[*] Trying next pattern
[*] pattern_2 has failed for link: https://www41.zippyshare.com/v/dcJOkMVs/file.html
[*] Trying next pattern
[*] pattern_3 has failed for link: https://www41.zippyshare.com/v/dcJOkMVs/file.html
[*] Trying next pattern
[*] pattern_4 has failed for link: https://www41.zippyshare.com/v/dcJOkMVs/file.html
[*] Trying next pattern
[*] pattern_5 has failed for link: https://www41.zippyshare.com/v/dcJOkMVs/file.html
[*] Trying next pattern
[*] pattern_6 has failed for link: https://www41.zippyshare.com/v/dcJOkMVs/file.html
[*] Trying next pattern
[*] pattern_7 has failed for link: https://www41.zippyshare.com/v/dcJOkMVs/file.html
[*] Trying next pattern
[*] All patterns have failed
Traceback (most recent call last):
  File "zippyshare.py", line 430, in <module>
    links, failed = zippy.parse_links(urls)
  File "zippyshare.py", line 129, in parse_links
    extract, link = f.result()
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "zippyshare.py", line 63, in get_download_link
    html = self.sess.get(link)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/sessions.py", line 516, in request
    prep = self.prepare_request(req)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/sessions.py", line 459, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/models.py", line 314, in prepare
    self.prepare_url(url, params)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/requests/models.py", line 388, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://?

dcrypt.it

I've noticed that this script goes to http://dcrypt.it/decrypt/paste to decrypt dlc containers. Sometimes when running the scraper though, it fails to scrape all the links in a dlc container, forcing me to re-run the script multiple times. This calls dcrypt.it more than necessary, sometimes making the server lock me out.

Can you amend the script so that it produces another file "decrypted-[name of dlc].txt" so that I won't have to keep going to dcrypt.it?

Trying this new scheme - got error

I'm running -

zippyshare.py --in-file dl.txt

and I got this error -

    File "C:\Python37\lib\site-packages\requests\models.py", line 387, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://?

Anything come to mind? I'm on Windows.

Update - Can't have any blank lines in the --in-file list of links! Please add to Readme.

Failed to Parse all 6 patterns

This is the Url i tried: https://www10.zippyshare.com/v/f3qwAQAG/file.html

script close itself in 1 second

Hello
I want to thank you very much for your great efforts.

1-I downloaded zippyshare-scraper-master
2- installed python from official site
3- these two packages:
*pip install requests
*pip install bs4
I think first one comes with python setup by default and the I don't know about the second but I downloaded it from here:
https://pypi.org/project/bs4/#files

4- I opened "zippyshare.py" but just cmd screen came to me for 1 second and closed itself!

So pleases, I don't have programming experience ... could you help me?
THANK you very much

.

No longer works

Traceback (most recent call last):
File "zippyshare.py", line 290, in
links = zippy.parse_links(urls)
File "zippyshare.py", line 131, in parse_links
extract, link = f.result()
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
return self.__get_result()
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "zippyshare.py", line 76, in get_download_link
extract = self.pattern_1(soup)
File "zippyshare.py", line 180, in pattern_1
a = math.floor(a / 3)
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

Questions about installation

For some reason, I'm having trouble setting up a fresh Ubuntu machine. I ran the following right after installing Ubuntu.
sudo apt-get update
sudo apt-get install -y python3
sudo apt-get install -y python3-pip
pip3 install requests
pip3 install bs4
sudo apt-get install -y aria2
sudo apt-get install -y python3-lxml

Then, I ran python3 zippyshare.py --dlc file.dlc

This is what I get (links removed)
[] pattern_1 has failed for link: https://www2.zippyshare.com/v/***/file.html
[] Trying next pattern
[] pattern_2 has failed for link: https://www2.zippyshare.com/v/***/file.html
[] Trying next pattern
[] pattern_3 has failed for link: https://www2.zippyshare.com/v/***/file.html
[] Trying next pattern
[] pattern_4 has failed for link: https://www2.zippyshare.com/v/***/file.html
[] Trying next pattern
[] pattern_5 has failed for link: https://www2.zippyshare.com/v/***/file.html
[] Trying next pattern
[] pattern_6 has failed for link: https://www2.zippyshare.com/v/***/file.html
[] Trying next pattern
[] All patterns have failed
[] Failed to parse - https://www2.zippyshare.com/v/***/file.html

The strange thing is that the same command and dlc file worked for an old Ubuntu machine I use, so I'm really confused why my installation failed. Any ideas?

REGEX_4 Failed.

While using the script i got this error, was working fine yesterday but today it says "REGEX_4 Failed"

Script stopped working

Hi,

Script stopped working. Could you please help update the code. Thank you!

Software no longer working on any zippyshare link

I've been using this script without issue for a few weeks now, however as of two days ago, every link i input on any of the three methods (even those links that i have previously successfully downloaded from) gives an error as shown in the attached pictures.

Not working - would love a fix. Thanks!

@Cartmanishere
Thank you for this amazing script. It seems to not be working currently

[*] Using JSEngine for generating links

PyJs_anonymous_1_

PyJs_anonymous_3_
Traceback (most recent call last):
File "zippyshare.py", line 144, in
links, fails = zippy.parse_links(urls)
File "zippyshare.py", line 68, in parse_links
extract, link = f.result()
File "/usr/lib/python3.7/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "zippyshare.py", line 33, in get_download_link
extract, link = self.engine.get_download_link(link)
File "/media/zippyshare-scraper-master/engines/js.py", line 88, in get_download_link
self.run_js(code)
File "/media/zippyshare-scraper-master/engines/js.py", line 59, in run_js
self.context.execute(code)
File "/home/.local/lib/python3.7/site-packages/js2py/evaljs.py", line 199, in execute
exec (compiled, self._context)
File "", line 23, in
File "/home/.local/lib/python3.7/site-packages/js2py/base.py", line 995, in callprop
cand.typeof(), repr(prop), repr(self.Class)))
js2py.internals.simplex.JsException: TypeError: 'undefined' is not a function (tried calling property 'getAttribute' of 'Object')

aria2c zippy links redirect

Once I parsed the unprocessed zippy links and the script outputs the processed .rar links into the links.txt file, I run the aria2c command that you suggested, and for 84/124 links it was successful in downloading the .rar file, however it then started spitting out errors such as this:

aria2c http://www90.zippyshare.com/d/1z3lzoUx/8136/Adventure%20Time%20%282010%29%20Season%201-10%20S01-S10%20%2b%20Extras%20%281080p%20BluRay%20x265%20HEVC%2010bit%20AAC%202.0%20ImE%29.part086.rar --file-allocation=none -c --auto-file-renaming=false

02/07 00:55:08 [NOTICE] Downloading 1 item(s)

02/07 00:55:08 [NOTICE] CUID#7 - Redirecting to http://www90.zippyshare.com/v/1z3lzoUx/file.html

02/07 00:55:09 [NOTICE] CUID#7 - Redirecting to https://www90.zippyshare.com/v/1z3lzoUx/file.html
[#e45c82 0B/0B CN:1 DL:0B]
02/07 00:55:09 [ERROR] CUID#7 - Download aborted. URI=http://www90.zippyshare.com/d/1z3lzoUx/8136/Adventure%20Time%20%282010%29%20Season%201-10%20S01-S10%20%2b%20Extras%20%281080p%20BluRay%20x265%20HEVC%2010bit%20AAC%202.0%20ImE%29.part086.rar
Exception: [AbstractCommand.cc:403] errorCode=13 URI=https://www90.zippyshare.com/v/1z3lzoUx/file.html
  -> [RequestGroup.cc:784] errorCode=13 File C:.../zippyshare-scraper/file.html exists, but a control file(*.aria2) does not exist. Download was canceled in order to prevent your file from being truncated to 0. If you are sure to download the file all over again, then delete it or add --allow-overwrite=true option and restart aria2.

02/07 00:55:09 [NOTICE] Download GID#e45c820358077691 not complete: C:...file.html

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
e45c82|ERR |       0B/s|C:.../file.html

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

For some reason, it is redirected back to the unprocessed zippy link and fails to download properly. Is there any fix for this?

Thanks

Script suddenly stopped working

Firstly would like to thank for the program, Script stopped working today. Receiving below errors:

C:\Users\Test>C:\Users\Test\Zippyshare\zippyshare_3.py
File(f) | List(l) | dlcfile (d)? f
File path: C:\Users\Test\Zippyshare\links.txt
File found. Beginning scraping.
Total Links = 1
Traceback (most recent call last):
File "C:\Users\Test\Zippyshare\zippyshare_3.py", line 184, in
links = parser.do_main()
File "C:\Users\Test\Zippyshare\zippyshare_3.py", line 24, in do_ma
in
extract, status = self.parse_link(i)
File "C:\Users\Test\Zippyshare\zippyshare_3.py", line 155, in pars
e_link
d = self.get_value_of_d()
File "C:\Users\Test\Zippyshare\zippyshare_3.py", line 179, in get_
value_of_d
return int(self.soup.select('span[id="omg"]')[0].get('class')[0]) * 2
IndexError: list index out of range

List of URLs with mixed patterns?

I have a list of zippyshare URLs with different patterns:

https://www17.zippyshare.com/v/2pJx1Rvd/file.html
https://www15.zippyshare.com/v/M0nygsaa/file.html
https://www119.zippyshare.com/v/lNehgCR9/file.html

I think the first 2 are pattern_4 and the 3rd one is pattern_2.

Running this list using --in-file doesn't work. It works OK if I give a file with the first 2 URLs only, or if I give a file with the 3rd URL only.

It seems if I run the in-file with all 3 URLs, it settles on pattern_4 for the first URL, but then assumes all the rest will be also, so it has a problem when it gets to the 3rd URL?

Patterns failed

Disregard. It works now