Light

iceflowre / unidown Goto Github PK

View Code? Open in Web Editor NEW

0.0 2.0 2.0 2.28 MB

Universal Downloader - update, manage, download.

License: GNU General Public License v3.0

Python 100.00%

downloader python

unidown's Introduction

Universal Downloader

unidown manages downloads and will only download items again when they are newer or not downloaded yet.

unidown is written with the main aspect on plugins. unidown itself can only manage the produced data from 3rd party plugins.

unidown saves all downloaded files with their modifications time and and will only download updated or not already downloaded files.

Find more information in the documentation at Read the Docs.

Web

https://github.com/IceflowRE/unidown

Plugins

mr_de

Crawl through german ebooks from Mobileread.
https://github.com/IceflowRE/unidown-mr_de

Credits

Developer
- Iceflower S
  
  [email protected]

Third Party

Packaging

Donald Stufft and individual contributors
https://github.com/pypa/packaging
BSD-3-Clause, Apache-2.0

tqdm

urllib3

License

Copyright (C) 2015-present Iceflower S

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see <https://www.gnu.org/licenses/>.

unidown's People

Contributors

Watchers

Forkers

abbyarmada fossabot

unidown's Issues

Passable arguments to a plugin

Restarting of downloads

If the downloader crashes or connection lost, etc. There should be the possibility to restart.

Use argparse to allow commandline arguments

Define types for argpares arguments

Own logfile for every plugin

Main log file
Module log files
Link in main log file to the module log file

Put updated download_data in plugin too

Ref: #43

Use PyPi as source for update mentioning

Use PyPi as new version source
Push new version to release branch in parallel
Remove release branch 4 weeks after

Fix updater to be ready with a new release

Option to download the first topic post

First topic contains mostly some informations about which is nice to have if possible.

Use utf8 for update.data instead of latin

This needs to update older update.data which uses latin.

Add documentation

Command line window should stay open after finishing

Review thread parser

Put update_time and downloaded_data inside the plugin?

Pro:

everything is at the same place
both are needed againg when, writing the savestate or checking the download

Con:

savestate and last update time are just generated once
needs an extra layer, to save them inside the plugin, because custom plugins shouldnt do this themself

Get rid of tqdm_option

https://github.com/IceflowRE/Universal-Downloader/blob/master/unidown/tools/tqdm_option.py

ebooks not found list

eBook not found list will be generated from found threads and not from really downloaded thread files.
A check if all threads are downloaded can be serialized too.

Delay between the requests

Proxy/SOCKS5 support

Add custom file checking

Need custom checking if a good file is downloaded.
Or add after and before methodes?

with filename.open(mode='r', encoding="utf-8") as reader:
  check = reader.readline(14)
    if check == "<!DOCTYPE html":  # if only the html file "Invalid Attachment specified." was downloaded
      download_failed_list.append(at_id)
    else:
      download_succeed_list.append(at_id)

Exceptions

Write own exceptions
Catch module exceptions
Log them

Code climate support

Add landscape.io badge again

When this is fixed: landscapeio/landscape-issues#290

Use explicit dependency versions

Use .format for strings if it is usefull

Stop program when no internet connection is available

Add a gui

Use modified time from header instead of downloading the whole html

Enable pylint testing as soon it support Python 3.6

Add GitCop

https://gitcop.com

Add 3rd party tool credits

Rename to universal-downloader (unidown)

Https support

Since mobileread uses https, the request should use https too.

Code Climate - pycodestyle and other checks

Since code climate is a little bit outdated, an own methode for checking this (pep8, duplicates, etc.) can be implemented. At least if Code Climate do not update as soon as possible.

Test cases

Nosetest would be the best.

Threads does not seemed to be downloaded

In Progress

Add unidown to PyPi

Add upload script to travis
Provide a long description for PyPi (use Readme)

Setup.py

Use setup.py as installation

Subject

Use setup.py as installation for the downloader.

Reason

Simplify execution.

Use pathlib module instead of os for check existing

Test manager

Version tag in update.data

Add an version tag in update.data

Add code coverage

Simplify version checking

It can be simplified by simply checking two strings, no splitting needed.

Compile python scripts as native applications

If possible.

Reoganize to a general downloader

Test

Add test for all raising exceptions of plugin package

Travis support

Module names should be lowercase

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.