nhorus / ponydownloader Goto Github PK

Currently, pipeline is... tangled and snarled.
Steamlining, in this context - functions should not pass poor image data five functions deep and three wide, but ten in line.

Hello.
When i'm running
./ponydownloader-linux-amd64 --dir=dock --key=.... --tag=dock
i've got error: panic: runtime error: index out of range

Most of other tags works just fine.

Version 0.7.0
OS: Linux arch 4.3.3-2-ARCH #1 SMP PREEMPT Wed Dec 23 20:09:18 CET 2015 x86_64 GNU/Linux

More log:
error.txt

It should remain secret, and it isn't after #11

Always loads files of size 0

Hey, would you mind uploading new executable files for your derpibooru-downloader?

The current executable stopped working sometime last year due to changes on derpibooru.

I even installed GO and tried compiling the recent source code myself, but I keep getting "cannot find package"-errors. Lacking any knowledge of GO-programming, I cannot fix them.

An executable for the recent version would be appreciated. :)

(I use Windows 7 [x64])

My apologies for putting this here, github does not seem to support private messaging, which is quite frankly odd.

For reference, below are the errors described:

C:\ponydownloader-0.11.1>go build

init_windows.go:10:2: cannot find package "github.com/inconshreveable/mousetrap"
in any of:
C:\Go\src\github.com\inconshreveable\mousetrap (from $GOROOT)
C:\Users\username\go\src\github.com\inconshreveable\mousetrap (from $GOPATH
)
settings.go:11:2: cannot find package "github.com/jessevdk/go-flags" in any of:
C:\Go\src\github.com\jessevdk\go-flags (from $GOROOT)
C:\Users\username\go\src\github.com\jessevdk\go-flags (from $GOPATH)
logging.go:9:2: cannot find package "gopkg.in/natefinch/lumberjack.v2" in any of
:
C:\Go\src\gopkg.in\natefinch\lumberjack.v2 (from $GOROOT)
C:\Users\username\go\src\gopkg.in\natefinch\lumberjack.v2 (from $GOPATH)

because currently it thinks that 404 is a correct one

Currently, it's much power and responsibility and complication that falls to the user.

Should collect filtered stream, retry stream and os.Signal channel
After signal is received, stop consumption of new images, close unbuffered imagechannel and let buffered incoming ones to fill and block their goroutines.
Use two-level select?
Must be done for #8 and #9
Automatically closes #9 when complete

So they would not clutter main function and would be more accessible.

Does not stop parsing last page at all. This is wrong

Handle ctrl-c and sigterm gracefully.
Leave no half-downloaded files.
Possibly add timeout?
Count processed, downloaded, erroneous and filtered images, possibly total download count?

Find any way to prune -f function and enable filters explicitly when cases are given

Currently it's horrible cludge, ability to use scheme for working around prefix crutch and other horrors would be much appreciated.

For #14
Currently that test may kill program. Also, can only be run once, because magical globals are magical and are closed/killed and reinitialized only on program rerun. And that's not the most conductive for repeating testing.

What if someone uploads image with same tag you are searching for?
last image on one page will be first image on next one, two attempts to write in one file, fail.

Investigate, find solution.

When I try to download images, this error happens:
Errors at 2015/11/12 11:11:08. Error while getting search page 1.
Errors at 2015/11/12 11:11:08. Get https://derpiboo.ru/search.json?q=applelion&page=1: x509: certificate signed by unknown authority.
event.txt

I got no idea why it's sometimes gives correct hash, sometimes doesn't
May be need to compare to other given hash from API, too?
For now, command line key and settings option to disable warnings about hash mismatch.

Downloader successfully gets the list (looking at url) but failed to download files itself:

$ ./ponydownloader-linux-amd64 --dir=test --key= --tag=test
Derpibooru.org Downloader version 0.9.0
Happened at 2016/07/23 12:30:40 Program start
Happened at 2016/07/23 12:30:40 Processing tags test
Happened at 2016/07/23 12:30:40 Starting worker
Happened at 2016/07/23 12:30:40 Worker started; reading channel
Happened at 2016/07/23 12:30:40 Searching as https://derpibooru.org/search.json?q=test&key=
Happened at 2016/07/23 12:30:40 Searching page 1
Happened at 2016/07/23 12:30:40 Searching page 2
Happened at 2016/07/23 12:30:40 Saving as 0.png
Happened at 2016/07/23 12:30:40 Saving as 0.gif
Happened at 2016/07/23 12:30:40 Saving as 0.gif
Happened at 2016/07/23 12:30:40 Saving as 0.gif
Happened at 2016/07/23 12:30:41 Searching page 3
Error at 2016/07/23 12:30:41 query.go:64: Incorrect request to server, error 404 Not Found
Error at 2016/07/23 12:30:41 query.go:65: Possible API changes
Happened at 2016/07/23 12:30:41 Saving as 0.jpg
Error at 2016/07/23 12:30:41 query.go:64: Incorrect request to server, error 404 Not Found
Error at 2016/07/23 12:30:41 query.go:65: Possible API changes
Happened at 2016/07/23 12:30:41 Saving as 0.png
Error at 2016/07/23 12:30:41 query.go:64: Incorrect request to server, error 404 Not Found
...

Flags should not be in the middle of paragraph, plus description must be cleaner.

Prevent multiple downloads if file with same name and size is already on disk
Also, add option to force clobber

Instead of current pass-and-modify system, filters, type modificators and other elements should return values of different types, to prevent accidental cross-pollination and to simplify testing

Freshly created directory for holding all images got borked permissions, nothing saves in it.

Hi. In russia derpibooru.org was banned (reason - roscomnadzor. You can see it on derpiboo.ru).
And trixiebooru.org still works.
So, i can change source files from derpi to trixie ('cause that's easy), but i don't know how to run it. Can you help me run it or can you make trixiebooru version of the program?

Content-length header in response.

So people could use different config files for different preferences/use causes.

Creation of config.ini or lack of fields in it is not mentioned in event.log

This is suboptimal.

To preserve and contain some kind of collection
Also, to prevent multiple downloads

Minimum - and maximum - that I expect to do from current state of development branch (at commit 6b70495 ) to be able to call it a day, bump version and publish as v0.9.0

Currently, only way to make sure everything works correctly is to compile and run against remote server.
Possibly dependent on #11

Make it an setting, not only an option

Need to add warning, delay and/or confirmation if it's enabled.