Git Product home page Git Product logo

iqdb_tagger's Introduction

IQDB TAGGER

https://travis-ci.org/rachmadaniHaryono/iqdb_tagger.svg?branch=master

Overview

Get result from iqdb.org from CLI using python3.

Feature:

- Written for python3
- iqdb and danbooru.iqdb parser
- Hydrus integration

Usage

Use as Hydrus as cli program

To parse folder of images (e.g. in this example image_folder:) and write tags to text file, use following command:

iqdb-tagger cli-run --resize --match-filter best-match --write-tags --input-mode folder image_folder

Use as Hydrus iqdb script server

  1. Run iqdb-tagger run and note the server address.
To run it on 127.0.0.1 on port 5006, run the following command:
`iqdb-tagger run -h 127.0.0.1 -p 5006`
  1. Import one of the parsing scripts below to Hydrus parsing scripts.
  2. Check the server address and edit it as needed.

IQDB parsing script

[32, "local iqdb", 2, ["http://127.0.0.1:5006", 1, 0, [55, 1, [[], "some hash bytes"]], "file", {"place": "0", "resize": "on"}, [[29, 1, ["link", [27, 5, [[["a", {"data-status": "best-match", "class": "img-match-detail"}, null]], 0, "href", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], [[30, 2, ["", 0, [27, 5, [[["li", {"class": "tag-creator"}, null]], 1, "", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], "creator"]], [30, 2, ["", 0, [27, 5, [[["li", {"class": "tag-series"}, null]], 1, "", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], "series"]], [30, 2, ["", 0, [27, 5, [[["li", {"class": "tag-character"}, null]], 1, "", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], "character"]], [30, 2, ["", 0, [27, 5, [[["li", {"class": "tag-general"}, null]], 1, "", [51, 1, [3, "", null, null, "example string"]], [55, 1, [[], "parsed information"]]]], ""]]]]]]]]

Every uploaded and match history can be seen on Front page (in this case http://127.0.0.1:5006).

Using IQDB-tagger with Hydrus API

Set up your hydrus to get the access key, which will be used for this feature.

Install the required hydrus package

pip install https://gitlab.com/cryzed/hydrus-api/-/archive/master/hydrus-api-master.zip

After that you can run the command below. For example to run the command with image tagged as 'thread:cat' on hydrus

# to get tags
iqdb-tagger search-hydrus-and-send-tag --access_key 1234_your_access_key 'thread:cat'
# to get matching urls
iqdb-tagger search-hydrus-and-send-url --access_key 1234_your_access_key 'thread:cat'

Note: hydrus version 349 have default bandwidth of 100 mb data per month, which may raise ApiError when the bandwidth reached.

to fix it, go to services menu -> manage services -> client api and raise your bandwidth limit

Setting Hydrus iqdb script server on NAS

Here is example for Synology DS1817+ with DSM6.1.7 running on an Intel Atom C2538

  1. Make sure SSH is turned on in your control panel
  2. Install python 3 community package: https://synocommunity.com/package/python3
  3. Install pip3
install pip3 with:
sudo -i
wget https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
  1. Install iqdb-tagger
cd /volume1/@appstore/py3k/usr/local/bin
./pip install iqdb_tagger

3.1 Add bin folder to path (optional)

export PATH=$PATH:/volume1/@appstore/py3k/usr/local/bin

That command line above can also be put on ~/.bashrc, so NAS will run it everytime user login.

Installation

Install it with from pypi

$ pip install iqdb_tagger

Or install it manually

$ git clone https://github.com/softashell/iqdb_tagger.git
$ cd iqdb_tagger
# run the command below
$ python setup.py install
# for windows user: to force it using python3 run following command
$ python -3 setup.py install
# or
$ pip install .

If you are in windows and get SyntaxError, check your python version. To install under python3 follow the instruction on this link https://stackoverflow.com/a/18059129/1766261

FAQ

libxml error in Windows

If you are encounter this error on Windows

Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?

Please follow this guide to install lxml: StackOverflow - how to install lxml on windows?

Contributing

To test the program do the following:

$ # install required package
$ pip install -e ".[dev]"
$ cd docs
$ make test

To upload the new version, do the following:

  1. register to pypi and test.pypi
  2. upgrade setuptools. setuptools>=38.6.0 is required to produce a distribution with the new metadata
  3. make a source distribution. command: python setup.py sdist. in this example it will produce dist/iqdb_tagger-0.3.2.tar.gz
  4. install twine>=1.11.0.
  5. upload first to test.pypi. command: twine upload --repository-url https://test.pypi.org/legacy/ dist/iqdb_tagger-0.3.2.tar.gz
  6. if upload success but result is not as intended, change the version with postn-suffix format. fix the program and go to number 5.
  7. if upload sucsess and result is as intended:
  1. check the program version. maybe rolled it back to original if possible
  2. upload it to pypi. command: twine upload --repository-url https://upload.pypi.org/legacy/ dist/iqdb_tagger-0.3.2.tar.gz

this guideline is based on this guide https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi

Licence

This project is licensed under the MIT License - see the LICENSE file for details

Authors

iqdb_tagger was written by softashell and maintained by Rachmadani Haryono

iqdb_tagger's People

Contributors

da3dsoul avatar rachmadaniharyono avatar softashell avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

iqdb_tagger's Issues

webm

try to screenshot a few parts of the webms and search iqdb-tagger

it could use ffmpeg and use first/middle/last frame of the webm

todo

  • fix link
  • todo from rachmadani/iqdb_tagger
  • minimum similarity
  • other iqdb place

quote from rachmadaniHaryono#3

other output mode

actually, i think something that would do this script some good and see more use. would be allowing the urls or post ids of the sites be saved in text files for the user.

on a per site basis. allowing the user to download the file itself instead of just tag them. which means it will be a bit easier to get better versions if it is better.

urls can be done when the dev adds the import option for the queues maybe.

before then the ids would be better i would think.

tags are great, but its not good if you have a shit copy aswell.

fasttracking getting the same image at better quality should always be a good thing to strive for i think

do think the urls should be saved in a text file per site to split them apart. makes sense now atleast. then again.. that shouldnt be an issue once the dev is done with his stuff.

Why not just on one file but sorted by domain?
easier to control + a and paste queries?
if ids are used then prefixes for "id:" should be used for the sites that support it.
hope the dev manages to do the import part soon though as it would remove the need to parse content as a whole for many things.

"Permission denied" error while trying to create temporary file on Windows

Hello, iqdb_tagger crashes on Windows during file upload procedure. I've been able to fix that on my copy by adding "delete=False" to NamedTemporaryFile parameters but I'm not a python developer so I don't know if this is a good fix or not.

ERROR:flask.app:Exception on / [POST]
Traceback (most recent call last):
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask-1.0.3-py3.7.egg\flask\app.py", line 2311, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask-1.0.3-py3.7.egg\flask\app.py", line 1834, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask_restful-0.3.6-py3.7.egg\flask_restful\__init__.py", line 273, in error_router
    return original_handler(e)
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask-1.0.3-py3.7.egg\flask\app.py", line 1737, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask-1.0.3-py3.7.egg\flask\_compat.py", line 36, in reraise
    raise value
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask-1.0.3-py3.7.egg\flask\app.py", line 1832, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask-1.0.3-py3.7.egg\flask\app.py", line 1818, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask_admin-1.5.3-py3.7.egg\flask_admin\base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\flask_admin-1.5.3-py3.7.egg\flask_admin\base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\iqdb_tagger\views.py", line 33, in index
    form.file.data.save(temp.name)
  File "c:\users\~~~~\appdata\local\programs\python\python37-32\lib\site-packages\werkzeug\datastructures.py", line 2799, in save
    dst = open(dst, "wb")
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\~~~~\\AppData\\Local\\Temp\\tmpc_41uzok'

export function

this function will export the existing database entries to json

requirement:

  • contain version entry (use 1 for now)
  • with output json it should be able to recreate database entry ( import function will be added on other issue)
  • add export command to cli

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.