hiyori-api / checker_mal Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 0.0 550 KB

maintains an ID cache for MAL anime/manga, with some complimentary web endpoints

Home Page: https://sean.fish/mal_unapproved/

License: MIT License

Elixir 91.74% SCSS 1.23% JavaScript 2.80% HTML 3.57% Shell 0.65%

caching database jikan mal myanimelist

checker_mal's People

Contributors

Stargazers

Watchers

checker_mal's Issues

Add assertion that checks if the pages have the expected entry types

Incase the MAL search interface ever changes, which could cause a bunch of false adds/deletes if the sfw/nsfw pages aren't being requested properly

Before any index runs, should request one of each page for one type, just to make sure this still parses the HTML properly

MAL removed gen page, need to come up with a new strategy

Could include skipping IDs till we find some latest, letting users enter in IDs to check. not sure right now

approved unapproved titles

found 102 approved titles that are still listed in https://sean.fish/mal_unapproved/manga page
some of them was approved month ago

Click to expand

asset basepath configuration

since this is behind a reverse proxy on my server, pheonix asset configuration path conflicts with other base paths

allow a path to be set in config/prod.exs which handles where the base path for assets should be, reflect those in mal_unapproved

General Plan/Feature List

Based off of mal-id-cache, but nicer since I dont have to deal with python async things.

Store state in an db/file locally, see config file for ranges.

General Strategy/improvements from mal_id_cache:

Use search indexed by reverse ID to check ranges of MAL pages, stopping at the page specified by the local database state if we dont find anything.

instead of deleting/rebuilding cache every 'x' days, use the index (this) of all (unapproved/approved entries), and check against the current pages' last ID on the page, to see if we've passed it.

We have to cache the last unapproved anime/manga ID (while its unapproved), since (in mal_id_cache terms), its used as the 'base case' for the -2's page search. If:

entry was denied (its no longer on the MAL index page), we should use the next smallest MAL id as the oldest value, and cache that.
entry was approved (but theres no way to know that, till you actually iterate through search pages till you reach the oldest unapproved entry), then it can be added to known IDs and the oldest unapproved entry can be changed to the next oldest unapproved Id, which would determine the next "till we reach last unapproved entry runs" loop condition.

The names from the mal index page should be parsed, and compared against the known names. Its possible that entries are merged/deleted on MAL (https://myanimelist.net/forum/?topicid=1795522), so in that case, the Hiyori combiner would be sent some sort of message which describes "this may have changed", but its not certain. Hiyori can use the cached attributes to check against the ID that has changed, to determine if it was just a change to the title, or a merging of entries. Define some enumeration that describes how to deal with deleted/merged entries, as opposied to unapproved -> approved entries, as far as updating the global index. This will never work 100% (since an entry could still change without having the MAL name change, and a name can change without the MAL entry having being merged/replaced), but its better than not implementing it at all.

It may be possible to skip pages while iterating, if you know the next ID is more than 50 IDs back (which would be done by sorting a list of unapproved+approved), see the oldest unapproved manga entries, but this isn't necessary. In the best case, it reduces the requests made by 5 every 10 days or so.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

hiyori-api / checker_mal Goto Github PK

checker_mal's People

Contributors

Stargazers

Watchers

checker_mal's Issues

Add assertion that checks if the pages have the expected entry types

MAL removed gen page, need to come up with a new strategy

approved unapproved titles

asset basepath configuration

General Plan/Feature List

A bunch of approved entries are on the list with (???) in place of their name

add CLI interface to modify PageState

unapproved page limit faulty

save ids to json so its avalable during boot

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent