hufman / vgmdb Goto Github PK

A custom frontend for vgmdb, providing JSON and RDF

Shell 0.10% Perl 0.08% Python 5.44% JavaScript 0.04% CSS 0.06% HTML 94.11% RAML 0.17%

vgmdb's Introduction

VGMdb

VGMdb.net is an excellent resource, containing gigabytes of information about video game music. It documents the relationships between games, albums that represent games, and all the people that contributed to an album.

However, VGMdb does not provide an API to access its information, which provides an obstacle to programs that want to use its information. This site is a way to programmatically access the information on VGMdb.net.

VGMdb.net stores the information of each item at a friendly URL, such as album/79 or artist/137. This friendly url will work on this site. Once there, an overview of the item's information can be found, along with a few extra features.

Usage

In the right sidebar, there are links to view the page's information in different formats. The raw information that has been parsed out of the original VGMdb.net item is available in JSON format by adding ?format=json to any URL. The YAML format is also available for this information, available by adding ?format=yaml.

Additionally, an effort has been made to produce RDF tuples representing some of the information in VGMdb. The default HTML view of the data has RDFa encoded into it, allowing any RDFa library to parse the page. Additionally, RDF/XML and Turtle serialization formats are available.

The page will produce the output format that most-closely matches the request's HTTP Accept header. A particular format can be forced by adding ?format=html, ?format=xml, or ?format=turtle arguments to the URL

There are also links to validation services, to verify that the page's data is in fact valid and parseable in the respective formats. The main page doesn't produce any other formats, but every other page does.

Development and Installation

Unit tests are employed, both in the parsing and in the output stages, to validate that the information is being correctly parsed and encoded. Run the tests.sh script to run the tests.

Most of any runtime requirements should be documented in requirements.txt. Use pip install -r requirements.txt to install them.

The run.py example script will start up an HTTP server process for the site. An example Apache2 configuration, using mod_wsgi, is also provided. It should work just as well under any other WSGI server.

The docker/docker-build.sh script should produce a vgmdb_reqs base image with the dependencies and a vgmdb image with the current version of the code installed. This image exposes port 80, and vgmdb/autoload_settings.py implements loading optional dependencies and settings based on environment variables.

vgmdb's People

Contributors

Stargazers

Watchers

Forkers

joyeusenoelle qood timlehr mahoyo jprjr nicholasyoannou gakireitsuki seyys something123something arpitpandey992 mertvn subarashiirx

vgmdb's Issues

Is it possible to add in Publish Format?

Is it possible to add in Publish Format?
here: http://vgmdb.info/album/79

also sometimes there is Label and Publisher, but appears only as Publisher, is it possible to separate this?

thank you for your great work.

Continuously test against actual vgmdb.net html

Right now the unit tests check against saved snapshots of specific vgmdb.net pages. It should be possible to automatically update those snapshots and run the unit tests against them, to make sure that they still handle the current vgmdb.net html.
Additionally, it should crawl around vgmdb.net and look for any parse errors and crashes automatically.

JSON returns incorrect store data

For Album ID-4 - "CHRONO CROSS ORIGINAL SOUNDTRACK":

http://vgmdb.info/album/4 - On the album page and the /sellers endpoint, it returns product pages for Discogs, CDJapan and Spotify and a search on Amazon.

http://vgmdb.info/album/4?format=json - But when looking at the JSON, the stores data returns:
"stores": [
{
"link": "/db/collection.php?do=show&albumid=4",
"name": "5 for sale "
}
]

It looks like its returning the # of albums up for sale in the Marketplace on vgmdb.net's album page. Shouldn't the store data return links/names for Discogds, CDJapan and Spotify?

But on https://vgmdb.net/album/4

it has it listed for Amazon.co.jp, Amazon.com, Play-Asia (OOP)
these are different the ones being displayed on the page (Discogs, CDJapan and Spotify)

For Album ID-270 - '"Bomberman Hero" Original Soundtrack':

http://vgmdb.info/album/270 - On the album page and the /sellers endpoint, it returns searches on Discogs, Amazon, CDJapan and Spotify.

http://vgmdb.info/album/270?format=json - But when looking at the JSON, the stores data returns:
"stores": [
{
"link": "/db/collection.php?do=show&albumid=270",
"name": "1 for sale "
}
]

On https://vgmdb.net/album/4 - it has it listed for Amazon.co.jp, Amazon.com, Play-Asia (OOP)

For Album ID-76054 - "Divinity: Original Sin II Original Soundtrack":

http://vgmdb.info/album/76054/ - On the album page and the /sellers endpoint, it returns searches on Discogs, Amazon, CDJapan and Spotify and a product page for Apple Music.

http://vgmdb.info/album/76054?format=json - But when looking at the JSON, the stores data returns:
"stores": [
{
"link": "http://itunes.apple.com/album/divinity-original-sin-2-original-soundtrack/1357654190",
"name": "iTunes"
},
{
"link": "http://play.google.com/store/music/album/Borislav_Slavov_Divinity_Original_Sin_2_Original_S?id=B7v2nxxwqmweffhsyuuhsyuexki",
"name": "Google Play"
},
{
"link": "http://www.amazon.com/exec/obidos/ASIN/B07BC4YR7S/",
"name": "Amazon"
}
]

For this last album, the stores being returned are the ones listed at https://vgmdb.net/album/76054 under the "Available at" section.

this is different from the one being shown on the album page (only a product page for Apple Music and searches for others)

Overall:

Marketplace results has a higher priority than the stores listed on vgmdb.net under "Available at" (which is being returned under stores in JSON)
If there are no marketplace results, it returns the stores listed on vgmdb.net under "Available at"
There is an undocumented way of retrieving stores that has higher priority than the marketplace results and the stores listed on vgmdb.net under "Available at" (as seen on the Chrono Cross page - album 4 and the Divinity page - album 76054)

Bandcamp link parsing

Some of the info pages have Bandcamp links, perhaps they could be exposed in the data

Wish to add advanced search function

Right now I can only search by one keyword, such as searching for albums, artists, and organizations individually.
When the album name is more common, the number of search results can be very large.
I wish to add an advanced search function to post two or three search keys at the same time (just like https://vgmdb.net/search )

Error 502

just reporting vgmdb.info is down. error 502 bad gateway.

Album -> Album can have n events

Example: http://vgmdb.net/album/35753

You only parse 1 event :) ( http://vgmdb.info/album/35753?format=json )

/product/{id} doesn't show any album

The product endpoint doesn't show any album

https://vgmdb.info/product/23?format=json

{ "albums": [], "description": "", "link": "product/23", "meta": { "added_date": "2009-03-22T21:27", "edited_date": "2015-09-08T09:30", "fetched_date": "2020-10-20T19:42", "ttl": 86400, "visitors": 20503 }, "name": "Final Fantasy", "name_real": "ファイナルファンタジー", "picture_full": "https://vgmdb.net/", "picture_small": "https://vgmdb.net/db/img/collapse_close.png", "type": "Franchise", "vgmdb_link": "https://vgmdb.net/product/23?perpage=99999", "websites": {} }

Return Json without line breaks (i.e. no pretty printer)

Hello,
This is more as a feature request rather than a bug.
Would it be an option to receive Json result in non pretty printer format?
Having line breaks in the result creates some issue in Mp3Tag, not because Mp3Tag cannot parse Json, but because we try to add some missing information in the json stream (in pure string manipulation) and there Mp3Tag has an issue removing line breaks.
The only option it works is to predict the exact number of line breaks in Mp3Tag, which is impossible of course.
Thanks a lot.

"Reprint" field in albums from /product is always FALSE

Reprint is always FALSE, even there's a Child albums

Album -> Publisher can be a list too

http://vgmdb.net/album/49938

http://vgmdb.info/album/49938?format=json

Add Docker build instructions

Hi @hufman,

I'd like to contribute to the API by adding some additional parsers and fixing some stuff, as I need those changes for my own project. I am trying to get the docker images of my code build using docker-build.sh script, but it doesn't succeed as the requirements image isn't found correctly. Can you update the development instructions to include docker?

Thank you!

build error

➜  ./docker/docker-build.sh                                                                                                                                                                                                                                         
given is experimental at docker/docker-compile.pl line 47.
when is experimental at docker/docker-compile.pl line 49.
when is experimental at docker/docker-compile.pl line 50.
when is experimental at docker/docker-compile.pl line 51.
given is experimental at docker/docker-compile.pl line 65.
when is experimental at docker/docker-compile.pl line 66.
when is experimental at docker/docker-compile.pl line 71.
when is experimental at docker/docker-compile.pl line 76.
when is experimental at docker/docker-compile.pl line 92.
when is experimental at docker/docker-compile.pl line 93.
when is experimental at docker/docker-compile.pl line 94.
when is experimental at docker/docker-compile.pl line 95.
when is experimental at docker/docker-compile.pl line 96.
when is experimental at docker/docker-compile.pl line 97.
when is experimental at docker/docker-compile.pl line 98.
when is experimental at docker/docker-compile.pl line 103.
Use of uninitialized value $v in concatenation (.) or string at docker/docker-compile.pl line 100, <DOCKERFILE> line 4.
Use of uninitialized value $v in concatenation (.) or string at docker/docker-compile.pl line 101, <DOCKERFILE> line 4.
*** docker run --cidfile=/var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/5lmtvEn_Qy/CID -v /var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/5lmtvEn_Qy:/.data phusion/baseimage:0.9.15 /.data/setup.sh
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/.data/setup.sh\": stat /.data/setup.sh: no such file or directory": unknown.
Died at docker/docker-compile.pl line 118.
sed: 1: "./Dockerfile.app": invalid command code .
given is experimental at docker/docker-compile.pl line 47.
when is experimental at docker/docker-compile.pl line 49.
when is experimental at docker/docker-compile.pl line 50.
when is experimental at docker/docker-compile.pl line 51.
given is experimental at docker/docker-compile.pl line 65.
when is experimental at docker/docker-compile.pl line 66.
when is experimental at docker/docker-compile.pl line 71.
when is experimental at docker/docker-compile.pl line 76.
when is experimental at docker/docker-compile.pl line 92.
when is experimental at docker/docker-compile.pl line 93.
when is experimental at docker/docker-compile.pl line 94.
when is experimental at docker/docker-compile.pl line 95.
when is experimental at docker/docker-compile.pl line 96.
when is experimental at docker/docker-compile.pl line 97.
when is experimental at docker/docker-compile.pl line 98.
when is experimental at docker/docker-compile.pl line 103.
*** docker run --cidfile=/var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/W188yTkwvy/CID -v /var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/W188yTkwvy:/.data vgmdb_reqs:36a64e /.data/setup.sh
Unable to find image 'vgmdb_reqs:36a64e' locally
docker: Error response from daemon: pull access denied for vgmdb_reqs, repository does not exist or may require 'docker login'.
See 'docker run --help'.
Died at docker/docker-compile.pl line 118.

how to build it?
macOS

500 Internal server errors

I am currently getting 500 Internal Server Errors on different endpoints, e.g.
http://vgmdb.info/search/albums/SVWC-7965?format=json
http://vgmdb.info/album/36309?format=json

It happens with and without the json query parameter.

500 errors

Keep seeing server errors when hitting http://vgmdb.info/album/67253. It seems to usually work the first time, then any successive call will yield a 500. If I wait for a few minutes it seems to stabilize again and then will succeed.

HTTP 500 for some pages

Hi,
I some pages are throwing http 500 errors. This doesn't seem to be a side wide issue but related to specific pages e.g.
https://vgmdb.info/album/107393
https://vgmdb.info/album/107565
https://vgmdb.info/album/107562

Search -> product 'aliases' returns a giant string as alias instead of an array

Example: http://vgmdb.info/search/products?q=haruhi&format=json

"products": [
{
"aliases": [
"Haruhi Suzumiya涼宮ハルヒHaruhi Suzumiya"
],
"link": "product/1364",
"names": {}
},
{
"aliases": [
"Suzumiya Haruhi no Chokuretsu涼宮ハルヒの直列Suzumiya Haruhi no Chokuretsu"
],
"link": "product/2739",
"names": {}
},

Set up Github Actions

As opposed to the current local Jenkins build, the unit tests and Docker image build should be implemented with Github Actions.

Date dont show in json

This album:

the jon:

Event name in Japanese

currently it's missing
e.g.
https://vgmdb.info/event/200
https://vgmdb.net/event/200

missing:

{
...
    "name_jp": "博麗神社例大祭14",
...
}

Artist -> 'aliases' -> strange behaviour

Clarification:

if alias == 1
--> alias.splitAfterEveryLetter

if alias > 1
--> perfekt split

Example:
(Single value) http://vgmdb.info/artist/4?format=json
(Multi value) http://vgmdb.info/artist/5?format=json

Error 500 on specific artist

https://vgmdb.info/artist/36 that gives me error 500

Artist -> duplicated entries & Feature Request "Artist.isPerson/isBand[Group/Unit]"

Example: http://vgmdb.info/artist/15942?format=json

Members is duplicated in Artist .Members & Artist .info.members ( same as Artist.aliases, Artist.Units & Artist.Birthdate ) Do you parse BirthLocation?
"info": {
......
"Members": [
{
"names": {
"en": "RYOTA"
}
},
......
},
"link": "artist/15942",
"members": [
{
"names": {
"en": "RYOTA"
}
},
......
{
"names": {
"en": "U"
}
},
......
],

Feature Request "Artist.isPerson/isBand[Group/Unit]"

Next to the Artist.name is a "(Unit)" in the html. This "field?" ( http://vgmdb.net/artist/6310?format=json ) means this artist is a group/band/unit, am I right? It is possible that the artist could have a boolean field for this?

Queries failing possibly due to new HTTPS feature

It's almost impossible to search anything anymore through the API. The reason might possibly be that VGMdb has switched over to auto-redirecting to HTTPS. Some queries that contain two or more words do still work for some reason. Example: http://vgmdb.info/search/bright burning shout

Another problem I've found is when going through the recent/albums JSON: the contents of the link object changed from the usual album/<id> to https://vgmdb.net/album/<id>. I suppose this might be what conflicts the querying as well in the underlying code.

Hope this could be fixed soon. Thanks in advance.

Album -> related albums 'catalog' returns the album's name instead of the catalog

Example: http://vgmdb.info/album/30552?format=json ( http://vgmdb.net/album/30552 )

"related": [
{
"catalog": "Bouken De-show De-show? / Aya Hirano",
"link": "album/30550",
"names": {
"en": "Bouken De-show De-show? / Aya Hirano",
"ja": "冒険でしょでしょ？ / 平野綾",
"ja-latn": "Bouken Desho Desho? / Aya Hirano"
},
"type": "anime"
},

500 Errors - Various Albums

I am getting 500 errors on a bunch of albums.

Not sure if it's just coincidence or not, but all of them are 'Vinyl' albums.

Some of the error albums:

http://vgmdb.info/album/76367 - 2 Vinyl - https://vgmdb.net/album/76367
http://vgmdb.info/album/72682 - 2 Vinyl - https://vgmdb.net/album/72682
http://vgmdb.info/album/62524 - 2 Vinyl - http://vgmdb.net/album/62524
http://vgmdb.info/album/77167 - 2 Vinyl - http://vgmdb.net/album/77167
http://vgmdb.info/album/72776 - 2 Vinyl - http://vgmdb.net/album/72776
http://vgmdb.info/album/76197 - 2 Vinyl - http://vgmdb.net/album/76197
http://vgmdb.info/album/82450 - 2 Vinyl - http://vgmdb.net/album/82450
http://vgmdb.info/album/52309 - 1 Vinyl - http://vgmdb.net/album/52309
http://vgmdb.info/album/79163 - 1 Vinyl - http://vgmdb.net/album/79163
http://vgmdb.info/album/102096 - 3 Vinyl - http://vgmdb.net/album/102096
http://vgmdb.info/album/102093 - 3 Vinyl - http://vgmdb.net/album/102093

I tried going through other Vinyl albums and did find a '1 Vinyl' album that was working (http://vgmdb.info/album/89792) so not all Vinyls are returning 500 errors.

NSFW Images Filtering/Removing

Do you think it would be possible to filter/remove the placeholder image or can you login with the craller to remove such "problem"?

http://vgmdb.net/album/50766
http://vgmdb.info/album/50766?format=json

Something is wrong in the search

Products -> don't have websites?

Just a question, a lot of products have websites ( official, reference (...?) ), do you mind to add them in the schema? 👍

Example:
product.type=game http://vgmdb.net/product/1019
product.type=video http://vgmdb.net/product/1022

Not all product.types have websites

Fetch single event throws 500 Error

https://vgmdb.info/event/28

Crash if no User-Agent in the request

curl -v -H User-Agent: http://vgmdb.info/album/79?format=json

Consistent 500 error for artist/174

http://vgmdb.info/artist/174 consistently returns a 500 for any output format.

All other artists I've tried are working fine.

Error 502

Something is wrong.

Error 500 since around 17/02/2019

I've been using vgmdb.info/album calls for a while now. A few days ago, I started getting "Error 500 Internal Server Error" systematically.

This happens with every call (except with the example album id 79, which I guess is cached or processed differently)

Example: http://vgmdb.info/album/84543

Artist -> info.Variations inconsistent

Sometimes variations return an array of strings of just a simple string. It is possible to add consistent to the variations?

1 Variation:

"Variations": "白河ことり"

http://vgmdb.info/artist/640?format=json

n Variations:

"Variations": [
"KUKEIHAKURABU",
"Konami",
"Club Kukeiha"
]

http://vgmdb.info/artist/161?format=json

API doesn't include barcode

I'm using json data of albums from the API. As far as I can tell, it doesn't include the barcode, not sure why. I would appreciate if it were to be added!

Product query generated on album pages leads to 0 results

Problem encountered with Tetris: The Grand Master and Else Heart.Break() despite launching the query from the album page directly.

They change the site again

so the people don't appear in Json.

Twitter handle parsing is broke

http://vgmdb.net/artist/77
Buried inside the source code is the twitter handle, but the parsing code hasn't caught up to the new location.

Link error

Any link here gives an error:

Is it possible to add in Albums Organizations?

Is it possible to add in Albums the Organizations, label per example?
here: http://vgmdb.info/album/79

Json Validation

Something is wrong in the json validation for an album:

The people are blank

The people are blank, i think they change stuff

I think they change the html and the names:

Redis out of memory

I'm using this project, and after making some petitions, it says something like:

ResponseError: OOM command not allowed when used memory > 'maxmemory'.

How can I flush the data, in order to continue performing petitions?

Stores field not fully populated

The stores field only includes the VGMDB marketplace entry and does not include other stores (e.g. Spotify, Amazon, etc.)

For example, the Octopath Traveler soundtrack has 7 stores but the Album API response only returns the marketplace.

    "stores": [
        {
            "link": "/db/collection.php?do=show&albumid=77081",
            "name": "1 for sale "
        }
    ],

I believe this is because the _parse_section_stores method sets recursive to false, so the links further down the tree are not identified.

Love the project, keep up the great work!

New tracklists are not synced correctly

Hi,
it seems that in some cases, tracklist information is not updated for existing albums, or at least with a delay of several days, e.g.

In all of the examples above, the disc section only contains japanese names and is missing tracklists in other languages from its respective vgmdb.net counterpart.

Artist -> 'Former Member' without links