Git Product home page Git Product logo

vgmdb's Introduction

VGMdb

Docker Pulls

VGMdb.net is an excellent resource, containing gigabytes of information about video game music. It documents the relationships between games, albums that represent games, and all the people that contributed to an album.

However, VGMdb does not provide an API to access its information, which provides an obstacle to programs that want to use its information. This site is a way to programmatically access the information on VGMdb.net.

VGMdb.net stores the information of each item at a friendly URL, such as album/79 or artist/137. This friendly url will work on this site. Once there, an overview of the item's information can be found, along with a few extra features.

Usage

In the right sidebar, there are links to view the page's information in different formats. The raw information that has been parsed out of the original VGMdb.net item is available in JSON format by adding ?format=json to any URL. The YAML format is also available for this information, available by adding ?format=yaml.

Additionally, an effort has been made to produce RDF tuples representing some of the information in VGMdb. The default HTML view of the data has RDFa encoded into it, allowing any RDFa library to parse the page. Additionally, RDF/XML and Turtle serialization formats are available.

The page will produce the output format that most-closely matches the request's HTTP Accept header. A particular format can be forced by adding ?format=html, ?format=xml, or ?format=turtle arguments to the URL

There are also links to validation services, to verify that the page's data is in fact valid and parseable in the respective formats. The main page doesn't produce any other formats, but every other page does.

Development and Installation

Unit tests are employed, both in the parsing and in the output stages, to validate that the information is being correctly parsed and encoded. Run the tests.sh script to run the tests.

Most of any runtime requirements should be documented in requirements.txt. Use pip install -r requirements.txt to install them.

The run.py example script will start up an HTTP server process for the site. An example Apache2 configuration, using mod_wsgi, is also provided. It should work just as well under any other WSGI server.

The docker/docker-build.sh script should produce a vgmdb_reqs base image with the dependencies and a vgmdb image with the current version of the code installed. This image exposes port 80, and vgmdb/autoload_settings.py implements loading optional dependencies and settings based on environment variables.

vgmdb's People

Contributors

dependabot[bot] avatar hufman avatar jprjr avatar mertvn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vgmdb's Issues

500 errors

Keep seeing server errors when hitting http://vgmdb.info/album/67253. It seems to usually work the first time, then any successive call will yield a 500. If I wait for a few minutes it seems to stabilize again and then will succeed.

500 Errors - Various Albums

Redis out of memory

I'm using this project, and after making some petitions, it says something like:

ResponseError: OOM command not allowed when used memory > 'maxmemory'.

How can I flush the data, in order to continue performing petitions?

Artist -> duplicated entries & Feature Request "Artist.isPerson/isBand[Group/Unit]"

Example: http://vgmdb.info/artist/15942?format=json

Members is duplicated in Artist .Members & Artist .info.members ( same as Artist.aliases, Artist.Units & Artist.Birthdate ) Do you parse BirthLocation?
"info": {
......
"Members": [
{
"names": {
"en": "RYOTA"
}
},
......
},
"link": "artist/15942",
"members": [
{
"names": {
"en": "RYOTA"
}
},
......
{
"names": {
"en": "U"
}
},
......
],

Feature Request "Artist.isPerson/isBand[Group/Unit]"

Next to the Artist.name is a "(Unit)" in the html. This "field?" ( http://vgmdb.net/artist/6310?format=json ) means this artist is a group/band/unit, am I right? It is possible that the artist could have a boolean field for this?

Add Docker build instructions

Hi @hufman,

I'd like to contribute to the API by adding some additional parsers and fixing some stuff, as I need those changes for my own project. I am trying to get the docker images of my code build using docker-build.sh script, but it doesn't succeed as the requirements image isn't found correctly. Can you update the development instructions to include docker?

Thank you!

API doesn't include barcode

I'm using json data of albums from the API. As far as I can tell, it doesn't include the barcode, not sure why. I would appreciate if it were to be added!

Stores field not fully populated

The stores field only includes the VGMDB marketplace entry and does not include other stores (e.g. Spotify, Amazon, etc.)

For example, the Octopath Traveler soundtrack has 7 stores but the Album API response only returns the marketplace.

    "stores": [
        {
            "link": "/db/collection.php?do=show&albumid=77081",
            "name": "1 for sale "
        }
    ],

I believe this is because the _parse_section_stores method sets recursive to false, so the links further down the tree are not identified.

Love the project, keep up the great work!

Artist -> 'Former Member' without links

Example: http://vgmdb.info/artist/15942?format=json

"info": {
"Album Votes": "9 votes for 3 albums",
"Credited works": [
"9 albums in database",
"9 composer credits",
"9 arranger credits",
"9 performer credits",
"0 lyricist credits"
],
"Formed": "2006",
"Former Members": [
"2RASH",
"ko-sk",
"TAKUMi",
"TSUYOSHi"
],
"Members": [
"RYOTA",
"U",
{
"link": "artist/15569",
"names": {
"en": "SHiNNOSUKE"
}
}
]

I think

"members": [
    {
        "names": {
            "en": "2RASH"
        }
    }

would be a little bit cleaner? :)

/product/{id} doesn't show any album

The product endpoint doesn't show any album

https://vgmdb.info/product/23?format=json

{ "albums": [], "description": "", "link": "product/23", "meta": { "added_date": "2009-03-22T21:27", "edited_date": "2015-09-08T09:30", "fetched_date": "2020-10-20T19:42", "ttl": 86400, "visitors": 20503 }, "name": "Final Fantasy", "name_real": "ファイナルファンタジー", "picture_full": "https://vgmdb.net/", "picture_small": "https://vgmdb.net/db/img/collapse_close.png", "type": "Franchise", "vgmdb_link": "https://vgmdb.net/product/23?perpage=99999", "websites": {} }

Bandcamp link parsing

Some of the info pages have Bandcamp links, perhaps they could be exposed in the data

Wish to add advanced search function

Right now I can only search by one keyword, such as searching for albums, artists, and organizations individually.
When the album name is more common, the number of search results can be very large.
I wish to add an advanced search function to post two or three search keys at the same time (just like https://vgmdb.net/search )

Continuously test against actual vgmdb.net html

Right now the unit tests check against saved snapshots of specific vgmdb.net pages. It should be possible to automatically update those snapshots and run the unit tests against them, to make sure that they still handle the current vgmdb.net html.
Additionally, it should crawl around vgmdb.net and look for any parse errors and crashes automatically.

Error 502

just reporting vgmdb.info is down. error 502 bad gateway.

Queries failing possibly due to new HTTPS feature

It's almost impossible to search anything anymore through the API. The reason might possibly be that VGMdb has switched over to auto-redirecting to HTTPS. Some queries that contain two or more words do still work for some reason. Example: http://vgmdb.info/search/bright burning shout

Another problem I've found is when going through the recent/albums JSON: the contents of the link object changed from the usual album/<id> to https://vgmdb.net/album/<id>. I suppose this might be what conflicts the querying as well in the underlying code.

Hope this could be fixed soon. Thanks in advance.

JSON returns incorrect store data

For Album ID-4 - "CHRONO CROSS ORIGINAL SOUNDTRACK":

http://vgmdb.info/album/4 - On the album page and the /sellers endpoint, it returns product pages for Discogs, CDJapan and Spotify and a search on Amazon.

http://vgmdb.info/album/4?format=json - But when looking at the JSON, the stores data returns:
"stores": [
{
"link": "/db/collection.php?do=show&albumid=4",
"name": "5 for sale "
}
]

It looks like its returning the # of albums up for sale in the Marketplace on vgmdb.net's album page. Shouldn't the store data return links/names for Discogds, CDJapan and Spotify?

But on https://vgmdb.net/album/4

  • it has it listed for Amazon.co.jp, Amazon.com, Play-Asia (OOP)
  • these are different the ones being displayed on the page (Discogs, CDJapan and Spotify)

For Album ID-270 - '"Bomberman Hero" Original Soundtrack':

http://vgmdb.info/album/270 - On the album page and the /sellers endpoint, it returns searches on Discogs, Amazon, CDJapan and Spotify.

http://vgmdb.info/album/270?format=json - But when looking at the JSON, the stores data returns:
"stores": [
{
"link": "/db/collection.php?do=show&albumid=270",
"name": "1 for sale "
}
]

On https://vgmdb.net/album/4 - it has it listed for Amazon.co.jp, Amazon.com, Play-Asia (OOP)

For Album ID-76054 - "Divinity: Original Sin II Original Soundtrack":

http://vgmdb.info/album/76054/ - On the album page and the /sellers endpoint, it returns searches on Discogs, Amazon, CDJapan and Spotify and a product page for Apple Music.

http://vgmdb.info/album/76054?format=json - But when looking at the JSON, the stores data returns:
"stores": [
{
"link": "http://itunes.apple.com/album/divinity-original-sin-2-original-soundtrack/1357654190",
"name": "iTunes"
},
{
"link": "http://play.google.com/store/music/album/Borislav_Slavov_Divinity_Original_Sin_2_Original_S?id=B7v2nxxwqmweffhsyuuhsyuexki",
"name": "Google Play"
},
{
"link": "http://www.amazon.com/exec/obidos/ASIN/B07BC4YR7S/",
"name": "Amazon"
}
]

For this last album, the stores being returned are the ones listed at https://vgmdb.net/album/76054 under the "Available at" section.

  • this is different from the one being shown on the album page (only a product page for Apple Music and searches for others)

Overall:

  • Marketplace results has a higher priority than the stores listed on vgmdb.net under "Available at" (which is being returned under stores in JSON)
  • If there are no marketplace results, it returns the stores listed on vgmdb.net under "Available at"
  • There is an undocumented way of retrieving stores that has higher priority than the marketplace results and the stores listed on vgmdb.net under "Available at" (as seen on the Chrono Cross page - album 4 and the Divinity page - album 76054)

build error

➜  ./docker/docker-build.sh                                                                                                                                                                                                                                         
given is experimental at docker/docker-compile.pl line 47.
when is experimental at docker/docker-compile.pl line 49.
when is experimental at docker/docker-compile.pl line 50.
when is experimental at docker/docker-compile.pl line 51.
given is experimental at docker/docker-compile.pl line 65.
when is experimental at docker/docker-compile.pl line 66.
when is experimental at docker/docker-compile.pl line 71.
when is experimental at docker/docker-compile.pl line 76.
when is experimental at docker/docker-compile.pl line 92.
when is experimental at docker/docker-compile.pl line 93.
when is experimental at docker/docker-compile.pl line 94.
when is experimental at docker/docker-compile.pl line 95.
when is experimental at docker/docker-compile.pl line 96.
when is experimental at docker/docker-compile.pl line 97.
when is experimental at docker/docker-compile.pl line 98.
when is experimental at docker/docker-compile.pl line 103.
Use of uninitialized value $v in concatenation (.) or string at docker/docker-compile.pl line 100, <DOCKERFILE> line 4.
Use of uninitialized value $v in concatenation (.) or string at docker/docker-compile.pl line 101, <DOCKERFILE> line 4.
*** docker run --cidfile=/var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/5lmtvEn_Qy/CID -v /var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/5lmtvEn_Qy:/.data phusion/baseimage:0.9.15 /.data/setup.sh
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/.data/setup.sh\": stat /.data/setup.sh: no such file or directory": unknown.
Died at docker/docker-compile.pl line 118.
sed: 1: "./Dockerfile.app": invalid command code .
given is experimental at docker/docker-compile.pl line 47.
when is experimental at docker/docker-compile.pl line 49.
when is experimental at docker/docker-compile.pl line 50.
when is experimental at docker/docker-compile.pl line 51.
given is experimental at docker/docker-compile.pl line 65.
when is experimental at docker/docker-compile.pl line 66.
when is experimental at docker/docker-compile.pl line 71.
when is experimental at docker/docker-compile.pl line 76.
when is experimental at docker/docker-compile.pl line 92.
when is experimental at docker/docker-compile.pl line 93.
when is experimental at docker/docker-compile.pl line 94.
when is experimental at docker/docker-compile.pl line 95.
when is experimental at docker/docker-compile.pl line 96.
when is experimental at docker/docker-compile.pl line 97.
when is experimental at docker/docker-compile.pl line 98.
when is experimental at docker/docker-compile.pl line 103.
*** docker run --cidfile=/var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/W188yTkwvy/CID -v /var/folders/29/4sf473ms063cqbcvbvgw5dtc0000gn/T/W188yTkwvy:/.data vgmdb_reqs:36a64e /.data/setup.sh
Unable to find image 'vgmdb_reqs:36a64e' locally
docker: Error response from daemon: pull access denied for vgmdb_reqs, repository does not exist or may require 'docker login'.
See 'docker run --help'.
Died at docker/docker-compile.pl line 118.

how to build it?
macOS

Error 500 since around 17/02/2019

I've been using vgmdb.info/album calls for a while now. A few days ago, I started getting "Error 500 Internal Server Error" systematically.

This happens with every call (except with the example album id 79, which I guess is cached or processed differently)

Example: http://vgmdb.info/album/84543

Set up Github Actions

As opposed to the current local Jenkins build, the unit tests and Docker image build should be implemented with Github Actions.

Return Json without line breaks (i.e. no pretty printer)

Hello,
This is more as a feature request rather than a bug.
Would it be an option to receive Json result in non pretty printer format?
Having line breaks in the result creates some issue in Mp3Tag, not because Mp3Tag cannot parse Json, but because we try to add some missing information in the json stream (in pure string manipulation) and there Mp3Tag has an issue removing line breaks.
The only option it works is to predict the exact number of line breaks in Mp3Tag, which is impossible of course.
Thanks a lot.

The people are blank

The people are blank, i think they change stuff


imagem

I think they change the html and the names:
imagem

Information

@hufman I don't now much about Json, i just ask if it is possible the Length to be separate in tracks, as it is joined in names?

imagem

imagem

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.