Provide a script to fill the local development database with data from Tournesol's public database about tournesol-meta HOT 5 CLOSED

Assigned @jstainer-kleis because I believe you already had such a script laying around. Please unassign yourself otherwise :)

from tournesol-meta.

I have a quick and dirty set of scripts. It allows:

• importing the public dataset that used to be distributed into the database with the new schema (from tournesol-backend repository)
• fetching the metadata of referenced videos (for now I only insert the title in DB, but the full metadata is retrieved)
• dumping SQL commands that could be directly used to fill the DB (without going through the previous steps)
• exporting a tarball containing the videos metadata

I'm not sure what and how to commit though.

• should I put the original .csv files somewhere?
• should I push the scripts to fill the DB from these or just the SQL file?
• should I put the tarball with the videos metadata somewhere? just the scripts to fetch / export them?
• should I insert more metadata in the video table (i.e. duration or things like that instead of just the title)?

If the public dataset distribution format is kept as-is for the foreseeable future I guess that the scripts should be made more robust but the SQL file may be more convenient. However, in the current form it contains data crawled from Youtube and I'm not sure we're allowed to distribute that. The same question arises for the videos metadata tarball, maybe even for the script to fetch them using youtube-dl...

IANAL, before pushing anything I'll wait for an informed PoV (@mahdi ?) on this topic.

from tournesol-meta.

Edited:

NOT Blocked by #45

from tournesol-meta.

Blocked by #45

I don't think it's a blocker, migration from the current schema to the future one will be easier than modifying the scripts to adapt them to the new schema.

from tournesol-meta.

Correct! I agree

from tournesol-meta.