oscar-project / oscar-tools Goto Github PK

Cli documentation is very lackluster, we need to add more details about operations.

Describe the bug
Error when using extract-text on Japanese dataset for OSCAR 2201 (v2).

To Reproduce
Steps to reproduce the behavior:

1. Ran oscar-tools v2 extract-text ja_meta_part_1.jsonl.gz ja_part_1.txt
2. Gets: Error: Io(Error { kind: InvalidData, message: "stream did not contain valid UTF-8" })
3. Produces empty destination file.

Expected behavior
Expected an OSCAR-2019 (v1) text format

Desktop (please complete the following information):

• OS: Linux 4.19.0-16-amd64

We should print an error and then go into the next file rather than stopping everything.

Since the new pipeline outputs a unique .jsonl file containing both textual content and metadata, working exclusively with textual data may be less comfortable:

1. Forced download of metadata that will be discarded
2. Forced usage of a jsonl-compatible parser in order to extract content

For now, we should provide a tool that splits up a given OSCAR Schema v2 file into a OSCAR Schema v1.2-compatible file.

The package command is in two steps:

• Create one folder per language, and move splits into these folders
• Compute sha256sums of each split, putting them in a sha256sum file.

The command name might be ambiguous, and we might have a way of doing all of this better.

This is the command to rebuild the corpus from rebuild (avro) files.
It might be trickier to port because it has more dependencies, iirc.

Code is here: https://github.com/oscar-project/ungoliant/blob/v1.2.3/src/processing/rebuild.rs

Describe the bug
Wrong order of documents in split op

To Reproduce
Steps to reproduce the behavior:
split file in multiple splits

Expected behavior
part one has the firtst docs, and so on

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Compress attempts to compress at max_depth=1, but splitting puts each language in a folder.

For now, each operation implementation needs some boilerplate code to generate a CLI command and options.

Implementing a specific operation on a corpus version should automatically add the said operation into the operations list.

This should however constrain the user to choose explicitly an OSCAR Schema version. We could add a versions subcommand listing the different OSCAR Schema versions and their corpora:

OSCAR Schema v1:     19XX
OSCAR Schema v1.1:   2109
OSCAR Schema v2:     2201