tool for language identification about opustools HOT 3 CLOSED

add_lang_ids script is now ready. Still need to update opus_read to be able to filter by language language labels.

from opustools.

could we rename the frontend script to opus_langid instead? I'd like to keep the opus prefix for the tools in this package and also like to keep names short. What kind of formats can this script now work with? Does it work with tokenized (the files in xml) and untokenized texts (the ones in raw)? Does it work with both, zipped archives of files and single xml files?

from opustools.

opus_langid works with tokenized and raw xml files in both, single xml files and xml files in zip archives. The script is also now more memory efficient than the first version, as it does not load the whole xml tree into memory at once. opus_read is able to filter by language labels.

from opustools.