m4t1ss / parallel-corpora-tools Goto Github PK

Remove sentences where the number of non-space characters is equal (or very close?) to the number of tokens.

English

( c o n t i n u a t i o n )

Slovenian

( n a d a l j e v a n j e )

The language identification is the single slowest part out of all these scripts.
It would be best to give it the least amount of data to process...

Also, specify a temporary directory location for sort in the unique-parallel script.

Hi ,
Thanks for the nice tool. I tried using the non-repeating token filter repeating-tokens.php for cleaning my back translated data. But the resultant files I am getting are not parallel corpora. I am not sure if I did any mistake.
Initial parallel corpora size - 836843 lines ( of both source and target language)
after filtering
source file - 734928 lines
target file - 746393 lines

The command I used was
php repeating-tokens.php source_corpus target_corpus

The input files could be split into parts as many as available cores and the results could be concatenated after all parts are processed

Do not remove ALL source sentences that align to multiple target sentences and target sentences align to multiple source sentences! Keep the first one 👍