paracrawl / b64filter Goto Github PK
View Code? Open in Web Editor NEWProgram for operating on one document per Base 64 encoded line files
Program for operating on one document per Base 64 encoded line files
For some input, e.g.
csd3:/rds/project/t2_vol4/rds-t2-cs119/internet_archive/wide00006-shards/mt/78/1/sentences.gz
the program deadlocks when given
echo 'while ($var = <STDIN>) { print($var) }' > cat.perl
gzip -dc sentences.gz | b64filter -q 1 perl cat.perl > /dev/null
however
gzip -dc sentences.gz | b64filter -q 1 cat > /dev/null
works just fine
Hi. I am having an issue using b64filter
in some Catalan tests with Apertium. Looks like given some kind of lines, b64filter
splits them. I was able to simplify the issue without apertium
, so this happens with a simple cat
:
zcat sentences.gz | base64 -d | wc -l
25217
zcat sentences.gz | ~/go/bin/b64filter cat - | base64 -d | wc -l
2020/05/08 13:50:34 b64filter.go:198: writeDocs: written 100 docs, 4992 lines in 23.551143ms
2020/05/08 13:50:34 b64filter.go:198: writeDocs: written 200 docs, 18275 lines in 46.941439ms
2020/05/08 13:50:35 b64filter.go:274: processed 252 documents
25469
See that the wc -l
of all lines in the input base64 decoded documents is 25217 and in the output after using cat -
with b64filter
gets more lines (25469).
This generates wrong output files using translation engines, as the translations don't belong to the sentences they are attached to.
In case you can't reproduce the same output as I did in the example, here I attach it (differences start at line 137):
Hi,
I am running Bitextor for martacrawl es-eu, and it crashed after outputting this:
2020/08/03 10:52:15 b64filter.go:198: writeDocs: written 22200 docs, 2233737 lines in 2m46.222660934s
2020/08/03 10:52:16 b64filter.go:198: writeDocs: written 22300 docs, 2242692 lines in 2m46.230375568s
terminate called after throwing an instance of 'util::EndOfFileException'
what(): End of file
2020/08/03 10:52:16 b64filter.go:258: error waiting for command: signal: aborted
@lpla told me that this is a b64filter issue, so this is why I'm reporting here :)
Any clue on what's going on? I kinda know which shards are failing, so I can provide them if you need to reproduce the issue.
Thanks!
The change ported from /paracrawl/b64map/1 caused the readNLines
function to fail because the underlying buffer will return io.EOF
if it is full. readNLines
sensibly takes this to mean there is no more data to read but that is not, in fact, the right thing to do. Masked this for now by simply not treating io.EOF
as an error which will be a problem if the command being wrapped does not, in fact, produce enough lines.
I've been able to reproduce it with this command:
gzip -cd ~/hieu/nn/0/1/sentences.gz | b64filter parallel --halt 2 -j 16 --pipe -k -l 1024 false
parallel
will indicate that false
failed, but b64filter won't return and hang indefinitely.
It only happens for small sets of input sentences. If I try this with a sentences.gz from English, it will fail and stop on a pipe error.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.