Comments (5)
Unfortunately, those files are not proper xml and currently opus_read crashes if the parsing fails. In the future, we might have an update where opus_read continues to run even if some of the files are not proper xml.
from opustools.
Furthermore, I am getting an error with EuBookshop dataset:
opustools.opus_read.AlignmentParserError: Alignment file "./EUbookshop_v2_xml_bg-en.xml.gz" could not be parsed: mismatched tag: line 225123, column 2
from opustools.
The latest version of OpusTools, 1.0.0, opus_read continues parsing from the next sentence file if a sentence file with invalid xml is encountered. If there is an error in an alignment file, the file is parsed up to the error, but cannot be parsed any further. There are plans to fix broken xml files in Opus.
from opustools.
@miau1 Any progress on fixing the xml files?
An error occured during the creation of parallel-sentences2/EUbookshop-en-sv.tsv.gz
type error: Error while parsing alignment file: Document './opus/EUbookshop_latest_xml_en-sv.xml.gz' could not be parsed: mismatched tag: line 1964268, column 2
The file EUbookshop_latest_xml_en-sv.xml.gz
seems to have many missing </linkGrp>
closing tags. The first <linkGrp>
has a closing tag, then none of them have one until the very end, where about 50 of them have a closing tag.
I somehow managed to sentence align this dataset a couple of months ago by downloading through here instead: https://opus.nlpl.eu/download.php?f=EUbookshop/v2/moses/en-sv.txt.zip
and using the non-corrupt alignment file EUbookshop.en-sv.ids
to sentence align the data. But I can't for the life of me remember what terminal command args I used to successfully do this. Everything I try now fails. Yet I have a successfully aligned file from a couple of months ago that is sitting there (just not able to recreate it...).
from opustools.
Keeping this thread alive by reporting the same issue with opustools 1.3.1
and the following command:
opus_read --directory EUbookshop \
--suppress_prompts \
--source en \
--target ar \
--preprocess raw \
--leave_non_alignments_out \
--write_mode moses \
--write EUbookshop.en.ar.txt
The error reads:
opustools.parse.alignment_parser.AlignmentParserError: Error while parsing alignment file: Document './EUbookshop_latest_xml_ar-en.xml.gz' could not be parsed: mismatched tag: line 1767, column 2
from opustools.
Related Issues (20)
- opus_express not checking correctly root directory HOT 1
- opus_express without confirmation? HOT 1
- Alignment problem with JW300 corpora? HOT 1
- PyPI wheel includes old files HOT 2
- Where are the missing language pairs? HOT 1
- Memory Issue: opus_read fails to extract MultiCCAligned HOT 1
- Missing alignment data for English(en) - Oromo(om)? HOT 1
- What is the tokenizer for all languages?
- Format of downloaded files does not match the format expected by opus_read HOT 1
- malformed tmx from opus_read HOT 1
- Using opus_read with -az, -sz, -tz options HOT 1
- Add progress indicator to opus_express HOT 2
- Misleading logging information in opus_express HOT 1
- Is it possible to download all corpus associate with the given language pair? HOT 1
- opus_get downloads *all* corpora with just the -s switch HOT 1
- opus_read fails to extract CCMatrix HOT 3
- DB for off-line search HOT 1
- support search with 3-letter language codes or BCP-47
- Recreate sample files shown in OpenSubtitles corpus HOT 1
- Spaces before punctation marks on opus_read output HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opustools.