Comments (3)
add_lang_ids
script is now ready. Still need to update opus_read
to be able to filter by language language labels.
from opustools.
could we rename the frontend script to opus_langid
instead? I'd like to keep the opus prefix for the tools in this package and also like to keep names short. What kind of formats can this script now work with? Does it work with tokenized (the files in xml) and untokenized texts (the ones in raw)? Does it work with both, zipped archives of files and single xml files?
from opustools.
opus_langid
works with tokenized and raw xml files in both, single xml files and xml files in zip archives. The script is also now more memory efficient than the first version, as it does not load the whole xml tree into memory at once. opus_read
is able to filter by language labels.
from opustools.
Related Issues (20)
- Opus_read: SentenceParserError HOT 5
- List of datasets | Monolingual raw files HOT 5
- opus_express not checking correctly root directory HOT 1
- opus_express without confirmation? HOT 1
- Alignment problem with JW300 corpora? HOT 1
- PyPI wheel includes old files HOT 2
- Where are the missing language pairs? HOT 1
- Memory Issue: opus_read fails to extract MultiCCAligned HOT 1
- Missing alignment data for English(en) - Oromo(om)? HOT 1
- What is the tokenizer for all languages?
- Format of downloaded files does not match the format expected by opus_read HOT 1
- malformed tmx from opus_read HOT 1
- Using opus_read with -az, -sz, -tz options HOT 1
- Add progress indicator to opus_express HOT 2
- Misleading logging information in opus_express HOT 1
- Is it possible to download all corpus associate with the given language pair? HOT 1
- opus_get downloads *all* corpora with just the -s switch HOT 1
- opus_read fails to extract CCMatrix HOT 3
- DB for off-line search HOT 1
- support search with 3-letter language codes or BCP-47
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opustools.