Comments (11)
miau1
commented on September 12, 2024
Could show the flags you used to initialize opus_reader?
from opustools.
gertva
commented on September 12, 2024
My your_script.py:
import opustools_pkg
opus_reader = opustools_pkg.OpusRead(["-d", "JW300", "-s", "en", "-t", "ta", "-wm" "tmx"])
opus_reader.printPairs()
from opustools.
miau1
commented on September 12, 2024
My your_script.py:
import opustools_pkg opus_reader = opustools_pkg.OpusRead(["-d", "JW300", "-s", "en", "-t", "ta", "-wm" "tmx"]) opus_reader.printPairs()
You are missing a comma between "-wm" and "tmx", but I don't think that's causing the error here. I suspect this is an issue with Windows encoding behavior. I'll try to setup a Windows environment and see if I can replicate the error. In the meanwhile, you could try to run your script in a unix-like environment, if you have access to one.
from opustools.
gertva
commented on September 12, 2024
it was the comma indeed! thank you so much for having spot this!
Op vr 30 aug. 2019 om 15:04 schreef miau1 <[email protected]>:
… My your_script.py:
import opustools_pkg
opus_reader = opustools_pkg.OpusRead(["-d", "JW300", "-s", "en", "-t", "ta", "-wm" "tmx"])
opus_reader.printPairs()
You are missing a comma between "-wm" and "tmx", but I don't think that's
causing the error here. I suspect this is an issue with Windows encoding
behavior. I'll try to setup a Windows environment and see if I can
replicate the error. In the meanwhile, you could try to run your script in
a unix-like environment, if you have access to one.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4?email_source=notifications&email_token=ACENS6WJCYFDED7BFAQ5HX3QHELGFA5CNFSM4ISLWNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5RS7XY#issuecomment-526594015>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACENS6WXLOTOF26IZ4LVTTTQHELGFANCNFSM4ISLWNYA>
.
from opustools.
gertva
commented on September 12, 2024
I think there is a problem anyhow, maybe only on WIN. I could not yet test
on Linux.
This is my script:
import opustools_pkg
opus_reader = opustools_pkg.OpusRead(["-d", "JW300", "-f", "-s", "en",
"-t", "ta", "-wm", "tmx", "-w", "enta.tmx"])
opus_reader.printPairs()
and the Error I get is:
Traceback (most recent call last):
File "your_script.py", line 3, in <module>
opus_reader.printPairs()
File
"C:\Users\gertv\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\opustools_pkg\opus_read.py",
line 350, in printPairs
lastline = self.readAlignment(gzipAlign)
File
"C:\Users\gertv\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\opustools_pkg\opus_read.py",
line 308, in readAlignment
lastline = self.outputPair(self.par, line)[1]
File
"C:\Users\gertv\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\opustools_pkg\opus_read.py",
line 251, in outputPair
self.sendPairOutput(wpair)
File
"C:\Users\gertv\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\opustools_pkg\opus_read.py",
line 210, in sendPairOutput
self.resultfile.write(wpair[0])
File "C:\Program
Files\WindowsApps\PythonSoftwareFoundation.Python.3.7_3.7.1264.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py",
line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position
126-132: character maps to <undefined>
and the TMX looks like this:
<?xml version="1.0" encoding="utf-8"?>
<tmx version="1.4.">
<header srclang="en"
adminlang="en"
segtype="sentence"
datatype="PlainText" />
<body>
<tu>
<tuv xml:lang="en"><seg>The Beauty of Bovine Design “ DAD , today our
schoolteacher said that a cow has four stomachs , which it has developed by
a process of evolution .</seg></tuv>
Op vr 30 aug. 2019 om 15:15 schreef Gert Van Assche <[email protected]>:
… it was the comma indeed! thank you so much for having spot this!
Op vr 30 aug. 2019 om 15:04 schreef miau1 ***@***.***>:
> My your_script.py:
>
> import opustools_pkg
> opus_reader = opustools_pkg.OpusRead(["-d", "JW300", "-s", "en", "-t", "ta", "-wm" "tmx"])
> opus_reader.printPairs()
>
> You are missing a comma between "-wm" and "tmx", but I don't think
> that's causing the error here. I suspect this is an issue with Windows
> encoding behavior. I'll try to setup a Windows environment and see if I can
> replicate the error. In the meanwhile, you could try to run your script in
> a unix-like environment, if you have access to one.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#4?email_source=notifications&email_token=ACENS6WJCYFDED7BFAQ5HX3QHELGFA5CNFSM4ISLWNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5RS7XY#issuecomment-526594015>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACENS6WXLOTOF26IZ4LVTTTQHELGFANCNFSM4ISLWNYA>
> .
>
from opustools.
miau1
commented on September 12, 2024
@gertva I haven't been able to set up a Windows environment, but I changed something in the program: I now specify the encoding of the result files, which might help with your issue. Upgrade opustools_pkg to version 0.0.43, run your script again and let me know if it works.
I'll probably get my hands on a Windows machine by tomorrow, so I'll be able to do proper debugging.
from opustools.
gertva
commented on September 12, 2024
works like magic now!
thank you so much.
gert
Op ma 2 sep. 2019 om 11:06 schreef miau1 <[email protected]>:
… @gertva <https://github.com/gertva> I haven't been able to set up a
Windows environment, but I changed something in the program: I now specify
the encoding of the result files, which might help with your issue. Upgrade
opustools_pkg to version 0.0.43, run your script again and let me know if
it works.
I'll probably get my hands on a Windows machine by tomorrow, so I'll be
able to do proper debugging.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4?email_source=notifications&email_token=ACENS6XVZIN73IFQJMWWTA3QHTJQZA5CNFSM4ISLWNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5VGVUA#issuecomment-527067856>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACENS6UUPD37GBKST473AF3QHTJQZANCNFSM4ISLWNYA>
.
from opustools.
miau1
commented on September 12, 2024
Great, good to hear!
from opustools.
gertva
commented on September 12, 2024
A question, if I want to filter on language, is this to correct construct?
opus_reader = opustools_pkg.OpusRead(["-d", "JW300", "-f", "-ln", "-s",
"en", "-t", "ta", "-wm", "tmx", "-w", "enta.tmx", "src_cld2", "en", "0.95",
"trg_cld2", "ta", "0.95", "src_langid", "en", "0.95", "trg_langid", "ta",
"0.95"])
My output file is always the same size, so I wonder if it is correct.
Thanks for your help.
Op ma 2 sep. 2019 om 12:53 schreef miau1 <[email protected]>:
… Closed #4 <#4>.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4?email_source=notifications&email_token=ACENS6V5ZGYQHKD2SKGWYVTQHTWC3A5CNFSM4ISLWNYKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTMCB5NY#event-2600738487>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACENS6TMFWOYMH5UDPDMC7DQHTWC3ANCNFSM4ISLWNYA>
.
from opustools.
miau1
commented on September 12, 2024
Almost, you have to include -- before the language id flags. You can also use split() to make the argument list formation a little easier like this:
opus_reader = opustools_pkg.OpusRead("-d JW300 -f -ln -s en -t ta -wm tmx -w enta.tmx --src_cld2 en 0.95 --trg_cld2 ta 0.95 --src_langid en 0.95 --trg_langid ta 0.95".split())
But for this to work, you have to create the language ids to the zip files, as they don't yet include them by default. First you need to install pycld2 and langid:
pip install pycld2
pip install langid
Then you can run a script like this to create the language ids:
from opustools_pkg.opus_langid import OpusLangid
OpusLangid("-f JW300_latest_xml_en.zip -v".split()).processFiles()
OpusLangid("-f JW300_latest_xml_ta.zip -v".split()).processFiles()
And then you can filter by language ids.
from opustools.
gertva
commented on September 12, 2024
thanks for explaining so clearly.
Op ma 2 sep. 2019 om 14:53 schreef miau1 <[email protected]>:
… Almost, you have to include -- before the language id flags. You can also
use split() to make the argument list formation a little easier like this:
opus_reader = opustools_pkg.OpusRead("-d JW300 -f -ln -s en -t ta -wm tmx -w enta.tmx --src_cld2 en 0.95 --trg_cld2 ta 0.95 --src_langid en 0.95 --trg_langid ta 0.95".split())
But for this to work, you have to create the language ids to the zip
files, as they don't yet include them by default. First you need to install
pycld2 and langid:
pip install pycld2
pip install langid
Then you can run a script like this to create the language ids:
from opustools_pkg.opus_langid import OpusLangid
OpusLangid("-f JW300_latest_xml_en.zip -v".split()).processFiles()
OpusLangid("-f JW300_latest_xml_en.zip -v".split()).processFiles()
And then you can filter by language ids.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4?email_source=notifications&email_token=ACENS6QQYVSVDS6BOS7IECDQHUEF3A5CNFSM4ISLWNYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5VXYBA#issuecomment-527137796>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACENS6R7QDOGTWT6LDOH7BTQHUEF3ANCNFSM4ISLWNYA>
.
from opustools.
Related Issues (20)
- Alignment problem with JW300 corpora? HOT 1
- PyPI wheel includes old files HOT 2
- Where are the missing language pairs? HOT 1
- Memory Issue: opus_read fails to extract MultiCCAligned HOT 1
- Missing alignment data for English(en) - Oromo(om)? HOT 1
- What is the tokenizer for all languages?
- Format of downloaded files does not match the format expected by opus_read HOT 1
- malformed tmx from opus_read HOT 1
- Using opus_read with -az, -sz, -tz options HOT 1
- Add progress indicator to opus_express HOT 2
- Misleading logging information in opus_express HOT 1
- Is it possible to download all corpus associate with the given language pair? HOT 1
- opus_get downloads *all* corpora with just the -s switch HOT 1
- opus_read fails to extract CCMatrix HOT 3
- DB for off-line search HOT 1
- support search with 3-letter language codes or BCP-47
- Recreate sample files shown in OpenSubtitles corpus HOT 1
- Spaces before punctation marks on opus_read output HOT 1
- change in OPUS yaml files HOT 1
- Cannot download resource due to `DH_KEY_TOO_SMALL`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opustools.