Comments (26)
kpu
commented on August 20, 2024
2
See if I fixed it in 20a04ea.
from marian.
hieuhoang
commented on August 20, 2024
try
./amun -c config.ens.yml < source.txt > target.txt
until we fix it
from marian.
hieuhoang
commented on August 20, 2024
fixed
27d6fb9
from marian.
mehmedes
commented on August 20, 2024
Great! Works perfectly!
from marian.
mehmedes
commented on August 20, 2024
Is there anything I need to consider when translating a file with several hundred MB?
The translation crashes stating
Killed
from marian.
emjotde
commented on August 20, 2024
This probably reads your whole file into memory and then get killed due to filling it. @hieuhoang shouldn't maxi-batch take care of this?
from marian.
hieuhoang
commented on August 20, 2024
more of the job of the queuing mechanism, or it could be a leak. I've replicated the problem
this is gonna take a while to sort out
from marian.
hieuhoang
commented on August 20, 2024
@mehmedes - how many lines does it translate before it crashes and how approx much memory did it use before it crashed? How much mem does your machine have?
Can you make your input & models available for download?
from marian.
kpu
commented on August 20, 2024
https://github.com/emjotde/amunmt/blob/master/src/common/decoder_main.cpp#L59
You want a bounded producer-consumer queue on the ThreadPool.
from marian.
hieuhoang
commented on August 20, 2024
i can't see the problem with L59.
aye - bounded queue. Know of any we can steal?
from marian.
emjotde
commented on August 20, 2024
About L59, what happens with sentences when it is smaller than maxiBatch?
from marian.
mehmedes
commented on August 20, 2024
It translates 20 lines and uses 87 % MEM before it crashes. I run a virtual machine with 7.2 GB RAM
I use Rico's WMT16 de->en translation model
wget -r --cut-dirs=2 -e robots=off -nH -np -R index.html* http://data.statmt.org/rsennrich/wmt16_systems/de-en/
My configs are:
allow-unk: false
batch-size: 1
beam-size: 12
bpe:
- /home/sariyildiznureddin/de-en/deen.bpe
bunch-size: 1
cpu-threads: 8
devices: [0]
gpu-threads: 0
n-best: false
no-debpe: false
normalize: false
relative-paths: true
return-alignment: false
scorers:
F0:
path: /home/sariyildiznureddin/de-en/model-ens1.npz
type: Nematus
F1:
path: /home/sariyildiznureddin/de-en/model-ens2.npz
type: Nematus
F2:
path: /home/sariyildiznureddin/de-en/model-ens3.npz
type: Nematus
F3:
path: /home/sariyildiznureddin/de-en/model-ens4.npz
type: Nematus
show-weights: false
softmax-filter:
[]
source-vocab:
- /home/sariyildiznureddin/de-en/vocab.de.json
target-vocab: /home/sariyildiznureddin/de-en/vocab.en.json
weights:
F0: 1
F1: 1
F2: 1
F3: 1
wipo: false
from marian.
emjotde
commented on August 20, 2024
@mehmedes your commands are fine. This is our bug to fix, hold on :)
from marian.
kpu
commented on August 20, 2024
I have a bounded threadpool, but it's not C++11 style. Modifying the one you're using.
from marian.
hieuhoang
commented on August 20, 2024
@kpu cheers
@emjotde - it reads in another line. If EOF then it goes to L70. Seems to work ok
@mehmedes
- i've not tested amunmt with multiple models, there may be bugs. Can you try with just 1. It def shouldn't crash after 20 sentences.
- Can you replace
batch-size: 1
with the following:
mini-batch: 100
maxi-batch: 2000
from marian.
emjotde
commented on August 20, 2024
@hieuhoang He is using the CPU version, so mini-batches won't work.
from marian.
hieuhoang
commented on August 20, 2024
i know, just making sure
Hieu Hoang
http://moses-smt.org/
…On 24 January 2017 at 15:20, Marcin Junczys-Dowmunt < ***@***.***> wrote:
@hieuhoang <https://github.com/hieuhoang> He is using the CPU version, so
mini-batches won't work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#26 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAqOFLWFiVYA13BBS4P9ZuBQkFw0M5kvks5rVha0gaJpZM4LsKUD>
.
from marian.
mehmedes
commented on August 20, 2024
@hieuhoang
I get
terminate called after throwing an instance of 'YAML::ParserException'
what(): yaml-cpp: error at line 21, column 9: illegal map value
Aborted (core dumped)
@kpu I'll try.
from marian.
mehmedes
commented on August 20, 2024
Looks good. So far no crash!
from marian.
hieuhoang
commented on August 20, 2024
yep, me too. Mem doesn't grow. No slowdown
from marian.
mehmedes
commented on August 20, 2024
The translation crashed after 5300 lines. I had it running for 4 hours.
Would this now be due to my low memory?
from marian.
hieuhoang
commented on August 20, 2024
can you please make your files available for download so we can replicate it
from marian.
mehmedes
commented on August 20, 2024
I used the following model...
wget -r --cut-dirs=2 -e robots=off -nH -np -R index.html* http://data.statmt.org/rsennrich/wmt16_systems/de-en/
...to translate this file...
wget http://www.statmt.org/wmt14/training-monolingual-news-crawl/news.2007.de.shuffled.gz
with this shell command...
#!/bin/sh
# this sample script translates a test set, including
# preprocessing (tokenization, truecasing, and subword segmentation),
# and postprocessing (merging subword units, detruecasing, detokenization).
# instructions: set paths to mosesdecoder, subword_nmt, and nematus,
# then run "./translate.sh < input_file > output_file"
# suffix of source language
SRC=de
# suffix of target language
TRG=en
# path to moses decoder: https://github.com/moses-smt/mosesdecoder
mosesdecoder=/home/sariyildiznureddin/mosesdecoder
# preprocess
$mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l $SRC | \
$mosesdecoder/scripts/tokenizer/tokenizer.perl -l $SRC | \
$mosesdecoder/scripts/recaser/truecase.perl -model truecase-model.$SRC | \
# translate
/home/sariyildiznureddin/amunmt/build/bin/amun -c /home/sariyildiznureddin/amunmt/build/bin/config.ens.yml | \
sed 's/\@\@ //g' | \
$mosesdecoder/scripts/recaser/detruecase.perl | \
$mosesdecoder/scripts/tokenizer/detokenizer.perl -l $TRG -penn
... and these configs ...
allow-unk: false
batch-size: 1
beam-size: 12
bpe:
- /home/sariyildiznureddin/de-en/deen.bpe
bunch-size: 1
cpu-threads: 8
devices: [0]
gpu-threads: 0
n-best: false
no-debpe: false
normalize: false
relative-paths: true
return-alignment: false
scorers:
F0:
path: /home/sariyildiznureddin/de-en/model-ens1.npz
type: Nematus
F1:
path: /home/sariyildiznureddin/de-en/model-ens2.npz
type: Nematus
F2:
path: /home/sariyildiznureddin/de-en/model-ens3.npz
type: Nematus
F3:
path: /home/sariyildiznureddin/de-en/model-ens4.npz
type: Nematus
show-weights: false
softmax-filter:
[]
source-vocab:
- /home/sariyildiznureddin/de-en/vocab.de.json
target-vocab: /home/sariyildiznureddin/de-en/vocab.en.json
weights:
F0: 1
F1: 1
F2: 1
F3: 1
wipo: false
:D
from marian.
hieuhoang
commented on August 20, 2024
looks like it uses at least 6GB. If you machine has 8GB, I would not be surprised that it crash due to running out of memory on a particularly long sentence
from marian.
mehmedes
commented on August 20, 2024
Ok. Thanks! I'll increase the RAM and have a new run
from marian.
hieuhoang
commented on August 20, 2024
I've been running your setup for 9 hours on my server with 128GB ram, 12 1.7GHz cores. Still going, translated 25k sentences. Currently using 8.5GB ram
from marian.
Related Issues (20)
- Doc error in the explanation of --lr-decay-inv-sqrt?
- Linking CXX executable ../marian
- Not regular training : BLEU is brutally decreasing for several batches, before increasing again HOT 2
- running marian-server in secure fashion (wss) HOT 1
- Validation set is decoded multiple times for ngram level metrics
- undefined reference to `libiconv' HOT 1
- negative cost with negative weights
- Freezing layers/modules HOT 1
- when will update the doc HOT 1
- half float and other 3rd_party problems
- Error: Aborted from static To marian::fastopt_helpers::Convert<To, std::__cxx11::basic_string<char> HOT 1
- Unable to install Marian HOT 1
- What versions of CUDA & GCC are required to compile Marian-nmt? HOT 3
- Major / Minor changes
- Cost is nan When adding guided alignment HOT 1
- How to develop a C++ tokenizer for MarianMT in C++
- Get no output when using marian-decoder
- Marian's Sentencepiece Not Passing Case Encoding Command Through Marian HOT 1
- will marian continue to develop and update again？ HOT 1
- Compilation errors due to warnings treated as errors
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from marian.