Comments (26)
See if I fixed it in 20a04ea.
from marian.
try
./amun -c config.ens.yml < source.txt > target.txt
until we fix it
from marian.
fixed
27d6fb9
from marian.
Great! Works perfectly!
from marian.
Is there anything I need to consider when translating a file with several hundred MB?
The translation crashes stating
Killed
from marian.
This probably reads your whole file into memory and then get killed due to filling it. @hieuhoang shouldn't maxi-batch
take care of this?
from marian.
more of the job of the queuing mechanism, or it could be a leak. I've replicated the problem
this is gonna take a while to sort out
from marian.
@mehmedes - how many lines does it translate before it crashes and how approx much memory did it use before it crashed? How much mem does your machine have?
Can you make your input & models available for download?
from marian.
https://github.com/emjotde/amunmt/blob/master/src/common/decoder_main.cpp#L59
You want a bounded producer-consumer queue on the ThreadPool.
from marian.
i can't see the problem with L59.
aye - bounded queue. Know of any we can steal?
from marian.
About L59, what happens with sentences when it is smaller than maxiBatch?
from marian.
It translates 20 lines and uses 87 % MEM before it crashes. I run a virtual machine with 7.2 GB RAM
I use Rico's WMT16 de->en translation model
wget -r --cut-dirs=2 -e robots=off -nH -np -R index.html* http://data.statmt.org/rsennrich/wmt16_systems/de-en/
My configs are:
allow-unk: false
batch-size: 1
beam-size: 12
bpe:
- /home/sariyildiznureddin/de-en/deen.bpe
bunch-size: 1
cpu-threads: 8
devices: [0]
gpu-threads: 0
n-best: false
no-debpe: false
normalize: false
relative-paths: true
return-alignment: false
scorers:
F0:
path: /home/sariyildiznureddin/de-en/model-ens1.npz
type: Nematus
F1:
path: /home/sariyildiznureddin/de-en/model-ens2.npz
type: Nematus
F2:
path: /home/sariyildiznureddin/de-en/model-ens3.npz
type: Nematus
F3:
path: /home/sariyildiznureddin/de-en/model-ens4.npz
type: Nematus
show-weights: false
softmax-filter:
[]
source-vocab:
- /home/sariyildiznureddin/de-en/vocab.de.json
target-vocab: /home/sariyildiznureddin/de-en/vocab.en.json
weights:
F0: 1
F1: 1
F2: 1
F3: 1
wipo: false
from marian.
@mehmedes your commands are fine. This is our bug to fix, hold on :)
from marian.
I have a bounded threadpool, but it's not C++11 style. Modifying the one you're using.
from marian.
@kpu cheers
@emjotde - it reads in another line. If EOF then it goes to L70. Seems to work ok
@mehmedes
- i've not tested amunmt with multiple models, there may be bugs. Can you try with just 1. It def shouldn't crash after 20 sentences.
- Can you replace
batch-size: 1
with the following:
mini-batch: 100
maxi-batch: 2000
from marian.
@hieuhoang He is using the CPU version, so mini-batches won't work.
from marian.
from marian.
@hieuhoang
I get
terminate called after throwing an instance of 'YAML::ParserException'
what(): yaml-cpp: error at line 21, column 9: illegal map value
Aborted (core dumped)
@kpu I'll try.
from marian.
Looks good. So far no crash!
from marian.
yep, me too. Mem doesn't grow. No slowdown
from marian.
The translation crashed after 5300 lines. I had it running for 4 hours.
Would this now be due to my low memory?
from marian.
can you please make your files available for download so we can replicate it
from marian.
I used the following model...
wget -r --cut-dirs=2 -e robots=off -nH -np -R index.html* http://data.statmt.org/rsennrich/wmt16_systems/de-en/
...to translate this file...
wget http://www.statmt.org/wmt14/training-monolingual-news-crawl/news.2007.de.shuffled.gz
with this shell command...
#!/bin/sh
# this sample script translates a test set, including
# preprocessing (tokenization, truecasing, and subword segmentation),
# and postprocessing (merging subword units, detruecasing, detokenization).
# instructions: set paths to mosesdecoder, subword_nmt, and nematus,
# then run "./translate.sh < input_file > output_file"
# suffix of source language
SRC=de
# suffix of target language
TRG=en
# path to moses decoder: https://github.com/moses-smt/mosesdecoder
mosesdecoder=/home/sariyildiznureddin/mosesdecoder
# preprocess
$mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l $SRC | \
$mosesdecoder/scripts/tokenizer/tokenizer.perl -l $SRC | \
$mosesdecoder/scripts/recaser/truecase.perl -model truecase-model.$SRC | \
# translate
/home/sariyildiznureddin/amunmt/build/bin/amun -c /home/sariyildiznureddin/amunmt/build/bin/config.ens.yml | \
sed 's/\@\@ //g' | \
$mosesdecoder/scripts/recaser/detruecase.perl | \
$mosesdecoder/scripts/tokenizer/detokenizer.perl -l $TRG -penn
... and these configs ...
allow-unk: false
batch-size: 1
beam-size: 12
bpe:
- /home/sariyildiznureddin/de-en/deen.bpe
bunch-size: 1
cpu-threads: 8
devices: [0]
gpu-threads: 0
n-best: false
no-debpe: false
normalize: false
relative-paths: true
return-alignment: false
scorers:
F0:
path: /home/sariyildiznureddin/de-en/model-ens1.npz
type: Nematus
F1:
path: /home/sariyildiznureddin/de-en/model-ens2.npz
type: Nematus
F2:
path: /home/sariyildiznureddin/de-en/model-ens3.npz
type: Nematus
F3:
path: /home/sariyildiznureddin/de-en/model-ens4.npz
type: Nematus
show-weights: false
softmax-filter:
[]
source-vocab:
- /home/sariyildiznureddin/de-en/vocab.de.json
target-vocab: /home/sariyildiznureddin/de-en/vocab.en.json
weights:
F0: 1
F1: 1
F2: 1
F3: 1
wipo: false
:D
from marian.
looks like it uses at least 6GB. If you machine has 8GB, I would not be surprised that it crash due to running out of memory on a particularly long sentence
from marian.
Ok. Thanks! I'll increase the RAM and have a new run
from marian.
I've been running your setup for 9 hours on my server with 128GB ram, 12 1.7GHz cores. Still going, translated 25k sentences. Currently using 8.5GB ram
from marian.
Related Issues (20)
- Doc error in the explanation of --lr-decay-inv-sqrt?
- Linking CXX executable ../marian
- Not regular training : BLEU is brutally decreasing for several batches, before increasing again HOT 2
- running marian-server in secure fashion (wss) HOT 1
- Validation set is decoded multiple times for ngram level metrics
- undefined reference to `libiconv' HOT 1
- negative cost with negative weights
- Freezing layers/modules HOT 1
- when will update the doc HOT 1
- half float and other 3rd_party problems
- Error: Aborted from static To marian::fastopt_helpers::Convert<To, std::__cxx11::basic_string<char> HOT 1
- Unable to install Marian HOT 1
- What versions of CUDA & GCC are required to compile Marian-nmt? HOT 3
- Major / Minor changes
- Cost is nan When adding guided alignment HOT 1
- How to develop a C++ tokenizer for MarianMT in C++
- Get no output when using marian-decoder
- Marian's Sentencepiece Not Passing Case Encoding Command Through Marian HOT 1
- will marian continue to develop and update again? HOT 1
- Compilation errors due to warnings treated as errors
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from marian.