Setting up config file | Test Rico's WMT16 EN-DE about marian HOT 19 CLOSED

This works for me (BTW, does this mean you were able to compile?):

allow-unk: false
batch-size: 1
beam-size: 12
bpe:
  - deen.bpe
bunch-size: 1
cpu-threads: 8
devices: [0]
gpu-threads: 0
n-best: false
no-debpe: false
normalize: false
relative-paths: true
return-alignment: false
scorers:
  F0:
    path: model.npz
    type: Nematus
show-weights: false
softmax-filter:
  []
source-vocab:
  - vocab.de.json
target-vocab: vocab.en.json
weights:
  F0: 1
wipo: false

BTW: You can let amun create this config file by doing:

path/to/amunmt/bin/amun -m model.npz -s vocab.de.json -t vocab.en.json \
 --bpe deen.bpe --cpu-threads 8 --gpu-threads 0 --dump-config \
 --relative-paths > config.yml

from marian.

And for four models created with:

path/to/amunmt/bin/amun -m model-ens*.npz -s vocab.de.json -t vocab.en.json \
 --bpe deen.bpe --cpu-threads 8 --gpu-threads 0 --dump-config \
 --relative-paths > config.ens.yml

you get the following config:

allow-unk: false
batch-size: 1
beam-size: 12
bpe:
  - deen.bpe
bunch-size: 1
cpu-threads: 8
devices: [0]
gpu-threads: 0
n-best: false
no-debpe: false
normalize: false
relative-paths: true
return-alignment: false
scorers:
  F0:
    path: model-ens1.npz
    type: Nematus
  F1:
    path: model-ens2.npz
    type: Nematus
  F2:
    path: model-ens3.npz
    type: Nematus
  F3:
    path: model-ens4.npz
    type: Nematus
show-weights: false
softmax-filter:
  []
source-vocab:
  - vocab.de.json
target-vocab: vocab.en.json
weights:
  F0: 1
  F1: 1
  F2: 1
  F3: 1
wipo: false

from marian.

Hi @mehmedes
Can you pull the lastest version? What gcc version do you have? I tried to reproduce but it works on my machine. I downloaded en-de model (using scripts/download_models.py) and replaced with your config.

from marian.

I went back to 3a548b3 for testing.

I'll try your latest commit asap and let you know if it works. I'm using gcc 5.4.

Thank you so much for your invaluable input! Have a nice weekend!

from marian.

I've now been able to build the latest commit of amunmt.

When trying to create the config file, however, I still get the error:

Segmentation fault (core dumped)

Do you have any clues?

Thanks!

from marian.

Can you please post your entire config file using the markup described here https://help.github.com/articles/creating-and-highlighting-code-blocks/

from marian.

Sure. Thanks for the link. I was always wondering how to get the proper formatting in Github.

Amunmt creates an empty config file, stating Segmentation fault (core dumped). I've also tried to use the both configs above but still Segmentation fault (core dumped). Again, at the moment I'm not able to create a "real" config file. All I get is the error message with an empty file.

from marian.

So, just to make sure I understand. You tried one of the above commands I posted, exactly the way I wrote them, and you get a segfault when you do this?

Unfortunately I cannot reproduce this at the moment. One straw of hope: Can you remove your build directory and try the entire compilation once more from scratch?

Also, earlier you wrote that you were using a previous revision. Did that one work for you?

And a last thing to check. What happens if you call amun without any options, just ./bin/amun? You should get an exception like this:

terminate called after throwing an instance of 'util::Exception'
  what():  /home/marcin/amunmt/src/common/config.cpp:85 in void Validate(const YAML::Node&) threw util::Exception because `!config["scorers"] || config["scorers"].size() == 0'.
No scorers given in config file
Aborted (core dumped)

from marian.

I first tried the exact commands above on 3a548b3.
I got Segmentation fault (core dumped).

Then I tried to create a config file according to the instructions:

path/to/amunmt/bin/amun -m model.npz -s vocab.de.json -t vocab.en.json
--bpe deen.bpe --cpu-threads 8 --gpu-threads 0 --dump-config
--relative-paths > config.yml

path/to/amunmt/bin/amun -m model-ens*.npz -s vocab.de.json -t vocab.en.json
--bpe deen.bpe --cpu-threads 8 --gpu-threads 0 --dump-config
--relative-paths > config.ens.yml

Again, Segmentation fault (core dumped).

Then, I deleted the entire installation and set up the latest commit after your fix.

I first tried one of the above commands. Didn't work.

Lastly I tried creating a config file with the above instructions.

Again Segmentation fault (core dumped).

The only command that works is ./amun -help

from marian.

Interestingly, if I type ./bin/amun -help I get the help message.
If I type --help or -h, I get Segmentation fault (core dumped).

from marian.

What's your boost version? We use boost::options in amuNMT for handling this.

from marian.

Boost 1.54

from marian.

Would this core file be of any use?
core.zip

from marian.

I personally don't use core files. Being able to replicate your segfault is better so your model files, config file and the exact command you executed is better

from marian.

This is one of the commands I tried for creating a config file for the English-German WMT model. The model files are in the same folder I ran the command in.

./bin/amun -m model.npz -s vocab.en-json -t vocab.de.json --bpe ende.bpe --cpu-threads 8 --dump-config --relative-paths > config.yml

The output:

Segmentation fault (core dumped)

from marian.

Running that command is ok with me. However, I'm running on Ubuntu 16, boost 1.62, gcc 5.4

I'm trying to install a VM with your specs but it's quite difficult.

It may be better for you to update your OS

from marian.

I've just test on OSX Sierra + boost 1.54 (1.54 doesn't build on Ubuntu 16). It works fine

So the problem is likely to be in Ubuntu 14 with the custom gcc & cmake install.

from marian.

Alright. I'll update the OS and keep you updated!

from marian.

Everything runs on Ubuntu 16 smoothly. Thanks!

from marian.