exe1023 / dialevalmetrics Goto Github PK

when i run "conda env update --name eval_base --file conda_envs/eval_base/environment.yml" on mac,
error reports :
Collecting package metadata (repodata.json): done
Solving environment:
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
Solving environment:
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (libgcc-ng):

I noticed there is just dialog-level DSTC9 data used in your work. May I ask if turn-level DSTC9 annotation data is made public? How can I get this data? Thank you very much!

when i run "python usr_fed/usr/usr_server.py", error arrears:
Traceback (most recent call last):
File "usr_server.py", line 1, in
import usr
File "/home/zper/DialEvalMetrics/usr_fed/usr/usr.py", line 6, in
import regression
File "/home/zper/DialEvalMetrics/usr_fed/usr/regression.py", line 6, in
with open('regr.pkl','rb')as f:
FileNotFoundError: [Errno 2] No such file or directory: 'regr.pkl'

after i wget https://github.com/Shikib/usr/blob/master/examples/regr.pkl from original usr model github,
error also appears:
pickle invalid load key, '\x0a'.
i wonder if you can provide your pkl file, thanks.

I'm unable to get the desired results by following the steps as given in the README due to multiple issues such as:-

1. Environment creation failures for some of the them
2. gen_data.py is giving errors for DSTC6 and Engage
3. Transformers version issues for FBD, Maude (Eval_base environment) and TF version issues for run_bert_as_service.sh
   and some others as well.
4. Various files such as restore/ensemble.yml, ./data/DailyDialog/keyword.vocab etc. not being present
   So I request to update the repository with the latest codes and files if possible to help get the results as highlighted in your paper.

比如我想单独直接使用ADEM，运行ADEM里面的eval.py

Modify the path in run_eval.sh as specified in the script, since we need to activate Conda environment when running the script. Run eval_metrics.sh to evaluate all quality-anntoated data.

I can't find "run_eval.sh". Who knows where it is？

你好，请问bert-ruber和PONE有什么区别呀
我发现使用gen_data.py创建bert-ruber的数据时，产生的数据是放在了pone文件夹中，运行相应run之后，会产生三个文件：refer_score_dailydialog_sampled.json、ruber_score_dailydialog_sampled.json、unrefer_score_dailydialog_sampled.json，我想问一下这三个文件分别代表什么？
哪一个是bert-ruber的分数，哪个是pone的分数呀？

So thanks for your existing work !

I have get the result of grade_convai and grade_dailydialog, for both pearson and spearman scores are the same with yours. While for grade_empatheticdialogues, I get the different results under the same computation and evaluate methods.

1. First I use nlg_eval to compute baseline metrics for both three datasets
2. I modify the human_correlation.py script in data/grade_data, to compute for all the scores in both three datasets, and as described in above, two of them get the same but last one is different.

Waiting for your suggestions, thanks ！