convlab / convlab-3 Goto Github PK

License: Apache License 2.0

Dockerfile 0.01% Python 97.32% Shell 1.05% Perl 0.19% HTML 0.64% Jsonnet 0.61% CSS 0.05% JavaScript 0.01% Jupyter Notebook 0.13%

conversational-ai

convlab-3's Introduction

ConvLab

ConvLab is an open-source multi-domain end-to-end dialog system platform, aiming to enable researchers to quickly set up experiments with reusable components and compare a large set of different approaches, ranging from conventional pipeline systems to end-to-end neural models, in common environments.

Package Overview

convlab	an open-source multi-domain end-to-end dialog research library
convlab.agent	a module for constructing dialog agents including RL algorithms
convlab.env	a collection of environments
convlab.experiment	a module for running experiments at various levels
convlab.evaluator	a module for evaluating a dialog session with various metrics
convlab.modules	a collection of state-of-the-art dialog system component models including NLU, DST, Policy, NLG
convlab.human_eval	a server for conducting human evaluation using Amazon Mechanical Turk
convlab.lib	a library of common utilities
convlab.spec	a collection of experiment spec files

Installation

ConvLab requires Python 3.6.5 or later. Windows is currently not offically supported.

Installing via pip

Setting up a virtual environment

Conda can be used to set up a virtual environment with the version of Python required for ConvLab. If you already have a Python 3.6 or 3.7 environment you want to use, you can skip to the 'installing via pip' section.

Download and install Conda.
Create a Conda environment with Python 3.6.5
```
conda create -n convlab python=3.6.5
```
Activate the Conda environment. You will need to activate the Conda environment in each terminal in which you want to use ConvLab.
```
source activate convlab
```

Installing the library and dependencies

Installing the library and dependencies is simple using pip.

pip install -r requirements.txt

If your Linux system does not have essential building tools installed, you might need to install it by running

sudo apt-get install build-essential

ConvLab uses 'stopwords' in nltk, and you need to download it by running

python -m nltk.downloader stopwords

Installation tips on CentOS

Please refer to the instructions here: https://github.com/daveta/convlab-notes/wiki/Installing-Convlab-on-Centos7

Installing using Docker

Docker provides more isolation and consistency, and also makes it easy to distribute your environment to a compute cluster.

Once you have installed Docker just run the following commands to get an environment that will run on either the CPU or GPU.

Pull docker
docker pull convlab/convlab:0.2.2
Run docker
docker run -it --rm convlab/convlab:0.2.2

Running ConvLab

Once you've downloaded ConvLab and installed required packages, you can run the command-line interface with the python run.py command.

$ python run.py {spec file} {spec name} {mode}

For non-RL policies:

# to evaluate a dialog system consisting of NLU(OneNet), DST(Rule), Policy(Rule), NLG(Template) on the MultiWOZ environment
$ python run.py demo.json onenet_rule_rule_template eval

# to see natural language utterances 
$ LOG_LEVEL=NL python run.py demo.json onenet_rule_rule_template eval

# to see natural language utterances and dialog acts 
$ LOG_LEVEL=ACT python run.py demo.json onenet_rule_rule_template eval

# to see natural language utterances, dialog acts and state representation
$ LOG_LEVEL=STATE python run.py demo.json onenet_rule_rule_template eval

For RL policies:

# to train a DQN policy with NLU(OneNet), DST(Rule), NLG(Template) on the MultiWOZ environment
$ python run.py demo.json onenet_rule_dqn_template train

# to use the policy trained above (this will load up the onenet_rule_dqn_template_t0_s0_*.pt files under the output/onenet_rule_dqn_template_{timestamp}/model directory)
$ python run.py demo.json onenet_rule_dqn_template eval@output/onenet_rule_dqn_template_{timestamp}/model/onenet_rule_dqn_template_t0_s0

Note that currently ConvLab can only train the policy component by interacting with a user simulator. For other components, ConvLab supports offline supervise learning. For example, you can train an NLU model using the local training script as in OneNet.

Creating a new spec file

A spec file is used to fully specify experiments including a dialog agent and a user simulator. It is a JSON of multiple experiment specs, each containing the keys agent, env, body, meta, search.

We based our implementation on SLM-Lab.

Instead of writing one from scratch, you are welcome to modify the convlab/spec/demo.json file. Once you have created a new spec file, place it under convlab/spec directory and run your experiments. Note that you don't have to prepend convlab/spec/ before your spec file name.

Participation in DSTC-8

Extend ConvLab with your code, and include submission.json under the convlab/spec directory.
In submission.json, specify up to 5 specs with the name submission[1-5].
Make sure the code with the config is runnable in the docker environment.
If your code uses external packages beyond the existing docker environment, please choose one of the following two approaches to specify your environment requirements:
- Add install.sh under the convlab directory. install.sh should include all required extra packages.
- Create your own Dockerfile with the name dev.dockerfile
Zip the system and submit.

Evaluation

Automatic end2end Evaluation: The submitted system will be evaluated using the user-simulator setting in spec milu_rule_rule_template in convlab/spec/baseline.json. We will use the evaluator MultiWozEvaluator in convlab/evaluator/multiwoz to report metrics including success rate, average reward, number of turms, precision, recall, and F1 score.
Human Evaluation: The submitted system will be evaluated in Amazon Mechanic Turk. Crowd-workers will communicate with your summited system, and provide a rating based on the whole experience (language understanding, appropriateness, etc.)

Contributions

The ConvLab team welcomes contributions from the community. Pull requests must have one approving review and no requested changes before they are merged. The ConvLab team reserves the right to reject or revert contributions that we don't think are good additions.

Citing

If you use ConvLab in your research, please cite ConvLab: Multi-Domain End-to-End Dialog System Platform.

@inproceedings{lee2019convlab,
  title={ConvLab: Multi-Domain End-to-End Dialog System Platform},
  author={Lee, Sungjin and Zhu, Qi and Takanobu, Ryuichi and Li, Xiang and Zhang, Yaoqin and Zhang, Zheng and Li, Jinchao and Peng, Baolin and Li, Xiujun and Huang, Minlie and Gao, Jianfeng},
  booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year={2019}
}

convlab-3's People

Contributors

Stargazers

Watchers

convlab-3's Issues

[Maintenance] check models are runnable in new package requirement in setup.py

Describe the bug
PR #2 update package version, need to test relevant models especially transformers based.

python >= 3.6 (~~>=3.5~~)
transformers >= 4.0 (~~>=2.3.0,<3.0.0~~)
torch >= 1.6 (~~>=1.2.0,<1.6.0~~)
spacy >= 3.1.4 (~~==2.1.9~~)
allennlp >= 2.8 (~~==0.9.0~~)

Part of relevant models:

MILU (allennlp)
BERTNLU (transformers)

Expected behavior
Code should work in the new environment. If necessary, re-train the model.

Additional context

T5 for DST (evaluation results)

I fine-tuned T5 for DST. I got the (predict metrics) after evaluation. Then, I used "merge_predict_res" and "evaluate_unified_datasets", I got different results.. why?

evaluate_unified_datasets results:

do you know how to train crosswoz dataset in tutorial

I follow the instructions in the tutorial, and change the dataset to crosswoz.
but it always return the following result. 抱歉,没有查到符合条件的餐馆信息,你看可以吗?

Load actions from file..
Dimension of system actions: 89
Dimension of user actions: 81
State dimension: 232
Load actions from file..
Dimension of system actions: 89
Dimension of user actions: 81
State dimension: 232
Load actions from file..
Dimension of system actions: 89
Dimension of user actions: 81
State dimension: 232
WARNING:root:nlu info_dict is not initialized
WARNING:root:dst info_dict is not initialized
WARNING:root:nlg info_dict is not initialized
[['Inform', '餐馆', '人均消费', '50-100元'], ['Request', '餐馆', '名称', '']]
{'belief_state': {'餐馆': {'人均消费': '50-100元'}}, 'booked': {}, 'history': [['sys', 'null'], ['user', '帮我推荐一个人均消费在50-100元的餐馆']], 'system_action': [], 'user_action': [['Inform', '餐馆', '人均消费', '50-100元'], ['Request', '餐馆', '名称', '']], 'terminated': False}
[['NoOffer', '餐馆', 'none', 'none']]
抱歉,没有查到符合条件的餐馆信息,你看可以吗?
抱歉,没有查到符合条件的餐馆信息,你看可以吗?

Process finished with exit code 0

I think it's fail to run
If anyone can help .
thanks very much

[BUG] Dependency conflicts when running in Colab

When I run the colab demo, I get this error in the first cell:

Cloning into 'ConvLab-3'...
remote: Enumerating objects: 1275, done.
remote: Counting objects: 100% (1275/1275), done.
remote: Compressing objects: 100% (998/998), done.
remote: Total 1275 (delta 321), reused 837 (delta 230), pack-reused 0
Receiving objects: 100% (1275/1275), 438.42 MiB | 19.93 MiB/s, done.
Resolving deltas: 100% (321/321), done.
Updating files: 100% (1153/1153), done.
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Obtaining file:///content/ConvLab-3
  Preparing metadata (setup.py) ... done
Collecting joblib>=1.2.0
  Downloading joblib-1.2.0-py3-none-any.whl (297 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 298.0/298.0 kB 6.2 MB/s eta 0:00:00
Collecting pillow>=9.3.0
  Downloading Pillow-9.5.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 52.0 MB/s eta 0:00:00
Requirement already satisfied: protobuf>=3.20.2 in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (3.20.3)
Requirement already satisfied: oauthlib>=3.2.1 in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (3.2.2)
Collecting accelerate
  Downloading accelerate-0.18.0-py3-none-any.whl (215 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 215.3/215.3 kB 19.8 MB/s eta 0:00:00
Collecting rouge-score
  Downloading rouge_score-0.1.2.tar.gz (17 kB)
  Preparing metadata (setup.py) ... done
Collecting sacrebleu
  Downloading sacrebleu-2.3.1-py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.9/118.9 kB 6.3 MB/s eta 0:00:00
Collecting tensorboardX
  Downloading tensorboardX-2.6-py2.py3-none-any.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.5/114.5 kB 8.5 MB/s eta 0:00:00
Collecting boto3
  Downloading boto3-1.26.110-py3-none-any.whl (135 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 kB 4.5 MB/s eta 0:00:00
Requirement already satisfied: matplotlib in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (3.7.1)
Requirement already satisfied: seaborn in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (0.12.2)
Requirement already satisfied: tabulate in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (0.8.10)
Collecting python-Levenshtein
  Downloading python_Levenshtein-0.20.9-py3-none-any.whl (9.4 kB)
Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (2.27.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (1.22.4)
Requirement already satisfied: nltk in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (3.8.1)
Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (1.10.1)
Requirement already satisfied: tensorboard in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (2.12.1)
Collecting torch<=1.13,>=1.10.1
  Downloading torch-1.13.0-cp39-cp39-manylinux1_x86_64.whl (890.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 890.2/890.2 MB 1.5 MB/s eta 0:00:00
Collecting transformers<=4.24.0,>=4.17.0
  Downloading transformers-4.24.0-py3-none-any.whl (5.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 82.6 MB/s eta 0:00:00
Collecting sentence-transformers>=2.2.2
  Downloading sentence-transformers-2.2.2.tar.gz (85 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.0/86.0 kB 7.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting datasets>=2.0
  Downloading datasets-2.11.0-py3-none-any.whl (468 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.7/468.7 kB 35.7 MB/s eta 0:00:00
Collecting seqeval
  Downloading seqeval-1.2.2.tar.gz (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.6/43.6 kB 4.4 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: spacy in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (3.5.1)
Collecting simplejson
  Downloading simplejson-3.19.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (137 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.4/137.4 kB 13.5 MB/s eta 0:00:00
Collecting unidecode
  Downloading Unidecode-1.3.6-py3-none-any.whl (235 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 235.9/235.9 kB 19.8 MB/s eta 0:00:00
Requirement already satisfied: jieba in /usr/local/lib/python3.9/dist-packages (from convlab==3.0.1.dev20230411) (0.42.1)
Collecting embeddings
  Downloading embeddings-0.0.8-py3-none-any.whl (12 kB)
Collecting visdom
  Downloading visdom-0.2.4.tar.gz (1.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 36.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting quadprog
  Downloading quadprog-0.1.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (453 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 453.4/453.4 kB 32.9 MB/s eta 0:00:00
Collecting fuzzywuzzy
  Downloading fuzzywuzzy-0.18.0-py2.py3-none-any.whl (18 kB)
Collecting json_lines
  Downloading json_lines-0.5.0-py2.py3-none-any.whl (6.8 kB)
Collecting gtts
  Downloading gTTS-2.3.1-py3-none-any.whl (28 kB)
Collecting deepspeech
  Downloading deepspeech-0.9.3-cp39-cp39-manylinux1_x86_64.whl (9.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.2/9.2 MB 82.4 MB/s eta 0:00:00
Collecting pydub
  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Collecting responses<0.19
  Downloading responses-0.18.0-py3-none-any.whl (38 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from datasets>=2.0->convlab==3.0.1.dev20230411) (1.4.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from datasets>=2.0->convlab==3.0.1.dev20230411) (23.0)
Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.9/dist-packages (from datasets>=2.0->convlab==3.0.1.dev20230411) (9.0.0)
Collecting huggingface-hub<1.0.0,>=0.11.0
  Downloading huggingface_hub-0.13.4-py3-none-any.whl (200 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.1/200.1 kB 18.7 MB/s eta 0:00:00
Collecting aiohttp
  Downloading aiohttp-3.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 54.3 MB/s eta 0:00:00
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from datasets>=2.0->convlab==3.0.1.dev20230411) (6.0)
Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.9/dist-packages (from datasets>=2.0->convlab==3.0.1.dev20230411) (4.65.0)
Collecting xxhash
  Downloading xxhash-3.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.2/212.2 kB 18.6 MB/s eta 0:00:00
Collecting multiprocess
  Downloading multiprocess-0.70.14-py39-none-any.whl (132 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.9/132.9 kB 11.5 MB/s eta 0:00:00
Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.9/dist-packages (from datasets>=2.0->convlab==3.0.1.dev20230411) (2023.3.0)
Collecting dill<0.3.7,>=0.3.0
  Downloading dill-0.3.6-py3-none-any.whl (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.5/110.5 kB 12.4 MB/s eta 0:00:00
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->convlab==3.0.1.dev20230411) (2.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->convlab==3.0.1.dev20230411) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->convlab==3.0.1.dev20230411) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->convlab==3.0.1.dev20230411) (1.26.15)
Requirement already satisfied: torchvision in /usr/local/lib/python3.9/dist-packages (from sentence-transformers>=2.2.2->convlab==3.0.1.dev20230411) (0.15.1+cu118)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.9/dist-packages (from sentence-transformers>=2.2.2->convlab==3.0.1.dev20230411) (1.2.2)
Collecting sentencepiece
  Downloading sentencepiece-0.1.97-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 63.0 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch<=1.13,>=1.10.1->convlab==3.0.1.dev20230411) (4.5.0)
Collecting nvidia-cudnn-cu11==8.5.0.96
  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 2.2 MB/s eta 0:00:00
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 59.1 MB/s eta 0:00:00
Collecting nvidia-cuda-runtime-cu11==11.7.99
  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 52.2 MB/s eta 0:00:00
Collecting nvidia-cublas-cu11==11.10.3.66
  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 2.5 MB/s eta 0:00:00
Requirement already satisfied: wheel in /usr/local/lib/python3.9/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<=1.13,>=1.10.1->convlab==3.0.1.dev20230411) (0.40.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch<=1.13,>=1.10.1->convlab==3.0.1.dev20230411) (67.6.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers<=4.24.0,>=4.17.0->convlab==3.0.1.dev20230411) (2022.10.31)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
  Downloading tokenizers-0.13.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 82.9 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from transformers<=4.24.0,>=4.17.0->convlab==3.0.1.dev20230411) (3.10.7)
Requirement already satisfied: psutil in /usr/local/lib/python3.9/dist-packages (from accelerate->convlab==3.0.1.dev20230411) (5.9.4)
Collecting botocore<1.30.0,>=1.29.110
  Downloading botocore-1.29.110-py3-none-any.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 84.3 MB/s eta 0:00:00
Collecting s3transfer<0.7.0,>=0.6.0
  Downloading s3transfer-0.6.0-py3-none-any.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 kB 6.5 MB/s eta 0:00:00
Collecting jmespath<2.0.0,>=0.7.1
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Requirement already satisfied: click<8.2,>=7.1 in /usr/local/lib/python3.9/dist-packages (from gtts->convlab==3.0.1.dev20230411) (8.1.3)
Requirement already satisfied: six in /usr/local/lib/python3.9/dist-packages (from json_lines->convlab==3.0.1.dev20230411) (1.16.0)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.9/dist-packages (from matplotlib->convlab==3.0.1.dev20230411) (2.8.2)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->convlab==3.0.1.dev20230411) (4.39.3)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->convlab==3.0.1.dev20230411) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->convlab==3.0.1.dev20230411) (1.4.4)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib->convlab==3.0.1.dev20230411) (0.11.0)
Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->convlab==3.0.1.dev20230411) (5.12.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->convlab==3.0.1.dev20230411) (1.0.7)
Collecting Levenshtein==0.20.9
  Downloading Levenshtein-0.20.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.5/175.5 kB 12.5 MB/s eta 0:00:00
Collecting rapidfuzz<3.0.0,>=2.3.0
  Downloading rapidfuzz-2.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 65.9 MB/s eta 0:00:00
Requirement already satisfied: absl-py in /usr/local/lib/python3.9/dist-packages (from rouge-score->convlab==3.0.1.dev20230411) (1.4.0)
Collecting portalocker
  Downloading portalocker-2.7.0-py2.py3-none-any.whl (15 kB)
Collecting colorama
  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Requirement already satisfied: lxml in /usr/local/lib/python3.9/dist-packages (from sacrebleu->convlab==3.0.1.dev20230411) (4.9.2)
Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (1.1.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (3.1.2)
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (3.3.0)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (3.0.12)
Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (0.7.0)
Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (8.1.9)
Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (6.3.0)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (3.0.8)
Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (0.10.1)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (1.10.7)
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (1.0.4)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (2.0.7)
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (2.0.8)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (1.0.9)
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.9/dist-packages (from spacy->convlab==3.0.1.dev20230411) (2.4.6)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.9/dist-packages (from tensorboard->convlab==3.0.1.dev20230411) (2.17.1)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from tensorboard->convlab==3.0.1.dev20230411) (2.2.3)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.9/dist-packages (from tensorboard->convlab==3.0.1.dev20230411) (1.8.1)
Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.9/dist-packages (from tensorboard->convlab==3.0.1.dev20230411) (1.0.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.9/dist-packages (from tensorboard->convlab==3.0.1.dev20230411) (0.7.0)
Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.9/dist-packages (from tensorboard->convlab==3.0.1.dev20230411) (1.53.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.9/dist-packages (from tensorboard->convlab==3.0.1.dev20230411) (3.4.3)
Requirement already satisfied: tornado in /usr/local/lib/python3.9/dist-packages (from visdom->convlab==3.0.1.dev20230411) (6.2)
Collecting jsonpatch
  Downloading jsonpatch-1.32-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: websocket-client in /usr/local/lib/python3.9/dist-packages (from visdom->convlab==3.0.1.dev20230411) (1.5.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from visdom->convlab==3.0.1.dev20230411) (3.0)
Collecting async-timeout<5.0,>=4.0.0a3
  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting frozenlist>=1.1.1
  Downloading frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.8/158.8 kB 14.4 MB/s eta 0:00:00
Collecting aiosignal>=1.1.2
  Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting multidict<7.0,>=4.5
  Downloading multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.2/114.2 kB 12.8 MB/s eta 0:00:00
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets>=2.0->convlab==3.0.1.dev20230411) (22.2.0)
Collecting yarl<2.0,>=1.0
  Downloading yarl-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 264.6/264.6 kB 20.7 MB/s eta 0:00:00
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from google-auth<3,>=1.6.3->tensorboard->convlab==3.0.1.dev20230411) (5.3.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.9/dist-packages (from google-auth<3,>=1.6.3->tensorboard->convlab==3.0.1.dev20230411) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.9/dist-packages (from google-auth<3,>=1.6.3->tensorboard->convlab==3.0.1.dev20230411) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.9/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard->convlab==3.0.1.dev20230411) (1.3.1)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources>=3.2.0->matplotlib->convlab==3.0.1.dev20230411) (3.15.0)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.9/dist-packages (from markdown>=2.6.8->tensorboard->convlab==3.0.1.dev20230411) (6.1.0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->datasets>=2.0->convlab==3.0.1.dev20230411) (2022.7.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from scikit-learn->sentence-transformers>=2.2.2->convlab==3.0.1.dev20230411) (3.1.0)
Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.9/dist-packages (from thinc<8.2.0,>=8.1.8->spacy->convlab==3.0.1.dev20230411) (0.0.4)
Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.9/dist-packages (from thinc<8.2.0,>=8.1.8->spacy->convlab==3.0.1.dev20230411) (0.7.9)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/dist-packages (from werkzeug>=1.0.1->tensorboard->convlab==3.0.1.dev20230411) (2.1.2)
Collecting jsonpointer>=1.9
  Downloading jsonpointer-2.3-py2.py3-none-any.whl (7.8 kB)
Collecting torchvision
  Downloading torchvision-0.15.1-cp39-cp39-manylinux1_x86_64.whl (6.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 68.4 MB/s eta 0:00:00
  Downloading torchvision-0.14.1-cp39-cp39-manylinux1_x86_64.whl (24.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.2/24.2 MB 39.9 MB/s eta 0:00:00
  Downloading torchvision-0.14.0-cp39-cp39-manylinux1_x86_64.whl (24.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.3/24.3 MB 15.1 MB/s eta 0:00:00
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.9/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->convlab==3.0.1.dev20230411) (0.4.8)
Building wheels for collected packages: sentence-transformers, rouge-score, seqeval, visdom
  Building wheel for sentence-transformers (setup.py) ... done
  Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125942 sha256=89d5170d8740793618d416a7e38f6764267deb1f162a4cce002f5aa630c1a92a
  Stored in directory: /root/.cache/pip/wheels/71/67/06/162a3760c40d74dd40bc855d527008d26341c2b0ecf3e8e11f
  Building wheel for rouge-score (setup.py) ... done
  Created wheel for rouge-score: filename=rouge_score-0.1.2-py3-none-any.whl size=24954 sha256=e9dc960b1f2edd43ca05fcee0ea6f18fa8a7269f198a55c4f92a5ed06e046a57
  Stored in directory: /root/.cache/pip/wheels/9b/3d/39/09558097d3119ca0a4d462df68f22c6f3c1b345ac63a09b86e
  Building wheel for seqeval (setup.py) ... done
  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16180 sha256=ffbd9afcf332b427c5143fe19ff34f281ea4b8cbdf56b0ca2cb1fd6b71348798
  Stored in directory: /root/.cache/pip/wheels/e2/a5/92/2c80d1928733611c2747a9820e1324a6835524d9411510c142
  Building wheel for visdom (setup.py) ... done
  Created wheel for visdom: filename=visdom-0.2.4-py3-none-any.whl size=1408216 sha256=d2062f3a4b943481110ad6f0da2b0843e94c520bea3a5dcf3a96a0e8ea74b7cb
  Stored in directory: /root/.cache/pip/wheels/58/9e/14/30f7cc4dafdd4d602fb00ca33c6edd1424fc0f5df10a02e060
Successfully built sentence-transformers rouge-score seqeval visdom
Installing collected packages: tokenizers, sentencepiece, pydub, fuzzywuzzy, xxhash, unidecode, tensorboardX, simplejson, rapidfuzz, quadprog, portalocker, pillow, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cublas-cu11, multidict, jsonpointer, json_lines, joblib, jmespath, frozenlist, dill, deepspeech, colorama, async-timeout, yarl, sacrebleu, responses, nvidia-cudnn-cu11, multiprocess, Levenshtein, jsonpatch, huggingface-hub, gtts, embeddings, botocore, aiosignal, visdom, transformers, torch, seqeval, s3transfer, rouge-score, python-Levenshtein, aiohttp, torchvision, boto3, accelerate, sentence-transformers, datasets, convlab
  Attempting uninstall: pillow
    Found existing installation: Pillow 8.4.0
    Uninstalling Pillow-8.4.0:
      Successfully uninstalled Pillow-8.4.0
  Attempting uninstall: joblib
    Found existing installation: joblib 1.1.1
    Uninstalling joblib-1.1.1:
      Successfully uninstalled joblib-1.1.1
  Attempting uninstall: torch
    Found existing installation: torch 2.0.0+cu118
    Uninstalling torch-2.0.0+cu118:
      Successfully uninstalled torch-2.0.0+cu118
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.15.1+cu118
    Uninstalling torchvision-0.15.1+cu118:
      Successfully uninstalled torchvision-0.15.1+cu118
  Running setup.py develop for convlab
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchtext 0.15.1 requires torch==2.0.0, but you have torch 1.13.0 which is incompatible.
torchdata 0.6.0 requires torch==2.0.0, but you have torch 1.13.0 which is incompatible.
torchaudio 2.0.1+cu118 requires torch==2.0.0, but you have torch 1.13.0 which is incompatible.
pandas-profiling 3.2.0 requires joblib~=1.1.0, but you have joblib 1.2.0 which is incompatible.
Successfully installed Levenshtein-0.20.9 accelerate-0.18.0 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 boto3-1.26.110 botocore-1.29.110 colorama-0.4.6 convlab-3.0.1.dev20230411 datasets-2.11.0 deepspeech-0.9.3 dill-0.3.6 embeddings-0.0.8 frozenlist-1.3.3 fuzzywuzzy-0.18.0 gtts-2.3.1 huggingface-hub-0.13.4 jmespath-1.0.1 joblib-1.2.0 json_lines-0.5.0 jsonpatch-1.32 jsonpointer-2.3 multidict-6.0.4 multiprocess-0.70.14 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 pillow-9.5.0 portalocker-2.7.0 pydub-0.25.1 python-Levenshtein-0.20.9 quadprog-0.1.11 rapidfuzz-2.15.1 responses-0.18.0 rouge-score-0.1.2 s3transfer-0.6.0 sacrebleu-2.3.1 sentence-transformers-2.2.2 sentencepiece-0.1.97 seqeval-1.2.2 simplejson-3.19.1 tensorboardX-2.6 tokenizers-0.13.3 torch-1.13.0 torchvision-0.14.0 transformers-4.24.0 unidecode-1.3.6 visdom-0.2.4 xxhash-3.2.0 yarl-1.8.2

[Feature] add SOLOIST & SYNERGY

Describe the feature
Add SOLOIST and SYNERGY

Expected behavior

reproduce result in static E2E evaluation
report performance of simulated E2E dialog

Additional context
Code should be tested in the following environment:

python >= 3.6
torch >= 1.6
transformers >= 4.0

MultiWozEvaluator's add_sys_da doesn't use self.cur_domain

This may be a boring question.
In the previous version (ConvLab-2), MultiWozEvaluator's add_sys_da used self.cur_domain to update self.booked. However, the current version (this ConvLab-3) does not.

convlab2's multiwoz_eval.py L102-106

if da == 'booking-book-ref' and self.cur_domain in ['hotel', 'restaurant', 'train']:
    if not self.booked[self.cur_domain] and re.match(r'^\d{8}$', value) and \
            len(self.dbs[self.cur_domain]) > int(value):
        self.booked[self.cur_domain] = self.dbs[self.cur_domain][int(value)].copy()
        self.booked[self.cur_domain]['Ref'] = value

the same process in convlab3; multiwoz_eval.py L190-104

if not self.booked[domain] and re.match(r'^\d{8}$', value) and \
        len(self.dbs[domain]) > int(value):
    self.booked[domain] = self.dbs[domain][int(
        value)].copy()
    self.booked[domain]['Ref'] = value

Both update self.cur_domain like that:

# convlab2 L96,97 / convlab3 L177,178
if domain in belief_domains and domain != self.cur_domain:
    self.cur_domain = domain

But it is only used ConvLab-2.

I don't see any errors, so it may be fine as it is, but why?

Policy for sgd dataset [BUG]

I was trying to build build the complete agent pipeline on Schema-guided (sgd) dataset but I got the following error:

[BUG] state annotations of unified SGD multi-domain dialogs

Describe the bug
In the original SGD dataset, they only annotate the dialog state of current active domain at each turn. For example, if the user first talk about domain A and then domain B, the state annotation will be like {A,A,...,A,A+B,B,...B}, where only the domain-switching turn will have annotation for both domains. However, when I transformed the dataset to the unified format, I included all domains (A+B) in the state annotation, but set the states of inactive domains to empty (values of all slots are empty string). This is inconsistent with MultiWOZ annotation where the state of the previous domain is preserved instead of set to empty.

Expected behavior
state of each turn contains accumulated annotation of all domains, which is consistent with MultiWOZ.

Actual behavior
state of each turn only contains annotation of the current domain while set all previous domain's annotation to empty.

Additional context
the models that use SGD's state annotation need to be re-trained and relevant experiments need to be run again. I think that involve the pre-train-then-fine-tune and joint training experiments of DST, SOLOIST, Policy and TUS.

Error with Seq2Seq2 training on multiple GPUs.

I am encountering an error when trying to use the Seq2Seq2 trainer to fine-tune T5 on multiple GPUs. The error message is as follows:

../.conda/envs/stance/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. warnings.warn('Was asked to gather along dimension 0, but all '

The warning message suggests that a gather operation is being performed along dimension 0 on a set of input tensors, but all of the input tensors are scalars. The error occurs during training and causes the training process to crash.

Note that when I set export CUDA_VISIBLE_DEVICES=0 to ensure that the training runs on a single GPU, the training process completes successfully without the warning mentioned earlier.

I am using PyTorch version 1.13.0.

Can you please advise on how to resolve this error? Is there a specific configuration or setting that needs to be modified when using the Seq2Seq2 trainer to fine-tune T5 on multiple GPUs?

[Feature] add TripPy DST

Describe the feature
Add TripPy DST

Expected behavior

Add TripPy model under convlab2/dst.
Report the model performance on DST and E2E simulation.

Additional context
Code should be tested in the following environment:

python >= 3.6
torch >= 1.6
transformers >= 4.0

Question: What is the difference between the `sgd` and `sgd1` ... `sgd5` datasets?

HI all!

I'm working on some experiments using Convlab and the unified dataset format, and wanted to run some with the SGD dataset. I saw there is an sgd for SGD, and it looks like sgd1 is for SGD-X, but what are the remaining dataset versions for, and which would be appropriate for use when comparing to prior work on these datasets?

DST for sgd dataset [BUG]

Trying to use DST for sgd dataset but I got the following error:

I am using (update) method in DST class similar to convlab-2..

Evaluate models' performance of each module using the standard transformations.

[Maintenance] pip install convlab fails due to version conflict in Colab

Describe the feature
Currently it is not possible to use the Tutorial Colab due to version conficts in convlab.

https://colab.research.google.com/github/ConvLab/ConvLab-3/blob/master/tutorials/Getting_Started.ipynb#scrollTo=qPqTs6EP0wxO

! pip install convlab ends in following error.
**ERROR: Cannot install convlab==3.0.0 and convlab==3.0.1 because these package versions have conflicting dependencies.

The conflict is caused by:
convlab 3.0.1 depends on deepspeech
convlab 3.0.0 depends on deepspeech**

Expected behavior
pip install work successfully

Additional context
Add any other context about the feature here.

[BUG] keyerror: 'semi'

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Go to Getting_Started.ipynb
Click on sys_agent.response("I want to find a moderate hotel")
Scroll down to '....'
See error

Expected behavior
no bug

Actual behavior
KeyError Traceback (most recent call last)
Cell In[3], line 1
----> 1 sys_agent.response("I want to find a moderate hotel")

File ConvLab-3/convlab/dialog_agent/agent.py:175, in PipelineAgent.response(self, observation)
172 state = deepcopy(state) # get rid of reference problem
173 # get action
174 # get rid of reference problem
--> 175 self.output_action = deepcopy(self.policy.predict(state))
177 # get model response
178 if self.nlg is not None:

File ConvLab-3/convlab/policy/rule/multiwoz/rule.py:31, in RulePolicy.predict(self, state)
23 def predict(self, state):
24 """
25 Predict an system action given state.
26 Args:
(...)
29 action : System act, with the form of (act_type, {slot_name_1: value_1, slot_name_2, value_2, ...})
30 """
---> 31 return self.policy.predict(state)

File ConvLab-3/convlab/policy/rule/multiwoz/rule_based_multiwoz_bot.py:96, in RuleBasedMultiwozBot.predict(self, state)
94 # User's talking about other domain
...
--> 184 for slot in state['belief_state'][domain.lower()]['semi']:
185 if state['belief_state'][domain.lower()]['semi'][slot] != "":
186 constraints.append([slot, state['belief_state'][domain.lower()]['semi'][slot]])

KeyError: 'semi'

Additional context
Add any other context about the problem here.

[BUG] Should use "-" instead of "_" to connect dialogue act.

Describe the bug

When example code PPO.py is run, ValueError occurs.

File [MY CURRENT DIRECTORY]/ConvLab-3/convlab/util/multiwoz/lexicalize.py:37, in deflat_da(meta)
     35 dialog_act = {}
     36 for da in meta:
---> 37     d, i, s, v = da.split('_')
     38     k = '_'.join((d, i))
     39     if k not in dialog_act:

ValueError: not enough values to unpack (expected 4, got 1)

In VectorBase class, the dialogue act is joined and split by _, but ontology is connected by -.

Use - to join and split. Or replace - with _ of vec2act and act2vec in VectorBase class initialization.

To Reproduce

Run examples/agent_examples/test_PPO.py.

The original code gets the PPO arguments wrong, so fix line 22 that:

- sys_policy = PPO(False, seed=seed, use_action_mask=True, shrink=False)
+ sys_policy = PPO(False, seed=seed)

I use seed=0 and n_dialogues=10 but it does not matter.

Expected behavior
PPO.py runs without error.

Actual behavior
as stated above

Additional context
I have only tried PPO.py but this bug may affect some other code as the VectorBase class is used by many modules.

soloist.py import error[BUG]

In convlab/e2e/soloist/multiwoz/soloist.py

in 13th line SOLOIST is tried to be imported from soloist_net, but soloist_net.py does not exist.

from convlab.e2e.soloist.multiwoz.soloist_net import SOLOIST, cuda_

[BUG] AttributeError: module 'main' has no attribute 'spec'

I did train (t5-small) model on a dataset in Convlab unified format. But when comes to test ( --do_predict ) I got the following error:

(

WARNING:root:nlu info_dict is not initialized[BUG]

Describe the bug
When I execute the code in getting_started in Pycharm, it warned as the title without outputting any responds, I am not sure how to solve this.
By the way, I downloaded the pretrained DDPT model manually and I put the model in convlab/policy/vtrace_DPT. Is this the correct directory?

[Feature] add LAVA policy

Describe the feature
add LAVA policy model

Expected behavior

Add LAVA under convlab2/policy.
Report the model performance on Policy & E2E simulation.

Additional context
Code should be tested in the following environment:

python >= 3.6
torch >= 1.6
transformers >= 4.0

Train MLE policy for sgd dataset

I was trying to train MLE policy for sgd. I've added default_state in convlab.util.sgd.state (similar to multiwoz21) but I got the following error. Would you please guid me in this:

NLU Usage for JointBERT and sgd dataset [BUG]

I've already trained the model on sgd user (context=3). I am trying to use it now (i.e. nlu.predict) but I got the following error:

[Feature] Deploy folder(Interactive Tool) update

Describe the feature
Will you update the deploy folder? I want to try interactive tools.

Expected behavior
Convlan-2 has a perfect Interactive Tool demo, Is it possible to update to Convlab-3?

Additional context
I try to run the deploy demo in ConvLab-3, but I failed!

base_models/BERT

What is the difference between JointBERT and base_models/BERT for NLU task?

is there a "read me" file or tutorial for the later?

(https://github.com/ConvLab/ConvLab-3/tree/master/convlab/base_models/bert)

Many thanks

Replicating the performance of T5-small (context=3) NLU on MultiWOZ 2.1

Hello, I'm trying to understand how were those numbers achieved for the T5-small (context=3) NLU model on the MultiWOZ 2.1 dataset:

I'd like to replicate this performance locally but I'm probably misunderstanding how it was calculated or just doing something wrong. There doesn't seem to be a training script in the T5 NLU directory for this model, so I made my own consulting the HF model page:

n_gpus=1
task_name="nlu"
dataset_name=multiwoz21
speaker="user"
context_window_size=3
data_dir="data/${task_name}/${dataset_name}/${speaker}/context_${context_window_size}"
output_dir="output/${task_name}/${dataset_name}/${speaker}/context_${context_window_size}"
cache_dir="../cache"
logging_dir="${output_dir}/runs"
train_file="${data_dir}/train.json"
validation_file="${data_dir}/validation.json"
test_file="${data_dir}/test.json"
metric_name_or_path="nlu_metric.py"
metric_for_best_model="overall_f1"
source_column="context"
target_column="dialogue_acts_seq"
truncation_side="left"
max_source_length=512
max_target_length=512
model_name_or_path="t5-small"
per_device_train_batch_size=128
per_device_eval_batch_size=64
gradient_accumulation_steps=2
lr=1e-3
num_train_epochs=10

python ../create_data.py -t ${task_name} -d ${dataset_name} -s ${speaker} -c ${context_window_size}

python ../run_seq2seq.py \
    --task_name ${task_name} \
    --train_file ${train_file} \
    --validation_file ${validation_file} \
    --source_column ${source_column} \
    --target_column ${target_column} \
    --max_source_length ${max_source_length} \
    --max_target_length ${max_target_length} \
    --truncation_side ${truncation_side} \
    --model_name_or_path ${model_name_or_path} \
    --do_train \
    --do_eval \
    --save_strategy epoch \
    --evaluation_strategy epoch \
    --save_total_limit 1 \
    --prediction_loss_only \
    --cache_dir ${cache_dir} \
    --output_dir ${output_dir} \
    --logging_dir ${logging_dir} \
    --overwrite_output_dir \
    --preprocessing_num_workers 4 \
    --per_device_train_batch_size ${per_device_train_batch_size} \
    --per_device_eval_batch_size ${per_device_eval_batch_size} \
    --gradient_accumulation_steps ${gradient_accumulation_steps} \
    --learning_rate ${lr} \
    --num_train_epochs ${num_train_epochs} \
    --optim adafactor \
    --gradient_checkpointing

python ../run_seq2seq.py \
    --task_name ${task_name} \
    --test_file ${test_file} \
    --source_column ${source_column} \
    --target_column ${target_column} \
    --max_source_length ${max_source_length} \
    --max_target_length ${max_target_length} \
    --truncation_side ${truncation_side} \
    --model_name_or_path ${output_dir} \
    --do_predict \
    --predict_with_generate \
    --metric_name_or_path ${metric_name_or_path} \
    --cache_dir ${cache_dir} \
    --output_dir ${output_dir} \
    --logging_dir ${logging_dir} \
    --overwrite_output_dir \
    --preprocessing_num_workers 4 \
    --per_device_train_batch_size ${per_device_train_batch_size} \
    --per_device_eval_batch_size ${per_device_eval_batch_size} \
    --gradient_accumulation_steps ${gradient_accumulation_steps} \
    --learning_rate ${lr} \
    --num_train_epochs ${num_train_epochs} \
    --optim adafactor \
    --gradient_checkpointing

python merge_predict_res.py -d ${dataset_name} -s ${speaker} -c ${context_window_size} -p ${output_dir}/test_generated_predictions.json

python ../../../nlu/evaluate_unified_datasets.py -p ${output_dir}/predictions.json

But the performance I got with a model trained this way is subpar:

{'accuracy': 0.5291644058600109,
 'binary': {'f1': 0.0, 'precision': 0.0, 'recall': 0.0},
 'categorical': {'f1': 0.0, 'precision': 0.0, 'recall': 0.0},
 'non-categorical': {'f1': 0.0, 'precision': 0.0, 'recall': 0.0},
 'overall': {'f1': 0.649805447470817,
             'precision': 0.8097811027985592,
             'recall': 0.5426104715930189}}

Next, I tried a second approach where I just downloaded the trained model from HF and did predictions with it, but the performance was the same:

{'accuracy': 0.5294357026587087,
 'binary': {'f1': 0.0, 'precision': 0.0, 'recall': 0.0},
 'categorical': {'f1': 0.0, 'precision': 0.0, 'recall': 0.0},
 'non-categorical': {'f1': 0.0, 'precision': 0.0, 'recall': 0.0},
 'overall': {'f1': 0.6505221061986226,
             'precision': 0.8097345132743363,
             'recall': 0.5436316375789083}}

Outputs are from evaluate_unified_datasets.py, which is run as part of the .sh script as can be seen above. Judging by what's written here, it's what was used to calculate the "correct" performance of the model. I must be missing something. Can you help me with this?

preprocess.py in jointBERT / unified_datasets [BUG]

I was trying to use JointBERT NLU for sgd dataset. I got the following error

[BUG] ModuleNotFoundError: No module named 'allennlp.data.tokenizers.word_splitter'

Describe the bug
ModuleNotFoundError: No module named 'allennlp.data.tokenizers.word_splitter'

system env:
macbook pro m1, ventura 13.0.1
Anaconda3-2022.10-MacOSX-arm64

conda env:
conda create -n convlab3 python=3.9
pip install -e .
pip install allennlp, overrides

To Reproduce
Steps to reproduce the behavior:

Go to tutorials/Getting_Started.ipynb
Click on the first cell of "build an agent"
Scroll down to '....'
See error

Expected behavior
no bug

Actual behavior
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /var/folders/4t/jbjy985148dd53fy6b34rtjw0000gn/T/ipykernel_47306/1050441968.py:3 in │
│ │
│ [Errno 2] No such file or directory: │
│ '/var/folders/4t/jbjy985148dd53fy6b34rtjw0000gn/T/ipykernel_47306/1050441968.py' │
│ │
│ /Users//03nlp/chatbot/ConvLab-3/convlab/nlu/milu/multiwoz/init.p │
│ y:1 in │
│ │
│ ❱ 1 from convlab.nlu.milu.multiwoz.nlu import MILU │
│ 2 │
│ │
│ /Users//03nlp/chatbot/ConvLab-3/convlab/nlu/milu/multiwoz/nlu.py:12 │
│ in │
│ │
│ 9 import torch │
│ 10 from allennlp.common.checks import check_for_gpu │
│ 11 from allennlp.data import DatasetReader │
│ ❱ 12 from allennlp.data.tokenizers.word_splitter import SpacyWordSplitter │
│ 13 # from allennlp.data.tokenizers import spacy_tokenizer │
│ 14 from allennlp.models.archival import load_archive │
│ 15 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'allennlp.data.tokenizers.word_splitter'

Additional context
Add any other context about the problem here.

[Feature] Transfer learning experiments for NLU, DST, NLG, E2E

Describe the feature
conduct transfer learning experiments for

sgd+tm => multiwoz21 (1%, 10%, 100%)

Expected behavior
results will be like: (Model B can be a pre-trained version of Model A)

Additional context
Should use the same data for few-shot experiments like this:

dataset = load_dataset('multiwoz21', dial_ids_order=0)
train_ratio = 0.1
dataset['train'] = dataset['train'][:round(len(dataset['train'])*train_ratio)]

dial_ids_order in [0, 9]

[Feature] add RL policy toolkit

Describe the feature
add RL policy toolkit and Fix bug for book signal

Expected behavior

Write a README to introduce the features of the RL policy toolkit.
report the performance of the existing RL policy.

Additional context
Add any other context about the feature here.

[Feature] Train a base model using unified dataset format

Describe the feature
Train a domain agnostic T5 model as a base model to solve different tasks in a text-to-text generation fashion.

Expected behavior
Support tasks:

Report the performance on NLU, DST, Policy, NLG, and end-to-end.

Additional context
Add any other context about the feature here.

[Question/Issue] Should SOLOISTAgent condition generation on DB state?

Hi all! Thanks for building this all out, its a very useful framework for interacting with dialogue systems. I was looking at SOLOISTAgent, and if I understand correctly, it does not actually condition on the DB result provided by a belief state, but instead:

predicts a belief state and delexicalized response at the same time
uses the belief state to fetch DB entries with a reader/rule-based matcher
lexicalizes items in the response with values from any DB entries found

If I understand right, the paper seems to imply that at response generation time, the number of DB entries found is known, and included in the input sequence:

Should there not be two generation steps, first generating a belief state, then a response based on state + retrieved DB entries? I thought this might be a bug but noticed a similar issue in the SOLOIST repo: pengbaolin/soloist#2

Similarly, could one use the existing SOLOISTAgent approach to reproduce their MultiWOZ results, assuming you had the right pre-trained weights?

[BUG] Spelling issue at ConvLab-3\convlab\base_models\t5\nlu\nlu.py line 1

Describe the bug

in file: ConvLab-3\convlab\base_models\t5\nlu\nlu.py, there is a spelling issue in line 1

there is:

gimport logging

should be:

import logging

To Reproduce

Go to Google Colab
Run: ! git clone --depth 1 https://github.com/ConvLab/ConvLab-3.git && cd ConvLab-3 && pip install -e .
Run: ! python -m spacy download en_core_web_sm
Run: from convlab.base_models.t5.nlu import T5NLU

Expected behavior

T5NLU imported

Actual behavior

Error:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3326, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)

File "", line 1, in
from convlab.base_models.t5.nlu import T5NLU

File "/content/ConvLab-3/convlab/base_models/t5/nlu/init.py", line 1, in
from convlab.base_models.t5.nlu.nlu import T5NLU

File "/content/ConvLab-3/convlab/base_models/t5/nlu/nlu.py", line 1
gimport logging
^
SyntaxError: invalid syntax

Additional context

[BUG] Trying to replicate TUS evaluation

Hello, thanks a lot for your work. I am very excited to use ConvLab 3, although I just started exploring it. So please forgive me for my mistakes. I am trying to run and evaluate TUS. This is the script I wrote, which is based on examples/agent_examples/test_BERTNLU-RuleDST-TUS-TemplateNLG.py and issue #101.

from convlab.nlu.jointBERT.multiwoz import BERTNLU
from convlab.nlg.template.multiwoz import TemplateNLG
from convlab.dialog_agent import PipelineAgent
from convlab.util.analysis_tool.analyzer import Analyzer
import random
import json
import numpy as np
import torch

from convlab.dst.rule.multiwoz.usr_dst import UserRuleDST
from convlab.policy.tus.multiwoz.TUS import UserPolicy

from convlab.base_models.t5.nlu import T5NLU
from convlab.base_models.t5.dst import T5DST
from convlab.base_models.t5.nlg import T5NLG
from convlab.policy.vector.vector_nodes import VectorNodes
from convlab.policy.vtrace_DPT import VTRACE


def set_seed(r_seed):
    random.seed(r_seed)
    np.random.seed(r_seed)
    torch.manual_seed(r_seed)


def test_end2end():
    # specify the user config
    user_config = "convlab/policy/tus/multiwoz/exp/default.json"
    user_mode = ""
    # BERT nlu trained on sys utterance
    user_nlu = BERTNLU()
    user_dst = UserRuleDST()
    # rule policy
    user_config = json.load(open(user_config))

    sys_nlu = T5NLU(speaker='system', context_window_size=0, model_name_or_path='ConvLab/t5-small-nlu-multiwoz21')
    sys_dst = T5DST(dataset_name='multiwoz21', speaker='system', context_window_size=100,
                    model_name_or_path='ConvLab/t5-small-dst-multiwoz21')
    # Download pre-trained DDPT model
    # ! wget https://huggingface.co/ConvLab/ddpt-policy-multiwoz21/resolve/main/supervised.pol.mdl- -directory - prefix = "convlab/policy/vtrace_DPT"
    vectorizer = VectorNodes(dataset_name='multiwoz21',
                             use_masking=True,
                             manually_add_entity_names=True,
                             seed=0,
                             filter_state=True)
    sys_policy = VTRACE(is_train=False,
                        seed=0,
                        vectorizer=vectorizer,
                        load_path="convlab/policy/vtrace_DPT/supervised")
    sys_nlg = T5NLG(speaker='system', context_window_size=0, model_name_or_path='ConvLab/t5-small-nlg-multiwoz21')
    # assemble
    sys_agent = PipelineAgent(sys_nlu, sys_dst, sys_policy, sys_nlg, name='sys')

    if user_mode:
        user_config["model_name"] = f"{user_config['model_name']}-{user_mode}"
    user_policy = UserPolicy(user_config)
    # template NLG
    user_nlg = TemplateNLG(is_user=True)
    # assemble
    user_agent = PipelineAgent(
        user_nlu, user_dst, user_policy, user_nlg, name='user')

    analyzer = Analyzer(user_agent=user_agent, dataset='multiwoz')

    set_seed(20200202)
    analyzer.comprehensive_analyze(
        sys_agent=sys_agent, model_name='BERTNLU-RuleDST-TUS-TemplateNLG', total_dialog=1000)


if __name__ == '__main__':
    test_end2end()

Overall, the dialogs work fine, but sometimes they end up in some errors. So I feel I probably made some mistakes in the script.

ERROR 1

  File ".../convlab/util/multiwoz/lexicalize.py", line 90, in lexicalize_da
    elif slot in state[domain]:
KeyError: 'train'

My guess is that VTRACE predicts an action of a domain different from the state. In fact the state was about hospital and the action was about the train domain.

How I (hot)fixed it
I modified line 90 of File ".../convlab/util/multiwoz/lexicalize.py"

elif slot in state[domain]:

as follows:

elif state.get(domain) is not None and slot in state.get(domain):

in this way, that "elif" condition will be skipped and it would end up in line 93: pair[1] = 'not available'

ERROR 2

  File ".../convlab/util/analysis_tool/analyzer.py", line 119, in comprehensive_analyze
    sys_response, user_response, session_over, reward = sess.next_turn(
  File ".../convlab/dialog_agent/session.py", line 122, in next_turn
    user_response = self.next_response(last_observation)
  File ".../convlab/dialog_agent/session.py", line 96, in next_response
    response = next_agent.response(observation)
  File ".../convlab/dialog_agent/agent.py", line 176, in response
    self.output_action = deepcopy(self.policy.predict(state))
  File ".../convlab/policy/tus/multiwoz/TUS.py", line 420, in predict
    return self.policy.predict(state)
  File ".../convlab/policy/tus/multiwoz/TUS.py", line 89, in predict
    self.goal.add_sys_da(sys_dialog_act)
  File ".../convlab/policy/tus/multiwoz/Goal.py", line 76, in add_sys_da
    self.evaluator.add_sys_da(sys_act)
  File ".../convlab/evaluator/multiwoz_eval.py", line 190, in add_sys_da
    if not self.booked[domain] and re.match(r'^\d{8}$', value) and \
KeyError: 'booking'

I've seen that the "booking" domain has been deleted or commented across the code base. Is there a reason behind this? Is this error expected and how can I fix it?

How I (hot)fixed it
I replaced line 190 of File ".../convlab/evaluator/multiwoz_eval.py"

if not self.booked[domain] and re.match(r'^\d{8}$', value) and \

with

if self.booked.get(domain) is not None and not self.booked[domain] and re.match(r'^\d{8}$', value) and \

ERROR 3

  File ".../convlab/policy/vtrace_DPT/transformer_model/EncoderDecoder.py", line 102, in select_action
    description_idx_list, value_list = self.get_descriptions_and_values(kg_list)
  File ".../convlab/policy/vtrace_DPT/transformer_model/EncoderDecoder.py", line 81, in get_descriptions_and_values
    description_idx_list = self.node_embedder.description_2_idx(kg_list[0]).to(DEVICE)
 File ".../convlab/policy/vtrace_DPT/transformer_model/node_embedder.py", line 81, in description_2_idx
    embedded_descriptions_idx = torch.Tensor([self.description2idx[node["description"]] for node in kg_info])\
  File ".../convlab/policy/vtrace_DPT/transformer_model/node_embedder.py", line 81, in <listcomp>
    embedded_descriptions_idx = torch.Tensor([self.description2idx[node["description"]] for node in kg_info])\
KeyError: 'user goal-hotel-book time'

It seems that 'user goal-hotel-book time' is not in file policy/vtrace_DPT/descriptions/semantic_information_descriptions_multiwoz21.json. Is it expected? How can I fix it?

I would really appreciate a feedback on these issues, and if possible some support on how to evaluate the user simulator. That's actually my main goal. Thanks a lot,

Silvia

Suggestion: Use More Up-to-date Version of AllenNLP?

Hello,

I've tried running the tutorials and bumped into a wall, though this is not as major an issue as I was worried about.

allennlp.data.tokenizers.word_splitter is only used in MILU, but for those on more up-to-date versions of AllenNLP (like myself, who just pip installed it to run the tutorial notebook) will be stopped here as WordSplitter appears to have been replaced by a more general tokeniser after the 1.0.0 release: allenai/allennlp#3345.

A solution is to obviously only use the relevant libraries (which I could have avoided from the get-go, as I will only be playing around with generation libraries). I appreciate that you might have your very own reasons for using an older version, but updating this and any other similar use cases (or at least a disclaimer) would be a great time-saver for beginner-level people like myself who are just looking to get started with this.

[BUG] colab error

the first and second step are success, but the third step is wrong.

[Feature] Add data processing code for models using the standard transformations #14 for training and evaluation #15

Describe the feature
Add data processing code for the following models using the standard transformations #14 and evaluate using #15.

NLU:

MILU
BERTNLU

DST:

TRADE
SUMBT
Neural Dialogue Belief Tracker (Van Niekek et al., 2021)
TripPy

Policy:

NLG:

SC-LSTM
SC-GPT

E2E:

SOLOIST
DAMD

Expected behavior
These models can be trained on other unified datasets easily.

[Feature] add domain-independent user simulator & user Rule-DST

Describe the feature
add Domain-independent user simulator

Expected behavior

the simulator can be used for policy training and E2E evaluation.
Report simulated dialog performance compared with existing simulators

Additional context
Code should be tested in the following environment:

python >= 3.6
torch >= 1.6
transformers >= 4.0

TripPy DST

I trained TripPy on a dataset in the unified format. I have the path to a model checkpoint. When I use it, I got the following error:

Then, I changed the followings in trippy/tracker.py

_ontology = relative_import_module_from_unified_datasets(self.dataset_name, 'preprocess.py', 'ontology')

_ontology = load_ontology(self.dataset_name)

I got the following error:

[Feature] add datasets in unified format

Describe the feature
Transform the following datasets into the Unified format under data/unified_datasets. Suggestions of adding/deleting datasets are welcome!

Expected behavior
Unified datasets can support related tasks.

Add data processing code of existing models using the standard transformations.

[Questions] Training PPO policy

Hi,

I really appreciate the time and efforts you put in this project. I you don't mind, I have a question regarding training PPO policy:

When I start training I got the following text files "unknown slots" and "Fuzzy recognition" (below). Does this mean there is something wrong ?

Thank you for help and support..

[BUG] Evaluate PPO (using --goals_from_data )

When I evaluate PPO policy using and add the argument --goals_from_data, I got the following error:

This error when using GenTUS for evaluation:

This error when using ABUS for evaluation:

[Feature] add module data transformation using unified datasets

Describe the feature
To facilitate the data processing of models under the same setting, we define standard module data transformations using unified datasets:

NLU&NLG: (split to user/system side)
- single turn: each sample contains an utterance and corresponding dialog acts.
- multi-turn: each sample contains an utterance, corresponding dialog acts, and dialog context.
DST: (system side only)
- each sample contains a dialog context and corresponding belief state.
Policy: (split to user/system side)
- act-level: each sample contains current dialog acts, last opponent dialog acts, and belief state.
- word-level: each sample contains current utterance, dialog context, and belief state.
E2E:
- each sample contains a dialog context, current utterance, current dialog acts, last opponent dialog acts, belief state, db results.

Note that these transformations will be used in Benchmark models evaluation.
You may also customize your data transformation to process the unified datasets, and comment here if you think it should be added to ConvLab.

Expected behavior
Add transformation functions in convlab2/util/unified_datasets_util.py.

Additional context
Previous dataloaders are in convlab2/util/dataloader/module_dataloader.py. Could be a good reference.

Example scripts for rule dst and policy raise AssertionError

I ran the example script in RuleDST and RulePolicy and got AssersionErrors.
It's just because those scripts are outdated, and can I ignore them? Thanks!

ModuleNotFoundError: No module named 'convlab.dst.sumbt.multiwoz' [BUG]

Code missing for sumbt. How to fix?

There are many scripts that rely on sumbt, e.g., python test_SUMBT-LaRL.py.

[Question] No module named 'data.unified_datasets'

Hi. Any suggestions to solve this issue..

First, I have the data folder (which contains the unified_datasets folder) here, outside the convlab folder:

I was trying to initialise an object from VectorBase class as follows:

As you can see, I got the exception message here, No module named 'data.unified_datasets' so self.db_domains and self.db not correctly initialized:

Thank you to the authors for their valuable contributions and exceptional support..

Static values explanation

Greetings,
I am trying to implmenet my own custom dataset in unified format and eventually train a PieplineAgent using my DST implementation, a DQN Policy and a TemplateNLG. I saw that the current implementation of the DQN only supports multiwoz21. As such, I started to make a few modifications to suit my needs and found out I need to implement a custom Vector as well. Looking at VectorBinary, I see this

def get_state_dim(self):
        self.belief_state_dim = 0
        for domain in self.ontology['state']:
            for slot in self.ontology['state'][domain]:
                self.belief_state_dim += 1

        self.state_dim = self.da_opp_dim + self.da_dim + self.belief_state_dim + \
            len(self.db_domains) + 6 * len(self.db_domains) + 1

I do not understand the last 3 factors to the sum. What is the intuiton behind 6* len(self.db_domains) + 1 ?
Thank you for your time !

[Feature] add Neural Dialogue Belief Tracker

Describe the feature
add Neural Dialogue Belief Tracker

Expected behavior

Add a dialog state tracking model with uncertainty measurement under convlab2/dst.
Improve policy learning.
Report the model performance on DST & Policy.

Additional context
Code should be tested in the following environment:

python >= 3.6
torch >= 1.6
transformers >= 4.0

[Feature] Benchmark Evaluation Script

Describe the feature
Provide unified, fast evaluation script for Benchmark using unified datasets #11 and standard dataloaders #14
Evaluation metrics:

NLU: Acc, dialog act tuple P/R/F1
DST: JGA, slot Acc, slot P/R/F1
NLG: BLEU, SER
E2E static: Inform, success, BLEU, combined score
Simulation: success rate, inform P/R/F1, avg. turn

Expected behavior
The scripts can evaluate different models of a module in the same ways to make the results comparable.

Additional context
Previous evaluation scripts convlab2/$module/evaluate.py could be a reference but they do not use batch inference.

JointBERT NLU training for sgd dataset (system) [BUG]

I did the preprocessing step and added the directory of the preprocessed data in the config file. When I tried to do the training, I got the following error:

%run train.py --config_path xxxx\sgd_sys_context3.json

convlab / convlab-3 Goto Github PK

convlab-3's Introduction

ConvLab

Package Overview

Installation

Installing via pip

Setting up a virtual environment

Installing the library and dependencies

Installation tips on CentOS

Installing using Docker

Running ConvLab

Creating a new spec file

Participation in DSTC-8

Evaluation

Contributions

Citing

convlab-3's People

Contributors

Stargazers

Watchers

Forkers

convlab-3's Issues

Recommend Projects

Recommend Topics

Recommend Org