Comments (9)
We did not evaluate the rule DST solely since it needs dialog acts as input. If you want to compare rule DST with other DST models, you may use the golden dialog acts as input or use an NLU model such as BERTNLU to parse both user and system acts.
from convlab-2.
I would like to use the output of BERTNLU as the input for the dst; however, it is not clear for me how to pass the data from one module to another, and I haven't find any code for that in convlab, for the moment.
Could you kindly link the convlab's page where this is described, or provide me more information about this process?
from convlab-2.
You can refer to the Colab tutorial or the interface class for nlu and dst. You can see PipelineAgent for how to build an agent with modules. Example usage:
https://github.com/thu-coai/ConvLab-2/blob/master/tests/test_BERTNLU-RuleDST-RulePolicy-TemplateNLG.py
from convlab-2.
Thank you for the info. Nevertheless, the Colab tutorial refers to an overall evaluation (nlu + dst+ nlg).
What if I would like to evaluate the nlu+dst only, in order to analyze if the defined rules are ok or need some improvements? Is that possible? Thanks again
from convlab-2.
Sure. Just feed the output of NLU to DST:
ConvLab-2/convlab2/dialog_agent/agent.py
Lines 122 to 132 in ad32b76
from convlab-2.
From the code you posted it doesn't seem that the module is evaluated with F1 scores or a similar measure... perhaps I don't understand your point...
from convlab-2.
Sorry, I thought you need instruction about how to pass the output of NLU to DST. If you want to evaluate NLU+DST, you can write a script to: 1) read the original data; 2) pass utterances to NLU to get the user dialog acts; 3) pass user dialog acts to RuleDST to get predicted state; 4) compare predictions with references
from convlab-2.
refer to https://github.com/thu-coai/ConvLab-2/blob/master/convlab2/dst/evaluate.py for dst metric
from convlab-2.
Ok, thanks, I'll try in this way!
from convlab-2.
Related Issues (20)
- [BUG] `try` does NOT work HOT 1
- Training Data HOT 1
- [Maintenance] pip takes too long in finding boto3 versions (100+ tries & still failed to install) HOT 7
- [BUG] Failed to build agent on CoLab HOT 2
- Spacy latest version compatibility HOT 2
- Different end-to-end results of DAMD HOT 1
- spacy tokenizer HOT 1
- Integrating with my own dataset HOT 3
- [BUG] 关于中文数据集crosswoz上policy-rule代码的问题 HOT 1
- BERTNLU postprocess.py 为什么可以通过 if intent_logits[j] > 0: 来获得intent的预测呢? HOT 2
- [Maintenance] docker for m1 mac, please HOT 1
- Issue in BertNLU HOT 4
- Unable to get the pretrained BERTNLU model. HOT 7
- Unable to get the pretrained BERTNLU model even after updating the URLs HOT 7
- installation python 3.7 on CentOS7 failed[BUG] HOT 2
- The Link to the datasets of LAUG is unreachable HOT 3
- [BUG] Failed to build tokenizers HOT 3
- 是否可以使用不经过训练后的BERTNLU呢 HOT 7
- Why the end-2-end performance mismatch with component level evaluation? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from convlab-2.