This repository contains the implementation of the research paper A post-processing framework for detecting unknown intent of dialogue system via pre-trained deep neural network classifier, submitted to Knowledge-based Systems by Tingen Lin, Hua Xu

In this paper, we propose SofterMax and deep novelty detection (SMDN), a simple yet effective post-processing method for detecting unknown intent in dialogue systems based on pre-trained deep neural network classifiers . We calibrate the confidence of DNN classifier via temperature scaling to make more reasonable/conservative prediction. Here is an example illustrating the difference between Softmax and SofterMax:

The architecture of the proposed SMDN method for the single-turn dialogue system.

The architecture of the proposed SMDN method for the multi-turn dialogue system.

1. Install all required library

pip install -r requirements.txt

2. Unzip and preprocess SwDA dataset (and download your GloVe embedding & change the path in file)

unzip data/swda.zip
python preprocessing_swda.py

3. Get the pre-trained intent classifiers with different settings

python train.py

or you can get a single pre-trained classifier with

python train_BiLSTM.py <dataset> <proportion>
python train_BiLSTM-DOC.py <dataset> <proportion>
python train_HCNN.py <proportion>
python train_HCNN-DOC.py <proportion>

4. Evaluate the results with different detection method (DOC, SofterMax, LOF, SMDN, ...)

python eval.py

or you can get a single result with

python eval_BiLSTM.py <dataset> <proportion>
python eval_BiLSTM-DOC.py <dataset> <proportion>
python eval_HCNN.py <proportion>
python eval_HCNN-DOC.py <proportion>

Confusion matrix for SMDN experiment results on three different datasets