50% train/dev accuracy for a binary classification task about esim HOT 3 CLOSED

Hi, what task did you train ESIM on ? The model is designed for natural language inference specifically, and it might not be appropriate for the task you're trying to solve, depending on what it is. Your issue might also be with the format of the data you fed to the model. It has to follow the same format as the SNLI and MNLI data sets.

from esim.

Hi Aurelien, many thanks for your feedback. The task I am doing is a natural language inference binary classification task. The input feature includes the context (facts + rules) and a question. The output is true/false. I can use RoBERTa, BERT to train/dev/test and get a high test accuracy above 90%. But it is 50% test accuracy by using ESIM. So it is quite weird for me. Here is the dataset format.

##Input feature (Context + Question)
##Context (Facts + Rules):
##Facts:
Red things are big. Anne is big. Erin is green. Anne is red. All red things are round. 
##Rules:
If something is green and cold then it is furry. Anne is green. Anne is cold. Erin is round. If something is rough then it is furry. If something is cold then it is furry. Anne is rough. Anne is furry. All furry things are round. Erin is furry. Erin is cold. If something is round then it is rough. All rough things are cold.

##Question:
'Anne is red.'

##Output (1/0) True: 1, False: 0
##Correct label: 
1 
##Predict label:
1

from esim.

Sorry for the (very) late reply. You'll need to modify the training and testing scripts quite significantly if you want to process data in a different format than the SNLI or MNLI data sets. In particular, you'll need to rewrite the preprocessing steps to format your data so it is readable by the model. You also need to change the data's labeldict to only contain two output labels instead of three, if your classification task is binary.

from esim.