checkstep / mole-stance Goto Github PK

I am executing the following command with the necessary parameters:
python3.8 src/stancedetection/models/trainer.py
For now, I am just using 3 datasets with a very small number of synthetic examples, just to make sure that the code is working.
However, I am getting the following error:

File "src/stancedetection/models/trainer.py", line 903, in <module>
    main()
  File "src/stancedetection/models/trainer.py", line 822, in main
    model = load_model_from_pretrained(
  File "src/stancedetection/models/trainer.py", line 291, in load_model_from_pretrained
    model = model_cls.from_pretrained(model_name_or_path, **from_pretrained_kwargs)
  File "venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 947, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/scratch/rrs99/mole-stance/src/stancedetection/models/nn.py", line 147, in __init__
    task2labels=config.task2labels,
AttributeError: 'RobertaConfig' object has no attribute 'task2labels'

Hi,

Any update on when the code will be released?

Hi,
Thanks for sharing the code. I have done a small-scale experiment to make sure that the code is working on my end. I followed the instruction mentioned under "Label Embedding" in the readme.txt. I used the following setting:

DATASETS=(arc fnc ibmcs)
TARGET=arc

In the generated test_predictions.csv, I see lines like the following:

14,"[0.4692992568016052, 0.11499807983636856, 0.10301849246025085, 0.07154946774244308, 0.2411346733570099]",0,0,fnc1_agree,14,arc,arc__disagree

Since I set the target dataset to be the arc dataset, I expected the predicted labels to be also from the arc dataset (here, it's fnc1_agree). Can you kindly explain this? Also, I checked the confusion matrix, it seems there are rows corresponding to labels from the fnc and ibmcs datasets, but not from the arc dataset. Can you also kindly explain this?

In the generated test_metric.json, I see a very low accuracy score. I expected that, whenever the "agree" label is predicted (same goes for other labels too), no matter whether it is fnc1_agree or arc_agree, it will be treated as a correct prediction because they both belong to POSITIVE _LABELS. However, I don't think that's how the accuracy score is being computed. Can you kindly clarify this part too?

Can this framework be used to detect stance in unseen and unlabelled custom datasets given a hypothesis and a premise?

The "Cross-Domain Label-Adaptive Stance Detection" is a great paper.
When do you plan to make the code and data available?

Hi, can you please release the datasets in json format or provide code for the same?

Hi, is there any plan to make the pretrained model weights public?