pranav-ust / bert-keyphrase-extraction Goto Github PK

View Code? Open in Web Editor NEW

108.0 4.0 49.0 7.48 MB

Keyphrase Extraction based on Scientific Text, Semeval 2017, Task 10

Python 100.00%

bert keyphrase-extraction pytorch semeval-2017

bert-keyphrase-extraction's Introduction

I don't maintain this repo anymore. There are now way better repos for you to find out keywords like this one

Keyphrase Extraction using SciBERT (Semeval 2017, Task 10)

Deep Keyphrase extraction using SciBERT.

Usage

Clone this repository and install pytorch-pretrained-BERT
From scibert repo, untar the weights (rename their weight dump file to pytorch_model.bin) and vocab file into a new folder model.
Change the parameters accordingly in experiments/base_model/params.json. We recommend keeping batch size of 4 and sequence length of 512, with 6 epochs, if GPU's VRAM is around 11 GB.
For training, run the command python train.py --data_dir data/task1/ --bert_model_dir model/ --model_dir experiments/base_model
For eval, run the command, python evaluate.py --data_dir data/task1/ --bert_model_dir model/ --model_dir experiments/base_model --restore_file best

Results

Subtask 1: Keyphrase Boundary Identification

We used IO format here. Unlike original SciBERT repo, we only use a simple linear layer on top of token embeddings.

On test set, we got:

F1 score: 0.6259
Precision: 0.5986
Recall: 0.6558
Support: 921

Subtask 2: Keyphrase Classification

We used BIO format here. Overall F1 score was 0.4981 on test set.

	Precision	Recall	F1-score	Support
Process	0.4734	0.5207	0.4959	870
Material	0.4958	0.6617	0.5669	807
Task	0.2125	0.2537	0.2313	201
Avg	0.4551	0.5527	0.4981	1878

Future Work

Some tokens have more than one annotations. We did not consider multi-label classification.
We only considered a linear layer on top of BERT embeddings. We need to see whether SciBERT + BiLSTM + CRF makes a difference.

Credits

SciBERT: https://github.com/allenai/scibert
HuggingFace: https://github.com/huggingface/pytorch-pretrained-BERT
PyTorch NER: https://github.com/lemonhu/NER-BERT-pytorch
BERT: https://github.com/google-research/bert

bert-keyphrase-extraction's People

Contributors

Stargazers

Watchers

bert-keyphrase-extraction's Issues

Is it convenient to provide preprocessing code?

Is it convenient to provide preprocessing code? Just mark the location of the complete keyphrase?

Tags generation

Can you please explain how did you generate tags.txt?

what did you do for word out of vocab ?

it seems you have not done anything to handle out of vocab during training .this code have many KeyError during train model . what is your suggestion to handle this error?

Is there any papers published?

I would like to ask if this model has been published. I would like to read it in detail.Working on this task recently, I found that many open source projects are from 2017.
thanks~

I've tried replicating your results for task 1 and task 2, but keep getting drastically lower F1 scores. I am getting an F1 score of 44 for task 1, and an average of 36.79 for task 2. Any suggestions?

Also this is really nice work, thanks for posting this.

'NoneType' object has no attribute 'convert_tokens_to_ids'

While running train.py I encountered this error:
Model name 'model/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese). We assumed 'model/vocab.txt' was a path or url but couldn't find any file associated to this path or url.

Traceback (most recent call last): File "train.py", line 168, in <module> train_data = data_loader.load_data('train') File "/content/BERT-keyphrase-extraction/data_loader.py", line 83, in load_data self.load_sentences_tags(sentences_file, tags_path, data) File "/content/BERT-keyphrase-extraction/data_loader.py", line 51, in load_sentences_tags sentences.append(self.tokenizer.convert_tokens_to_ids(tokens)) AttributeError: 'NoneType' object has no attribute 'convert_tokens_to_ids'

I think it isn't registering the pytorch_model.bin file, which I directly downloaded as bert-base-uncased.tar.gz

Also, when I modify the command to go in task1/train, python train.py --data_dir data/task1/train/ --bert_model_dir model/ --model_dir experiments/base_model the error is:
Loading the datasets... Traceback (most recent call last): File "train.py", line 165, in <module> data_loader = DataLoader(args.data_dir, args.bert_model_dir, params, token_pad_idx=0) File "/content/BERT-keyphrase-extraction/data_loader.py", line 28, in __init__ self.tag_pad_idx = self.tag2idx['O'] KeyError: 'O'