cdqa-suite / cdqa Goto Github PK

⛔ [NOT MAINTAINED] An End-To-End Closed Domain Question Answering System.

Home Page: https://cdqa-suite.github.io/cdQA-website/

License: Apache License 2.0

Python 100.00%

reading-comprehension question-answering deep-learning natural-language-processing information-retrieval bert artificial-intelligence nlp pytorch transformers

cdqa's Issues

Add document retriever script

Adapt retriever.py to BNP Paribas Newsroom dataset v.1.0

See issue #31

Article retrieval underperforms paragraph retrieval by a large margin.

Split run_squad.py in processing/train/predict

The idea is to use /reader/run_squad.py as a script to import without main() and to break the main() into subparts that would be added to processing/train/predict python scripts in /pipeline.

Cannot load bert sklearn .joblib model

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location='cpu' to map your storages to the CPU.

Upload json version of BNP Paribas Newsroom dataset v.1.0 in release

Fine-tune BERT on SQuAD train then SQuAD dev

Use output weights of first BERT fine-tuned on SQuAD train as input starter weights for new BERT fine-tuned on SQuAD dev.

https://github.com/huggingface/pytorch-pretrained-BERT#squad

Update docs with apex instructions

Upload weights and metrics and update download.py script

Build a document retriever to find best article candidates

Add github badges

https://shields.io/

pypi
codecov
notebook

NameError: name 'device' is not defined in predict() method

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-14-65cedbb88ef8> in <module>()
      2 test_examples, test_features = test_processor.fit_transform(X=squad_examples)
      3 # model = load('model.joblib')
----> 4 predictions = model.predict(X=(test_examples, test_features))

~/cdQA/cdqa/reader/bertqa_sklearn.py in predict(self, X)
   1037             if len(all_results) % 1000 == 0:
   1038                 logger.info("Processing example: %d" % (len(all_results)))
-> 1039             input_ids = input_ids.to(device)
   1040             input_mask = input_mask.to(device)
   1041             segment_ids = segment_ids.to(device)

NameError: name 'device' is not defined

Add scrapper folder under /cdqa

Set correct boolean values for parameters in sklearn wrapper (--lower-case, etc)

Try BERT prediction on sample file

Ideas: #7 (comment)

Wrong url for squad evaluate-v1.1.py on dowload.py script

The url 'https://github.com/allenai/bi-att-flow/blob/master/squad/evaluate-v1.1.py' used to download the evaluate-v1.1.py script actually downloads a html code

The correct url should be: 'https://raw.githubusercontent.com/allenai/bi-att-flow/master/squad/evaluate-v1.1.py'

read_squad_examples() does not work with our custom input object

https://github.com/fmikaelian/cdQA/blob/6e26415fe604b70bfb9ba062b555680567a8ea58/cdqa/reader/bertqa_sklearn.py#L125

Run evaluation script with bert sklearn wrapper to check metrics are consistent with run_squad script

@andrelmfarias How did you generate metrics.json previously?

Implement download.py script (SQuAD fetch)

Here: https://github.com/fmikaelian/reading-comprehension/blob/develop/reading-comprehension/pipeline/download.py

predict() method should also give back index of document + paragraph

We could add a document index to squad_examples then test_examples?

squad_examples = generate_squad_examples(question=question,
                                         article_indices=article_indices,
                                         metadata=df)

test_processor = BertProcessor(bert_model='bert-base-uncased', do_lower_case=True, is_training=False)
test_examples, test_features = test_processor.fit_transform(X=squad_examples)

model = load(os.path.join('models/bert_qa_squad_v1.1_sklearn', 'bert_qa_squad_v1.1_sklearn.joblib'))
final_prediction, all_predictions, all_nbest_json, scores_diff_json = model.predict(X=(test_examples, test_features))

Wrong dataset filename in examples

Should be bnpp_newsroom-v1.0.csv

Update download.py for new models

Add DrQA's paper to references

https://arxiv.org/abs/1704.00051

Add license

Add comments + docstrings + changelog

code --diff cdqa/reader/run_squad.py cdqa/reader/bertqa_sklearn.py

Allows to compare sklearn wrapper vs. original script in vscode.

Ability to tune parameters at prediction time

Following huggingface/transformers#126:

Parameters predict_fp16, max_seq_length and predict_batch_size should be tunable at predict time:

For train:

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

For predict with fp16:

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_predict \
  --predict_fp16 \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --predict_batch_size 128 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

Be compliant with the Github open source guide

Interesting links:

FileNotFoundError at prediction time

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-20-bff63677b429> in <module>()
----> 1 final_prediction, all_predictions, all_nbest_json, scores_diff_json = model.predict(X=(test_examples, test_features))

~/cdQA/cdqa/reader/bertqa_sklearn.py in predict(self, X)
   1195             self.verbose_logging,
   1196             self.version_2_with_negative,
-> 1197             self.null_score_diff_threshold)
   1198 
   1199         return final_prediction, all_predictions, all_nbest_json, scores_diff_json

~/cdQA/cdqa/reader/bertqa_sklearn.py in write_predictions(all_examples, all_features, all_results, n_best_size, max_answer_length, do_lower_case, output_prediction_file, output_nbest_file, output_null_log_odds_file, verbose_logging, version_2_with_negative, null_score_diff_threshold)
    636     final_prediction = list(final_predictions_sorted.items())[0][1]['text']
    637 
--> 638     with open(output_prediction_file, "w") as writer:
    639         writer.write(json.dumps(all_predictions, indent=4) + "\n")
    640 

FileNotFoundError: [Errno 2] No such file or directory: 'logs/bert_qa_squad_v1.1_sklearn/predictions.json'

Let's brainstorm!

(OrderedDict([('2398202a-41b4-11e9-beaa-796013f1ec43', 'BNP Paribas'),
              ('239828b8-41b4-11e9-beaa-796013f1ec43',
               'Chronicle of a revolution'),
              ('2398294e-41b4-11e9-beaa-796013f1ec43',
               'machine learning, deep learning, language processing, etc.'),
              ('23983056-41b4-11e9-beaa-796013f1ec43', 'Edouard d’Archimbaud'),
              ('2398309c-41b4-11e9-beaa-796013f1ec43', 'AI'),
              ('239830e2-41b4-11e9-beaa-796013f1ec43', 'BNP Paribas'),
              ('23983128-41b4-11e9-beaa-796013f1ec43', 'Marvin Lee Minsky'),
              ('23983164-41b4-11e9-beaa-796013f1ec43',
               'Artificial Intelligence is in fact likely to surpass humans in performing tasks that require reasoning and learning.'),
              ('239831a0-41b4-11e9-beaa-796013f1ec43', 'Watson'),
              ('239831e6-41b4-11e9-beaa-796013f1ec43', 'Google'),
              ('2398322c-41b4-11e9-beaa-796013f1ec43', 'Accenture'),
              ('23983268-41b4-11e9-beaa-796013f1ec43', 'AI'),
              ('239832a4-41b4-11e9-beaa-796013f1ec43', 'Partnership on AI'),
              ('239832e0-41b4-11e9-beaa-796013f1ec43', 'BNP Paribas'),
              ('23983326-41b4-11e9-beaa-796013f1ec43', 'Edouard d’Archimbaud'),
              ('23983362-41b4-11e9-beaa-796013f1ec43', 'data scientists'),
              ('2398339e-41b4-11e9-beaa-796013f1ec43', 'Edouard d’Archimbaud'),
              ('239833e4-41b4-11e9-beaa-796013f1ec43',
               'AI system’s ability to learn “by example” or “by experience”.'),
              ('23983420-41b4-11e9-beaa-796013f1ec43',
               'Deep learning is a learning technology that uses artificial neural networks, which approximate human learning to process “raw data”.'),
              ('2398345c-41b4-11e9-beaa-796013f1ec43', 'Alan Turing'),
              ('23983498-41b4-11e9-beaa-796013f1ec43', 'TEDxParis'),
              ('239834d4-41b4-11e9-beaa-796013f1ec43', 'BNP Paribas'),
              ('23983510-41b4-11e9-beaa-796013f1ec43', 'BNP Paribas'),
              ('23983a60-41b4-11e9-beaa-796013f1ec43', 'change management'),
              ('23983ad8-41b4-11e9-beaa-796013f1ec43', 'BNP Paribas'),
              ('23983b1e-41b4-11e9-beaa-796013f1ec43', 'Julien Dinh'),
              ('23983f92-41b4-11e9-beaa-796013f1ec43', 'Julien Dinh')]),
 OrderedDict(),
 OrderedDict())

The ground truth is Marvin Lee Minsky, available in context 23983128-41b4-11e9-beaa-796013f1ec43:

{'context': 'One of the creators of Artificial Intelligence, Marvin Lee Minsky, notably defines it as “the construction of computer programs that engage in tasks that are, for now, more satisfactorily accomplished by humans because they require high-level mental processes”. ',
    'qas': [{'answers': [],
      'question': 'Who is the creator of Artificial Intelligence?',
      'id': '23983128-41b4-11e9-beaa-796013f1ec43'}]},

How to get the best answer from predictions (see #36) ?
What is nbest_predictions.json (empty in my case) ?

Originally posted by @fmikaelian in #33 (comment)

Add fetch of BNP Paribas Newsroom dataset v.1.0 to download.py

Disable logger info for BertProcessor()

03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   *** Example ***
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   unique_id: 1000000000
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   example_index: 0
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   doc_span_index: 0
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   tokens: [CLS] [UNK] is the creator of [UNK] [UNK] ? [SEP] [UNK] [UNK] launches the prototype [UNK] , first online community for corporate clients [SEP]
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   token_to_orig_map: 10:0 11:1 12:2 13:3 14:4 15:5 16:5 17:6 18:7 19:8 20:9 21:10 22:11
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   token_is_max_context: 10:True 11:True 12:True 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   input_ids: 101 100 2003 1996 8543 1997 100 100 1029 102 100 100 18989 1996 8773 100 1010 2034 3784 2451 2005 5971 7846 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   *** Example ***
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   unique_id: 1000000001
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   example_index: 1
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   doc_span_index: 0
03/11/2019 09:42:42 - INFO - cdqa.reader.bertqa_sklearn -   tokens: [CLS] [UNK] is the creator of [UNK] [UNK] ? [SEP] [UNK] [UNK] has progressed at lightning speed in recent years . [UNK] are now able to beat humans in [UNK] matches , understand natural language , reason and learn . [UNK] a result , software and robots have something to offer in every field to make business more productive , profitable and innovative . [UNK] of a revolution fore ##to ##ld . [SEP]

How to allow comparision of BERT answer span predictions for final ranking?

to allow comparison and aggregation of results from different segments, we remove the final softmaxlayer over different answer spans.

See Issue #31

Explore BNP Paribas Newsroom dataset v.1.1

Generate some KPIs on the dataset we collected. Ideas of KPIs:

Number of paragraphs per article
Length of paragraphs
Paragraphs patterns

Before and after filtering.

cdqa-suite / cdqa Goto Github PK

cdqa's Issues

Recommend Projects

Recommend Topics

Recommend Org