Find a robust method to get articles paragraphs about cdqa HOT 2 CLOSED

Several options are possible:

• Get articles raw text and split on /n like in current implementaiton of converter.py
• When scrapping articles get all <p> tags

Danqi Chen's thesis also shows they filtered out some paragraphs of their custom QA datasets (shorter than 25 or longer than 1500 characters). It would be good to implement rules like this but maybe some more info on the distribution of the SQuAD passage length would help.

from cdqa.

Some python ideas for processing.py

def filter_paragraphs(paragraphs):
    # filter out paragraphs shorter than 10 words and longer than 15 words
    paragraph_long = [paragraph for paragraph in paragraphs if len(paragraph.split()) >= 10 and len(paragraph.split()) <= 15]
    return paragraph_long

df['paragraphs'] = df['paragraphs'].apply(filter_paragraphs)

from cdqa.