BERTurk Training Dataset Preparation about turkish-bert HOT 6 OPEN

Hi @kaansonmezoz ,

thanks for your interest in our models 🤗

1. The complete training corpus was filtered and sentence segmented, basically with:

from nltk.tokenize import sent_tokenize

for sent in sent_tokenize(line, "turkish"):
  if len(sent.split()) > 5:
    print(sent)

So it is not only applied to the OSCAR subcorpus here.

2. I used sentences longer than 5 tokens (split on whitespaces), see above :)

3. Not only full stops are considered for sentence segmentation, NLTK has some more tokens to be considered.

4. I just looked it up in my "data lake", the trwiki-latest-pages-articles.xml.bz2 dump has a 480M 2. Feb 2020 timestamp.

5. I could found the following OPUS-related files:

bible-uedin.txt GNOME.txt JW300.txt  OpenSubtitles.txt  opus.all QED.txt  SETIMES.txt  Tanzil.txt  Tatoeba.txt  TED2013.txt  Wikipedia.txt

With a timestamp of 3. Feb 2020.

6. For pre-processing (of pre-training data) the official BERT implementation was used, so basically all pre-processing steps can be found here: https://github.com/google-research/bert/blob/master/tokenization.py#L161-L182, so first a basic tokenization step is done, followed by the wordpiece stuff. I did not add extra steps.

Please just give me your mail addresse and I can immediately send you the link to the corpus used for pre-training 🤗

from turkish-bert.

Hey @hazalturkmen , no problem, just give me an email addresse where I can contact you 🤗

from turkish-bert.

Mails are out 🤗

from turkish-bert.

Hi @stefan-it,
Can i get the links to the corpus used for pre-training?
thanks,

from turkish-bert.

Thanks, @stefan-it ,
Here is my email address:

[email protected]

from turkish-bert.

@stefan-it Thank you for detailed explanation. My email is [email protected] 🙂

You are a life saver ! ❤️

from turkish-bert.