Data files used by botok Filenames don't have any meaning in botok, so they can be used by the user to organize wordlists.

A botok profile can contain any of the following folders:

Wordlists are used to populate the trie that botok uses for the tokenization process.

All the files within the following wordlist folders must be .tsv files that follow the following format:

<form>\t<pos>\t<lemma>\t<sense>\t<freq>

Empty lines and lines commented with # are allowed.

• words_bo/: Tibetan words
• words_skrt/: Sanskrit words
• words_non_inflected/: place here words that can't be the host of affixed particles. Typical cases are particle such as གྱི་ that we need within the trie, yet that shouldn't be inflected like regular words. This particle can't host the affixed locative particle ར་.

Include here words that should not appear in the trie even though you don't (yet) want to remove them from the wordlists.

Contains .txt files containing one word per line.

Contains .tsv formatted as follows:

• each rule should be as follows: <matchcql>\t<index>\t<operation>\t<replacecql>
• comments with # and empty lines are allowed
• CQL rules: <text> can be used without specifying that there is text_cleaned=
• Index format: either <matching_index> or <matching_index>-<splitting-index>
• Adjustment format:
  • + for merge
  • : for split (default: syllable mode)
  • :: for split in character mode
  • = for replace
• Constraint: <matching_index>-<splitting-index> is only allowed if adjustment is : or ::

• adjustment/rdr_basis.tsv : basic particle agreement rules implemented as a placeholder.
• words_bo/tsikchen.tsv

Monlam Grand Dictionary with added Frequency. Done by Thubten Rinzin

XXX

XXX

These files come from tibetan-spellchecker and indicate exceptions to Classical Tibetan norms.

This is a list of all particles, compiled by hand, with the PART POS tag.

This file has been extracted from:

Meelen, Marieke, Hill, Nathan, & Handy, Christopher. (2017). The Annotated Corpus of Classical Tibetan (ACTib), Part II - POS-tagged version, based on the BDRC digitised text collection, tagged with the Memory-Based Tagger from TiMBL [Data set]. Zenodo. http://doi.org/10.5281/zenodo.822537

It is available under the CC-BY 4.0 license.

This file has been extracted from a digitized version of:

Yisun, Zhang. 1985. བོད་རྒྱ་ཚིག་མཛོད་ཆེན་མོ།. Beijing: མི་རིགས་དཔེ་སྐྲུན་ཁང་།

Although the book is under copyright, we consider that the bare list of words we provide is not.

XXX

XXX

Unless indicated otherwise, all files are in the public domain.