This github contains codes which are supplement to my Medium Post related to tokenizing Chinese phrases.

This Github contains 3 folders related to 3 major sections in the post:

1. Poem
2. Written Chinese
3. Cantonese

The section of "Tokenizing individual Chinese characters of an ancient Chinese Poem" investigates how to tokenize Chinese Phrases by spliting individual Chinese words with Chinese poem written in 1000 years ago, or Tang Dynasty:

1. Unigram (Tokenize by Chinese words)
2. Tokenize using jieba (Did not mention in the Medium Post)
   
   You may find the related codes by going to the folder by clicking here

The section of "Tokenizing individual Chinese characters of a Hong Kong news article" and "Using the package jieba to tokenize" investigate how and why to tokenize Chinese Phrases with jieba.

You may find the related codes by going to the folder by clicking here

The section of "How about Cantonese?" investigates the performance of jieba on written Cantonese.

You may find the related codes by going to the folder by clicking here

• sino_punc.txt This file contains all punctuation of Chinese, for regex.