abcNLP

AB-Natural Chinese Languange Processing, The movement of C makes it ab-natural.

Thanks to censorship of internet, people oftern meet trouble to express themselves freely. Usually the SYSTEM will filter contents by (keywords) pattern matching. If we can make the language hard understood to SYSTEM, but not to human, we make the expression “Immune 2 Censorship” (at least, to some extent)!

• develop: CJKLIB, sqlite3
• use: sqlite3

• for example :

    abc = abcChineseChar()    
    for word in abc.getHxWords(u'茉莉花',  maxnum=5):
        print word    
      

• output:

  苿峲芘
  苿峲芼
  苿萂芘
  炗峲芘
  苿峲芭
      

  (~ use tempory db, need to refine ~)

• for example :
• output :
  • ouput of 锦

    abc = abcChineseChar()    
    for char in abc.getHxCharacters(u'锦'):
        print char    		
      

    jǐn
    钅帛
    仅
    尽
    錦
    馑
      

  • ouput of 陈水扁:

    abc = abcChineseChar()    
    for word in abc.getHxWords(u'陈水扁',  maxnum=15):
        print word  
      

    <chén><shuǐ><biān>
    <阝东><shuǐ><biān>
    <chén><shuǐ>边
    <阝东><shuǐ>边
     尘<shuǐ><biān>
     臣<shuǐ><biān>
    <chén><shuǐ>便
    <chén><shuǐ>碥
     <chén><shuǐ>艑
    <chén>氵<biān>
     <阝东><shuǐ>便
     <阝东><shuǐ>碥
     <阝东><shuǐ>艑
     <阝东>氵<biān>
     尘<shuǐ>边
      

(~ use db version 1 ~)

Metaphor related methods. This method is not used in this project.

The variant of a character is alao a valid character. It can be seen as another shape (form) of original one. But some of forms are rare used nowdays.

㒲 --> 財 
㒲 --> 才 
㒲 --> 财 
㒲 --> 纔 
㒷 --> 興 
㒷 --> 兴 
㓁 --> 网 
㓁 --> 網 
㓁 --> 罔

In the following list, value in brackets is score for the replacement. More small, more better.

㐲 --> dài (0) * Pinyin is a special type
㐲 --> 大 (3)
㐲 --> 代 (5)
㐲 --> 黱 (22)

㐳 --> wù (0)
㐳 --> 兀 (3)
㐳 --> 乌 (4)
㐳 --> 鼿 (17)

After split character, each of its part can be further replaced with its similar character, which has ending mark of “1” in the following list.

川 --> <丿丨丨> 0
巧 --> <工丂> 0
垛 --> <土朶> 1
垜 --> <土朵> 1
⽻ --> <習習> 1
⾽ --> <镸三> 1
䜌 --> <⺯讠⺯> 1
丬 --> <氷丨> 1
乢 --> <山隠> 1
乣 --> <庅乚> 1
乨 --> <枱乚> 1
乩 --> <佔乚> 1
⽻ --> <习习> 0
⾽ --> <镸彡> 0
丬 --> <冫丨> 0
乢 --> <山乚> 0
乣 --> <幺乚> 0
乨 --> <台乚> 0
乩 --> <占乚> 0
亿 --> <亻乙> 0
什 --> <亻十> 0
仁 --> <亻二> 0
亿 --> <人乙> 1
什 --> <人十> 1
仁 --> <人二> 1
仂 --> <人力> 1
仃 --> <人丁> 1
仅 --> <人又> 1
仆 --> <人卜> 1
仇 --> <人九> 1

The score smaller is the better.

⺡ --> ⺍ (1)
⺡ --> 乊 (3)
⺡ --> 丬 (3)
⺡ --> 习 (4)
⺡ --> 乥 (8)
⺆ --> ⼌ (2)
⺆ --> ⼓ (3)
⺆ --> ⼏ (4)
⺆ --> 九 (4)
丨 --> ⼁ (0)
丨 --> ⼃ (2)
丨 --> 丿 (2)
丨 --> ⼅ (2)

This is the most important part of the project. After the module is refined, more examples will be added.

1. Sun Mar 25 17:54:46 CST 2012 ,initial version
2. Thu Mar 29 01:30:37 CST 2012, re-establish Repository
3. Sun Apr 1 12:04:11 CST 2012, two db with different size

by wuliang [www.wuliang.cn at gmail.com]