Git Product home page Git Product logo

pinyin2hanzi's Introduction

Pinyin2Hanzi

拼音转汉字,可以作为拼音输入法的转换引擎,兼容Python 2、Python 3。

安装

Python 2:

$ python setup.py install --user

Python 3:

$ python3 setup.py install --user

使用

下面的示例在Python 3中运行。

基于HMM的转换

原理是viterbi算法。

from Pinyin2Hanzi import DefaultHmmParams
from Pinyin2Hanzi import viterbi

hmmparams = DefaultHmmParams()

## 2个候选
result = viterbi(hmm_params=hmmparams, observations=('ni', 'zhi', 'bu', 'zhi', 'dao'), path_num = 2)
for item in result:
    print(item.score, item.path)
'''输出
1.3155294593897203e-08 ['你', '知', '不', '知', '道']
3.6677865125992192e-09 ['你', '只', '不', '知', '道']
'''

## 2个候选,使用对数打分
result = viterbi(hmm_params=hmmparams, observations=('ni', 'zhi', 'bu', 'zhi', 'dao'), path_num = 2, log = True)
for item in result:
    print(item.score, item.path)
'''输出
-18.14644152864202 ['你', '知', '不', '知', '道']
-19.423677486918002 ['你', '只', '不', '知', '道']
'''

## 2个候选,使用对数打分
result = viterbi(hmm_params=hmmparams, observations=('ni', 'zhii', 'bu', 'zhi', 'dao'), path_num = 2, log = True)
for item in result:
    print(item.score, item.path)
# 发生KeyError,`zhii`不规范

基于DAG的转换

原理是词库+动态规划。

from Pinyin2Hanzi import DefaultDagParams
from Pinyin2Hanzi import dag

dagparams = DefaultDagParams()

## 2个候选
result = dag(dagparams, ('ni', 'bu', 'zhi', 'dao', 'de', 'shi'), path_num=2)
for item in result:
    print(item.score, item.path)
''' 输出
0.08117536840088911 ['你不知道', '的是']
0.04149191639287887 ['你不知道', '的诗']
'''

## 2个候选,使用对数打分
result = dag(dagparams, ('ni', 'bu', 'zhi', 'dao', 'de', 'shi'), path_num=2, log=True)
for item in result:
    print(item.score, item.path)
''' 输出
-2.5111434226494866 ['你不知道', '的是']
-3.1822566564324477 ['你不知道', '的诗']
'''

## 1个候选
print( dag(dagparams, ['ti', 'chu', 'le', 'bu', 'cuo', 'de', 'jie', 'jve', 'fang', 'an'], path_num=1) )
'''输出
[< score=0.0017174549839096384, path=['提出了', '不错', '的', '解决方案'] >]
'''

## 2个候选,使用对数打分
result = dag(dagparams, ('ni', 'bu', 'zhi', 'dao', 'de', 'shii'), path_num=2, log=True)
print(result)
# 输出空列表,因为`shii`不存在

自定义params

实现AbstractHmmParams, AbstractDagParams这两个接口即可。具体可以参考源码。

关于拼音

给出的拼音必须是“规范”的。例如

  • 略 -> lve
  • 据 -> ju

列举所有“规范”的拼音:

from Pinyin2Hanzi import all_pinyin
for py in all_pinyin():
        print(py)

将拼音转换为“规范”的拼音:

from Pinyin2Hanzi import simplify_pinyin

print(simplify_pinyin('lue'))
# 输出:'lve'

print(simplify_pinyin('lüè'))
# 输出:'lve'

判断是否是“规范”的拼音:

from Pinyin2Hanzi import is_pinyin

print(is_pinyin('lue'))
# 输出:False

print(is_pinyin('lüè'))
# 输出:False

print(is_pinyin('lvee'))
# 输出:False

print(is_pinyin('lve'))
# 输出:True

训练

原始数据和训练代码在train目录下。数据来自jpinyinpinyin搜狗语料库-互联网词库等。处理数据时用到了汉字转拼音 工具ChineseTone

原理

如何实现拼音与汉字的互相转换

License

MIT

pinyin2hanzi's People

Contributors

letiantian avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pinyin2hanzi's Issues

请问HMM和DAG出的分数有什么不同?

由拼音生成字的过程中,使用了HMM模型,并对生成的字进行了打分。
然后又使用了DAG对字重新打分。请问这两个分数有什么不同呢?不是很理解这个重新打分的意义?训练的语言模型体现在哪一部分了呢?
谢谢!

BUG?

dag/train.sh最后直接cp不行的吧,还是要跑gen_final的吧

请问怎么把连续的拼音分开呢

我看这里面的例子是每个汉字的拼音都分开的,比如['ni', 'hao'].

如果输入是'nihao'的话,怎么知道这个长拼音是可以分成nihao的呢?

更改拼音的标签进行分类

感谢朋友您分享你的研究成果;
针对您提出的拼音合理问题,我想提出一个问题,比如说“学”在中文里面的拼音是“xue2”(其中2代表的是声调),如果按照您做的应该是“xve”,这样的标签无法明确显示标签的声调,能否做一个以字母为声调的拼音建模方式,实现拼音到汉字的转换!
我的QQ:76859420
欢迎朋友加我QQ,一起讨论一些具体问题!

用户词典的添加

你好,我在你的项目中的train下看到了 百年孤独.txt 简爱.txt等文件。麻烦想问一下,该项目是否支持添加用户自定义词典?添加之后是要重新train吗

‘你’在py2hz.json中即有'n'又有'ni'

在py2hz.json中,为什么'你'即出现在'n'这个k-v中,又出现在'ni'这个k-v中?‘男’这个字也出现在了'n'中。导致'n'这个音通过hmm出现了'你'和‘男’这两个字。

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.