Git Product home page Git Product logo

greedycws's Introduction

greedyCWS

Hi, this code is easy to use!

Please check the src/train.py for all hyper-parameter and IO settings.

You can modify the src/train.py to speficy your own model settings or datasets.

  • For training, use the command line python train.py. Training details will be printed on the screen. The learned parameters will be saved in in the same directory as train.py per epoch, which will be named as epoch1, epoch2, ....
  • For test, the same command line python train.py is used, but with a specified parameter file (e.g., epoch1), via the function argument load_params in train.py (Note load_params should be None when training). In addition, tell your test file by setting dev_file (Yes, when test, consider it as "test_file"). The segmented result will be saved in src/result.

The code is originally designed for reasearch purpose, but adaptable to industrial use.

Citation

This code implements an efficient and effective neural word segmenter proposed in the following paper.

Deng Cai, Hai Zhao, etc., Fast and Accurate Neural Word Segmentation for Chinese. ACL 2017.

If you find it useful, please cite the paper.

@InProceedings{cai-EtAl:2017:Short,
  author    = {Cai, Deng  and  Zhao, Hai  and  Zhang, Zhisong  and  Xin, Yuan  and  Wu, Yongjian  and  Huang, Feiyue},
  title     = {Fast and Accurate Neural Word Segmentation for Chinese},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  month     = {July},
  year      = {2017},
  address   = {Vancouver, Canada},
  publisher = {Association for Computational Linguistics},
  pages     = {608--615},
  url       = {http://aclweb.org/anthology/P17-2096}
}

Contact

Drop me (Deng Cai) an email at thisisjcykcd (AT) gmail.com if you have any question.

greedycws's People

Contributors

jcyk avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

greedycws's Issues

Dynet版本问题

请问使用的Dynet版本是多少?

Traceback (most recent call last):
File "train.py", line 28, in
word_proportion= 0.5
File "/home/lyt/workspace/CWS/greedyCWS-master/src/model.py", line 187, in dy_train_model
cws = CWS(Cemb,character_idx_map,options)
File "/home/lyt/workspace/CWS/greedyCWS-master/src/model.py", line 18, in init
self.trainer = dy.MomentumSGDTrainer(model,options['lr'],options['momentum'],options['edecay']) # we use Momentum SGD
File "_dynet.pyx", line 5878, in _dynet.MomentumSGDTrainer.cinit
TypeError: cinit() takes at most 3 positional arguments (4 given)

错误

image
请问这是什么错误?

实验结果的一些疑问

在重现PKU测试文本实验结果时,遇到下列问题:
论文源码(未修改)

cws.trainer.update_epoch(1.)
end_time = time.time()
print 'Trained %s epoch(s) (%d samples) took %.lfs per epoch'%(eidx+1,nsamples,(end_time-start_time)/(eidx+1))
test(cws,dev_file,'../result/dev_result%d'%(eidx+1))
os.system('python score.py %s %d %d'%(dev_file,eidx+1,eidx+1))
cws.save('epoch%d'%(eidx+1))
print 'Current model saved'
按照README.md 运行代码,测试PKU,F1值95.1

[@gpu01 src]$ ./score ../data/pku_train ../data/pku_test ../result/dev/dev_result28 > ../../result/score/greedy_pku_test_seg.utf8
[
@gpu01 src]$ tail -n 14 ../../result/score/greedy_pku_test_seg.utf8
=== SUMMARY:
=== TOTAL INSERTIONS: 1766
=== TOTAL DELETIONS: 1496
=== TOTAL SUBSTITUTIONS: 3477
=== TOTAL NCHANGE: 6739
=== TOTAL TRUE WORD COUNT: 104371
=== TOTAL TEST WORD COUNT: 104641
=== TOTAL TRUE WORDS RECALL: 0.952
=== TOTAL TEST WORDS PRECISION: 0.950
=== F MEASURE: 0.951
=== OOV Rate: 0.927
=== OOV Recall Rate: 0.950
=== IV Recall Rate: 0.988
../result/dev/dev_result28 1766 1496 3477 6739 104371 104641 0.952 0.950 0.951 0.927 0.950 0.988

1、但是,test(cws,dev_file,'../result/dev_result%d'%(eidx+1)),测试文件dev_file是../data/pku_test文件,这个pku_test是已经分词的,test函数输出分词结果保存为../result/dev_result%d'%(eidx+1),然后运行score评分脚本,=== F MEASURE: 0.951 是分词结果pku_test对比得出。

2、分词结果标准分割icwb2-data/gold/pku_test_gold.utf8 对比=== F MEASURE: 0.843

3、运行test模式,load最好的模型,测试文件为未分词的icwb2-data/testing/pku_test.utf8,得出的分词结果和标准分割icwb2-data/gold/pku_test_gold.utf8对比是F值是79。

这是我疑惑的地方,是不是1中得出的评分结果有失偏颇,因为一是使用已分词的文本pku_test做测试,二是使用这个已分词文本作为标准分割。

小白一枚,如有问题,多多包涵

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.