Git Product home page Git Product logo

lionsoul2014 / jcseg Goto Github PK

View Code? Open in Web Editor NEW
907.0 92.0 212.0 21.59 MB

Jcseg is a light weight NLP framework developed with Java. Provide CJK and English segmentation based on MMSEG algorithm, With also keywords extraction, key sentence extraction, summary extraction implemented based on TEXTRANK algorithm. Jcseg had a build-in http server and search modules for lucene,solr,elasticsearch,opensearch

Home Page: http://gitee.com/lionsoul/jcseg

License: Apache License 2.0

Java 99.07% JavaScript 0.01% CSS 0.09% HTML 0.53% Shell 0.31%
java jcseg mmseg chinese-word-segmentation natural-language-processing pos-tagging nlp nlp-keywords-extraction lucene-analyzer lucene-tokenizer

jcseg's Issues

BUILD FAILED 2.4.0

ant all
Buildfile: build.xml

init:
[echo] ------------------------------------------------------------
[echo] ----------BUILDING JCSEG PACKAGE----------------------------
[echo]
[mkdir] Created dir: /www/git/jcseg-2.4.0-release/src
[mkdir] Created dir: /www/git/jcseg-2.4.0-release/classes
[mkdir] Created dir: /www/git/jcseg-2.4.0-release/doc
[echo] Coping sources of jcseg-core/src/main/java to /www/git/jcseg-2.4.0-r elease/src
[copy] Copying 50 files to /www/git/jcseg-2.4.0-release/src
[echo] Coping lexicons to /www/git/jcseg-2.4.0-release/classes
[copy] Copying 29 files to /www/git/jcseg-2.4.0-release/classes/lexicon
[echo] Coping sources of jcseg-analyzer to /www/git/jcseg-2.4.0-release/src
[copy] Copying 4 files to /www/git/jcseg-2.4.0-release/src
[echo] Coping sources of jcseg-elasticsearch to /www/git/jcseg-2.4.0-releas e/src
[copy] Copying 10 files to /www/git/jcseg-2.4.0-release/src
[echo] Coping sources of jcseg-server to /www/git/jcseg-2.4.0-release/src
[copy] Copying 60 files to /www/git/jcseg-2.4.0-release/src

BUILD FAILED
/www/git/jcseg-2.4.0-release/build.xml:60: Warning: Could not find file /www/git /jcseg-2.4.0-release/jcseg.properties to copy.

Total time: 0 seconds

同义词或是中文数字转阿拉伯数字的问题

您好,您的 jcseg 惠我良多,致上敝人万分感谢。
我在使用jcseg,有两个地方一直觉得有问题,就是当有同义词或是中文数字转阿拉伯数字的时候,所衍生出来的同义词或阿拉伯数字有以下问题:
1.其 endOffset 应该和原词的 endOffset 一样,而不应该是以衍生词的长度去定 endOffset。
2.衍生出来的词的 positionIncrement 应该是0,而不应该是1,否则在phrase search的时候会因为词的位置被衍生词隔断而查不到。
因为这两个值若与原词不同,会造成 phrase search 以及 highlight 的定位错误,请您协助确认,感谢您。

自定义词典构造问题

/**
     * create a singleton ADictionary object according to the JcsegTaskConfig
     * 
     * @param   config
     * @param   loadDic
     * @param   ADictionary
    */
    public static ADictionary createSingletonDictionary(JcsegTaskConfig config, boolean loadDic)
    {
        synchronized (LOCK) {
            if ( singletonDic == null ) {
                singletonDic = createDefaultDictionary(config);
            }
        }
        
        return singletonDic;
    }

DictionaryFactory 这个代码还是执行了构造默认词典的逻辑

关于ILexicon.CJK_WORD类型的疑问

hi,咨询个关于使用自定义词典的问题。
场景:简单的电商商品搜索场景,自定义一个品牌词典(品牌可能包含中英混合情况):brand.lex。词典类型采用了CJK_WORD,词典具体为:

CJK_WORD
骆驼
花花公子

在验证时候,发现在ADictionary类中的652行输出了格式信息不准确,具体代码是会通过特定分隔符把词典每行进行分割,分割的数组长度小于4就会出现此问题:

wd = line.split("\\s*/\\s*");
                if ( wd.length < 4 ) {    //format check
                    System.out.println("Word: \"" + wd[0] + "\" format error. -ignored");
                    continue;
                }

仔细看了下这个格式需要变为以下才可以通过:

CJK_WORD
骆驼///null
花花公子///null

可以请问下这么设计是为了解决什么问题吗?或者说,我的场景,适合用哪种词典类型?

引号导致json反解析错误

如果存在引号",反斜杠\ ,会出现 "word":""" , "word":""这种词,然后json反解析出错,
在Word类toString方法
sb.append("\"word\":\"").append(value).append('"');
改为

        String tmpV = value;
        tmpV = tmpV.replaceAll("\\\\", "\\\\\\\\");
        tmpV = tmpV.replaceAll("\"", "\\\\\"");
        sb.append("\"word\":\"").append(tmpV).append('"');

解决
其它特殊字符类似

jcseg-server内存占用

不想让服务占太多内存……jcseg-server至少需要多少内存才能正常工作

如果减少设置中的内存用量会有什么影响

使用jcseg-core-2.1.0.jar 出错

1 java.lang.NoClassDefFoundError: Failed resolution of: Lorg/lionsoul/jcseg/tokenizer/core/JcsegTaskConfig;
难道是因为jdk版本问题?

summary不是重点内容

summary不是重点内容,都是doc里面的前面若干个字截取,没啥用。缺少语义分析,文章的关键句子提取才比较准确。

ant 编译错误

build:
[echo] -------------Compiling Application--------------
[echo] D:\fenci\jcseg-master/vendors/lib
[javac] D:\fenci\jcseg-master\build.xml:72: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 109 source files to D:\fenci\jcseg-master\classes
[javac] 警告: [options] 未与 -source 1.7 一起设置引导类路径
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegAnalyzer5X.java:5: 错误: 程序包org.apache.lucene.analysis不存在
[javac] import org.apache.lucene.analysis.Analyzer;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegAnalyzer5X.java:6: 错误: 程序包org.apache.lucene.analysis不存在
[javac] import org.apache.lucene.analysis.Tokenizer;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegAnalyzer5X.java:18: 错误: 找不到符号
[javac] public class JcsegAnalyzer5X extends Analyzer
[javac] ^
[javac] 符号: 类 Analyzer
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegAnalyzer5X.java:94: 错误: 找不到符号
[javac] protected TokenStreamComponents createComponents(String fieldName)
[javac] ^
[javac] 符号: 类 TokenStreamComponents
[javac] 位置: 类 JcsegAnalyzer5X
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegFilter.java:5: 错误: 程序包org.apache.lucene.analysis不存在
[javac] import org.apache.lucene.analysis.TokenFilter;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegFilter.java:6: 错误: 程序包org.apache.lucene.analysis不存在
[javac] import org.apache.lucene.analysis.TokenStream;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegFilter.java:15: 错误: 找不到符号
[javac] public class JcsegFilter extends TokenFilter
[javac] ^
[javac] 符号: 类 TokenFilter
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegFilter.java:20: 错误: 找不到符号
[javac] protected JcsegFilter(TokenStream input)
[javac] ^
[javac] 符号: 类 TokenStream
[javac] 位置: 类 JcsegFilter
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:5: 错误: 程序包org.apache.lucene.analysis不存在
[javac] import org.apache.lucene.analysis.Tokenizer;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:6: 错误: 程序包org.apache.lucene.analysis.tokenattributes不存在
[javac] import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:7: 错误: 程序包org.apache.lucene.analysis.tokenattributes不存在
[javac] import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:8: 错误: 程序包org.apache.lucene.analysis.tokenattributes不存在
[javac] import org.apache.lucene.analysis.tokenattributes.TypeAttribute;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:41: 错误: 找不到符号
[javac] public class JcsegTokenizer extends Tokenizer
[javac] ^
[javac] 符号: 类 Tokenizer
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:46: 错误: 找不到符号
[javac] private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
[javac] ^
[javac] 符号: 类 CharTermAttribute
[javac] 位置: 类 JcsegTokenizer
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:47: 错误: 找不到符号
[javac] private final OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);
[javac] ^
[javac] 符号: 类 OffsetAttribute
[javac] 位置: 类 JcsegTokenizer
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizer.java:48: 错误: 找不到符号
[javac] private final TypeAttribute typeAtt = addAttribute(TypeAttribute.class);
[javac] ^
[javac] 符号: 类 TypeAttribute
[javac] 位置: 类 JcsegTokenizer
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizerFactory.java:6: 错误: 程序包org.apache.lucene.analysis不存在
[javac] import org.apache.lucene.analysis.Tokenizer;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizerFactory.java:7: 错误: 程序包org.apache.lucene.analysis.util不存在
[javac] import org.apache.lucene.analysis.util.TokenizerFactory;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizerFactory.java:8: 错误: 程序包org.apache.lucene.util不存在
[javac] import org.apache.lucene.util.AttributeFactory;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizerFactory.java:19: 错误: 找不到符号
[javac] public class JcsegTokenizerFactory extends TokenizerFactory
[javac] ^
[javac] 符号: 类 TokenizerFactory
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizerFactory.java:78: 错误: 找不到符号
[javac] public Tokenizer create( AttributeFactory factory )
[javac] ^
[javac] 符号: 类 AttributeFactory
[javac] 位置: 类 JcsegTokenizerFactory
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\analyzer\v5x\JcsegTokenizerFactory.java:78: 错误: 找不到符号
[javac] public Tokenizer create( AttributeFactory factory )
[javac] ^
[javac] 符号: 类 Tokenizer
[javac] 位置: 类 JcsegTokenizerFactory
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalysisBinderProcessor.java:3: 错误: 程序包org.elasticsearch.index.analysis不存在
[javac] import org.elasticsearch.index.analysis.AnalysisModule;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalysisBinderProcessor.java:10: 错误: 程序包AnalysisModule不存在
[javac] public class JcsegAnalysisBinderProcessor extends AnalysisModule.AnalysisBinderProcessor
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalysisBinderProcessor.java:13: 错误: 找不到符号
[javac] @OverRide public void processAnalyzers(AnalyzersBindings analyzersBindings)
[javac] ^
[javac] 符号: 类 AnalyzersBindings
[javac] 位置: 类 JcsegAnalysisBinderProcessor
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalysisBinderProcessor.java:19: 错误: 找不到符号
[javac] @OverRide public void processTokenizers(TokenizersBindings tokenizersBindings)
[javac] ^
[javac] 符号: 类 TokenizersBindings
[javac] 位置: 类 JcsegAnalysisBinderProcessor
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalysisBinderProcessor.java:25: 错误: 找不到符号
[javac] @OverRide public void processTokenFilters(TokenFiltersBindings tokenFiltersBindings)
[javac] ^
[javac] 符号: 类 TokenFiltersBindings
[javac] 位置: 类 JcsegAnalysisBinderProcessor
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:5: 错误: 程序包org.elasticsearch.common.inject不存在
[javac] import org.elasticsearch.common.inject.Inject;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:6: 错误: 程序包org.elasticsearch.common.inject.assistedinject不存在
[javac] import org.elasticsearch.common.inject.assistedinject.Assisted;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:7: 错误: 程序包org.elasticsearch.common.settings不存在
[javac] import org.elasticsearch.common.settings.Settings;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:8: 错误: 程序包org.elasticsearch.env不存在
[javac] import org.elasticsearch.env.Environment;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:9: 错误: 程序包org.elasticsearch.index不存在
[javac] import org.elasticsearch.index.Index;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:10: 错误: 程序包org.elasticsearch.index.analysis不存在
[javac] import org.elasticsearch.index.analysis.AbstractIndexAnalyzerProvider;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:11: 错误: 程序包org.elasticsearch.index.settings不存在
[javac] import org.elasticsearch.index.settings.IndexSettingsService;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:21: 错误: 找不到符号
[javac] public class JcsegAnalyzerProvider extends AbstractIndexAnalyzerProvider
[javac] ^
[javac] 符号: 类 AbstractIndexAnalyzerProvider
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:27: 错误: 找不到符号
[javac] public JcsegAnalyzerProvider(Index index, IndexSettingsService indexSettingsService,
[javac] ^
[javac] 符号: 类 Index
[javac] 位置: 类 JcsegAnalyzerProvider
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:27: 错误: 找不到符号
[javac] public JcsegAnalyzerProvider(Index index, IndexSettingsService indexSettingsService,
[javac] ^
[javac] 符号: 类 IndexSettingsService
[javac] 位置: 类 JcsegAnalyzerProvider
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:28: 错误: 找不到符号
[javac] Environment env, @assisted String name, @assisted Settings settings)
[javac] ^
[javac] 符号: 类 Environment
[javac] 位置: 类 JcsegAnalyzerProvider
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegAnalyzerProvider.java:28: 错误: 找不到符号
[javac] Environment env, @assisted String name, @assisted Settings settings)
[javac] ^
[javac] 符号: 类 Settings
[javac] 位置: 类 JcsegAnalyzerProvider
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegTokenizerFactory.java:3: 错误: 程序包org.apache.lucene.analysis不存在
[javac] import org.apache.lucene.analysis.Tokenizer;
[javac] ^
[javac] D:\fenci\jcseg-master\src\org\lionsoul\jcseg\elasticsearch\index\analysis\JcsegTokenizerFactory.java:4: 错误: 程序包org.elasticsearch.common.inject不存在

词库添加单词的疑问

lex-custom.lex

CJK_CUSTOM
纯手工分酒器醒酒器/null/null/null

在词典添加单词之后 分词依然会将纯手工分酒器醒酒器拆分

jcseg2.2.0在elasticsearch5.6.5中access denied的问题!且安装了这个的节点能够正常运行但是数据基本是废了!

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[node-156][192.168.8.156:9300][indices:admin/analyze[s]]"}],"type":"access_control_exception","reason":"access denied ("java.io.FilePermission" "plugins/jcseg/jcseg.properties" "read")"},"status":500}

已经给文件添加足够的权限了。但是还是报文件读取没有权限的问题!装有jcseg的elasticsearch节点中没有索引数据,其他elasticsearch节点也不能同步索引数据!移除了jcseg后,这个问题没有了!

升级2.1.0 词库格式遇到问题

我想问下是不是词库格式变化了 ,下面是报错:
image

我看了您写的change文档 好像是变化了 但是没有格式说明
我的格式为:
image

用2.0.0就不会报错 我DEBUG 看了下是 ADictionary 的报错 ,是必须 词语要加上词性等3个元素么,长度只要不等于4就会有问题,不知道是不是个BUG 您帮忙看下!!!

ES5.3版本中不能指定为自定义分析器的tokenizer吗?

看了源码貌似并没有注册相关的tokenizer,通过下面的(部分)代码建立index会出错:"type":"illegal_argument_exception","reason":"Custom Analyzer [reuters] failed to find tokenizer under name [jcseg]"}]
我只是想在分词前去掉一些无意义的标签,不知用char_filter是否正确,还请指导如何设置最合适。
谢谢!

						'analyzer' => [
							'reuters' => [
								'type' => 'custom',
								'char_filter' =>  ['html_strip', 'custom_char_filter'],
								'tokenizer' => 'jcseg',
								'analyzer' => 'jcseg_nlp',
								'filter' => ['lowercase', 'stop', 'kstem']
							]
						]

高亮显示异常

在elasticsearch索引中配置如下分词:

"analyzer": {
"text_analyzer": {
"type": "custom",
"char_filter": [
"html_strip"
],
"tokenizer": "jcseg",
"filter": [
"lowercase",
"tsconvert"
]
},
"complex_analyzer": {
"type": "custom",
"char_filter": [
"html_strip"
],
"tokenizer": "jcseg_complex",
"filter": [
"lowercase",
"tsconvert"
]
}

这时,被分析的文本如果出现html标签时,搜索结果就会出现高亮部分偏移问题,偏移字符数与html标签个数一致,如图所示:图片地址

暂时未测试其他分词插件与html_strip过滤器一起使用时是否存在高亮偏移问题;

自动摘要

我试试了自动摘要,根本是没意义的算法,就不是完全是没法用,乱七八糟的。做就好好做,做个有意思的。这个是什么破玩意儿。浪费我们的时间了

代码示例错误

在Jcseg自动摘要/关键句子提取Api这部分中,代码:
Segment seg = SegmentFactory.createJcseg(
JcsegTaskConfig.COMPLEX_MODE,
new Object[]{config, dic}
);
和SummaryExtractor extractor = new TextRankSummaryExtractor(seg, new SentenceSeg());中,
Segment应该为ISegment,SummaryExtractor应该为具体的TextRankSummaryExtractor类,请作者看一下是否正确?刚开始学这方面的知识,若提示错误敬请原谅!!!

拆词后搜索结果不理想

使用了jcseg(复合模式)对下面这句话拆词索引:‘年轻人不能打架’ 使用年轻人,打架 甚至整句话‘年轻人不能打架’搜索 均无法搜索到对应的文档,尝试把‘年轻人’和‘打架’加进了词典,还是不行,该如何优化?

jcseg 如何解析外国姓名翻译后的**人名?

例如:
“亨利·福特(1863年7月30日—1947年4月7日),出生于美国密歇根州韦恩郡的史普林威尔镇,他是汽车工程师,1915年,美国总统伍德罗·威尔逊接见福特,盛赞福特汽车公司,他是世界上第一位使用流水线大批量生产汽车的人。”
如何把 “亨利·福特” 这个人名在关键词中解析出来,而不是单独解析出 “亨利”
我的配置为:
image
请您帮忙解答一下,谢谢

请问一下该项目的后期发展计划

该项目有没有一个todo list 呢;
很想知道下一步以及以后的规划

还请问一下,有没有计划做 实体识别、文本分类的功能;
此外目前的关键词提取还是有些不够准确,有没有计划考虑集成 TF/IDF 进行完善呢

本机的外网网址为什么不支持

我测试了各种模式,都存在问题
1.换行没了,这个我估计我只能逻辑处理
2.遇到¥(各种特殊字符)会丢失,这个我无法处理
3.全部变成了小写,有没有选项保证词不变!
4.网址也被拆分了,这个我用代码也实现了,还原不过不完美!

总之,我实现不了,只能纯文字的处理似乎还行!

关于nlp分词模式下,组合时间的问题

现在用的是2.1.1版本,nlp分词模式
1当遇到13:00-15:00这种时间段,似乎不能识别,自然与其他datatime类型组合也不能识别。所以想深究一下为什么这个时间段会识别,但是给定的entity是en,这个en是怎么给定的,假如我改成datatime类型是不是就可以与其他datatime类型组合一起识别
2组合时间测试的时候“张华平明天要结婚了”,分词结果只会有“张华平 明天”,而把结婚这么重要的词给省略。查看源码知道“明天”这个datatime 类型,会判断后面一个词的是否是datatime类型,是否要组合到一起,此时结婚是nv类型,不是datatime类型,所以不用组合,跳出组合循环代码的时候,没有把结婚这个词写回StringReader,因为源码里面是只有是null类型才会写回String流,所以稍微改了 一下源码,只要不是datatime类型,都要写会String流,后来测试成功验证了我的猜想。请问作者我的思路对否?

接口不兼容

/faywong/devel/elasticsearch-1.7.5/plugins/jcseg/jcseg-elasticsearch-1.9.9.jar!/es-plugin.properties]
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.lionsoul.jcseg.elasticsearch.plugin.analysis.jcseg.AnalysisJcsegPlugin]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:536)
    at org.elasticsearch.plugins.PluginsService.loadPluginsFromClasspath(PluginsService.java:411)
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:116)
    at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:157)
    at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
    at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: java.lang.IncompatibleClassChangeError: class org.lionsoul.jcseg.elasticsearch.plugin.analysis.jcseg.AnalysisJcsegPlugin has interface org.elasticsearch.plugins.Plugin as super class
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:513)

运行环境:
elasticsearch-1.7.5
最新的Jcseg代码

编译源代码 报错

在项目根目录, 通过以下命令 编译代码:

mvn compile

报错:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] jcseg .............................................. SUCCESS [ 0.002 s]
[INFO] jcseg-core ......................................... FAILURE [ 0.588 s]
[INFO] jcseg-analyzer ..................................... SKIPPED
[INFO] jcseg-elasticsearch ................................ SKIPPED
[INFO] jcseg-server ....................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.676 s
[INFO] Finished at: 2016-09-11T16:08:39+08:00
[INFO] Final Memory: 14M/303M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:2.5:copy-resources (properties-file-copy) on project jcseg-core: Execution properties-file-copy of goal org.apache.maven.plugins:maven-resources-plugin:2.5:copy-resources failed: Plugin org.apache.maven.plugins:maven-resources-plugin:2.5 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.maven:maven-plugin-api:jar:2.0.6, org.apache.maven:maven-project:jar:2.0.6, org.apache.maven:maven-profile:jar:2.0.6, org.apache.maven:maven-artifact-manager:jar:2.0.6, org.apache.maven:maven-plugin-registry:jar:2.0.6, org.apache.maven:maven-core:jar:2.0.6, org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.6, org.apache.maven:maven-repository-metadata:jar:2.0.6, org.apache.maven:maven-error-diagnostics:jar:2.0.6, org.apache.maven:maven-plugin-descriptor:jar:2.0.6, classworlds:classworlds:jar:1.1, org.apache.maven:maven-artifact:jar:2.0.6, org.apache.maven:maven-settings:jar:2.0.6, org.apache.maven:maven-model:jar:2.0.6, org.apache.maven:maven-monitor:jar:2.0.6, org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-9-stable-1: Could not transfer artifact org.apache.maven:maven-plugin-api:jar:2.0.6 from/to central (
https://repo.maven.apache.org/maven2): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]


请问 如何 可以在本地 编译并 生成 所有 jar 包?

一类英文分词错误的问题

版本号:2.0及以上
英文(java异常日志的package名):com.com.com.com.com.com.com.com.com.com.com.java.java.java.java.java.java.java
例如:
jcseg~tokenizer:complex>> com.com.com.com.com.com.com.com.com.com.com.java.java.java.java.java.java.java
分词结果:
com.com.com.com.com.com.com.com.com.com.com.java.java.java.java.j/en com/en com/en com/en com/en com/en com/en com/en com/en com/en com/en com/en java/en java/en java/en java/en ava.java.java/en ava/en java/en java/en
其中错误的分词:com.com.com.com.com.com.com.com.com.com.com.java.java.java.java.j/en ava.java.java/en ava/en

**仓库无法下载2.3.0版本

你好,想依赖2.3.0版本,报错无法下载相关依赖。查看**仓库,发现/repositories/central/org/lionsoul/jcseg目录下,只更新到2.2.0版本,parent pom无法下载下来。是否发布到**仓库时遗漏了?

停用词自定义使用

jcseg 我在自定义停用词的时候 将一个名为lex-stopword.lex 的文件加到了resource文件夹中,然后将停用词开关打开 但是在写测试类 测试的时候发现 程序将这个文件中所有自定义的词当成了自定义词库分词 而不是当做停用词 我在网上找不到具体的自定义停用词的案例 请帮忙解答一下 谢谢

还有一个问题:
自动中英文停止词过滤功能(需要在jcseg.properties中开启该选项,lex-stopwords.lex为停止词词库)。
以上是你的说明中的一条信息 我可以理解为 只能往这个lex-stopwords.lex
JSegTextAnalyser.txt
pic

文件中加入停用词么 不能自己定义个文件放入自定义停用词

出现format error. -ignored错误

Word: "com" format error. -ignored
Word: "net" format error. -ignored
Word: "top" format error. -ignored
Word: "tech" format error. -ignored
Word: "org" format error. -ignored
Word: "gov" format error. -ignored
Word: "edu" format error. -ignored
Word: "mil" format error. -ignored
Word: "pub" format error. -ignored
Word: "mobi" format error. -ignored
Word: "cn" format error. -ignored
Word: "com.cn" format error. -ignored
Word: "net.cn" format error. -ignored
Word: "gov.cn" format error. -ignored
Word: "kilometer" format error(single word degree should be an integer). -ignored
Word: "刘翔" format error(single word degree should be an integer). -ignored
Word: "诸葛亮" format error(single word degree should be an integer). -ignored
1355 ERROR (coreLoadExecutor-6-thread-1) [ ] o.a.s.c.CoreContainer Error creating core [core]: Could not load conf for core core: Can't load schema D:\dev\Tomcat\solr-server\webapps\solr\solrhome\core\conf\managed-schema: Plugin init failure for [schema.xml] fieldType "textComplex": Plugin init failure for [schema.xml] analyzer/tokenizer: Error instantiating class: 'org.lionsoul.jcseg.analyzer.v5x.JcsegTokenizerFactory'

没有同义词啊

看了一下词典,貌似只有“ 研究”这个demo用的词是有同义词的,其他的几乎没有啊。。。
什么时候能加上?或者我自己怎么添加同义词表比较好?

POST大文本分词的问题

在Python上测试

import requests
r = requests.post('http://127.0.0.1:1990/tokenizer/nlp', params={'text': '我' * 902})

时,响应r的headers显示{'Content-Length': '0', 'Connection': 'close'},且content为空字符串。分词服务器的日志显示

2017-07-19 11:39:18.952:WARN:oejh.HttpParser:qtp1283928880-13: Header is too large >8192
2017-07-19 11:39:18.952:WARN:oejh.HttpParser:qtp1283928880-13: bad HTTP parsed: 413 for HttpChannelOverHttp@2e8b8e3a{r=0,c=false,a=IDLE,uri=-}

请问是需要设置请求的Header什么的吗?

结合ES使用时词典更新问题

我将Jcseg配置到ES里面,并开启词典自动更新,更新周期为1s,但实际使用中并没有更新。

以及ES作为一个分布式的搜索引擎,是否有考虑在ES中如何将新词同步到每个节点的词典中?

是否能识别新词?

如下文本: 在贾跃亭一手带领下,近几年一直在资本市场炮制概念,宣称打造体育、影业、手机、汽车等等生态的乐视,从未停止过烧钱模式。 里,不论使用6种模式,贾跃亭都会被切分成 //

jieba分词里有新词识别功能,请问Jcseg是否有类似的功能?

elasticsearch权限问题

使用版本号:jcseg-2.0.1-RC2
es版本:2.4.1
异常信息:
Exception in thread "main" java.security.AccessControlException: access denied ("java.io.FilePermission" "plugins/jcseg/jcseg.properties" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
at java.security.AccessController.checkPermission(AccessController.java:559)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.io.File.exists(File.java:808)
at org.lionsoul.jcseg.elasticsearch.index.analysis.JcsegIndicesAnalysis.(JcsegIndicesAnalysis.java:39)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at <<>>
at org.elasticsearch.node.Node.(Node.java:213)
at org.elasticsearch.node.Node.(Node.java:140)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

经过搜索,发现es有自己的权限策略,https://discuss.elastic.co/t/permission-denied-error-in-plugin-for-es-2-2-0/40915

hi 我现在使用es 6.2.2,使用jcseg的插件数组字段分词的时候会有问题

hi 我有个字段sim_q 是个数组字段,值如下格式 "sim_q":["xxxxx","xxxxx","xxxxx"] 在es里面指定jcseg_complex分词后,创建es索引的时候报如下错误:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=1,endOffset=2,lastStartOffset=5 for field 'sim_q'"}],"type":"illegal_argument_exception","reason":"startOffset must be non-negative,and endOffset must be >= startOffset, and offsets must not go backwards startOffset=1,endOffset=2,lastStartOffset=5 for field 'sim_q'"},"status":400}

原因可能是自es 6.0.0以后,分词的endOffset必须要>=startOffset
详见hankcs/hanlp-lucene-plugin#27
orangemi/elasticsearch-jieba-plugin#1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.