Comments (6)
from gojieba.
虽然不太礼貌,不过既然是不正常了,那么求看一下,gojieba库,我的调用方法是不是有问题:
我换了个源,改成小说,来源是:http://www.iplaysoft.com/1326-txt-science-fiction.html
,
(源文件是cp936编码,手动转码成utf8)
大约1300个文件,43M。
索引完后的文件大约在900M
899.3 MiB [##########] /gojieba.bleve
43.3 MiB [ ] /source
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"github.com/blevesearch/bleve"
. "github.com/qiukeren/go-utils/common"
"github.com/yanyiwu/gojieba"
_ "github.com/yanyiwu/gojieba/bleve"
)
func Example() {
INDEX_DIR := "gojieba.bleve"
dirEntries, err := ioutil.ReadDir("/Users/XXX/gopath/src/search/bleve/source")
if err != nil {
log.Panicln(err)
}
type Message struct {
Id string
Content string
}
indexMapping := bleve.NewIndexMapping()
os.RemoveAll(INDEX_DIR)
// clean index when example finished
// defer os.RemoveAll(INDEX_DIR)
err = indexMapping.AddCustomTokenizer("gojieba",
map[string]interface{}{
"dictpath": gojieba.DICT_PATH,
"hmmpath": gojieba.HMM_PATH,
"userdictpath": gojieba.USER_DICT_PATH,
"type": "gojieba",
"idf": gojieba.IDF_PATH, //idf 与stop_words必须要加,不然报错,此处采用自带的idf
"stop_words": gojieba.STOP_WORDS_PATH,
},
)
if err != nil {
panic(err)
}
err = indexMapping.AddCustomAnalyzer("gojieba",
map[string]interface{}{
"type": "gojieba",
"tokenizer": "gojieba",
},
)
if err != nil {
panic(err)
}
indexMapping.DefaultAnalyzer = "gojieba"
index, err := bleve.New(INDEX_DIR, indexMapping)
if err != nil {
panic(err)
}
log.Println(len(dirEntries))
for k, v := range dirEntries {
log.Println(k, "of", len(dirEntries))
data, _ := ReadToString("/Users/XXX/gopath/src/search/bleve/source/" + v.Name())
message := Message{Id: v.Name(), Content: string(data)}
// go func(name string, content Message) {
index.Index(v.Name(), message)
// }(v.Name(), message)
}
querys := []string{
"你好世界",
"亲口交代",
}
for _, q := range querys {
req := bleve.NewSearchRequest(bleve.NewQueryStringQuery(q))
req.Highlight = bleve.NewHighlight()
res, err := index.Search(req)
if err != nil {
panic(err)
}
fmt.Println(res)
}
}
func main() {
Example()
}
from gojieba.
sphinx挺好的
from gojieba.
@wenduniang
我自己选型的规则是,Go>java>php>python>C/C++。
单单就搜索方面的选型是,Go>java>python>c/c++/php。
java尚且还有elk、lucene等一系列框架可以选,我没必要再舍近求远了。
from gojieba.
解决了吗
from gojieba.
并没有
from gojieba.
Related Issues (20)
- 这个和fxsjy/jieba的结果是一致的吗 HOT 2
- [已解决] 我尝试了最新master代码,依旧内存泄露 HOT 1
- cgo 奔溃
- cc1.exe: sorry, unimplemented: 64-bit mode not compiled in HOT 1
- 容器化报错 HOT 1
- 报错,看不懂 HOT 4
- RemoveWord方法无效,而且AddWord针对英文词组无效
- 日期切分的问题
- 请问下,支持基于 TF-IDF 算法的关键词抽取么
- 编译时遇到Warming HOT 1
- 内存泄漏问题 HOT 1
- I used it in the goframe framework project. In the Ubuntu, the compilation will report an error HOT 1
- x.Addword 不生效
- NewJieba not declared by package gojiebacompiler HOT 1
- 跨平台编译问题
- 支持打包为arm 和arm64架构吗 HOT 1
- 如何动态删除一个词 HOT 1
- Extract 和Cut的区别 HOT 1
- 集成到bleve的时候,stop words无效 HOT 1
- x.AddWord之后,希望增加x.RemoveWord的功能 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gojieba.