otoolep / bleve-bench Goto Github PK

I tried to use this with a wikipedia data dump and I think it choked because the lines are ridiculously long.

I think the problem is that scanner has a fixed-size buffer, and if a line exceeds that it just returns an error and stops. I rewrote that section to use:

reader := bufio.NewReader(f)
var l []byte
l, err = reader.ReadBytes(byte('\n'))
for err == nil {
    docs = append(docs, l)
    l, err = reader.ReadBytes(byte('\n'))
}

Then it worked fine with this data set.

Unfortunately, the current test doesn't seem to actually be indexing data (looking at boltdb files I only see keys for the back index). I believe the problem is that the struct only contains data of type []byte, which is not currently supported by bleve. See https://github.com/otoolep/bleve-bench/blob/master/bleve_indexer.go#L70

Also, the mapping sets up a field named "body", but then the struct being indexed doesn't use that name.

I would recommend you change that field to type string, and name it Body with struct tag json:"body".