tidwall / buntdb Goto Github PK

View Code? Open in Web Editor NEW

4.4K 105.0 281.0 418 KB

BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

License: MIT License

Go 100.00%

database in-memory key-value golang geospatial

buntdb's People

Contributors

Stargazers

Watchers

Forkers

rtvt123 jacobh number0 southpolesteve olurin damilare bpd1069 caidongyun lewisliang82 maniacs-db wxiaowar davidmr001 golang-kit olivierbarbier mantyr fnet123 duzhanyuan weijingtao nguyenducnhaty hilerchyn richard-deng zhoudaqing namistudio sdboyer e4x tharanga-abeyseela jiulu313 boreys dw250100785 jmptrader ssor pombredanne appleboy k0mrade prisoner devopsbox montazze beijinggao dongfanliang bpatelcs nvhungiph eeertekin diycp bremmbo joe2hpimn cuulee xww pilgrim2go alex-xiao liexusong kokizzu rootfs-analytics bobquest33 zmyer hemel-cse e1101 glycerine patrick-park barrycug fifthsegment uk702 gsdu8g9 chaitanya9186 guangminglion ruschein kryptblue zfl926 devopsmi askagirl sheelkhanna fengweijp sycdtk graha oneoaas-golib legendzhu gallir cristaloleg icron aa55941767 covrom knewic cluo exlimit anismiles leearmstrong lorenzomcmaster chennqqi huaiguoguo vinayselvaraj arryboom bowin lhslll smartfish fitan luomeiqin augustand hzhang08 valinurovam uniseraph avinashdvv

buntdb's Issues

Question: Retrieve by index

How to select data for condition - json field absent ?

    db.CreateIndex("age", "*", buntdb.IndexJSON("age"))
       tx.Set("5", `{"name":{"first":"Alan"},"age":28}`, nil)
        tx.Set("6", `{"name":{"first":"Dred"}}`, nil)

How to get number key=6 by age == nil?

And second question how to get key=5 by exact value age=28 ?

Is the function of data expiration still valid in such case?

I want to create a database on disk, and let all item expire in 7 days after creating.

Here is the code:

db.Update(func(tx *buntdb.Tx) error {
		_, _, err = tx.Set("word", "define",
			&buntdb.SetOptions{Expires: true, TTL: time.Duration(7*24) * time.Hour})
		return
})

However, the program will not run for such long time. Is the function of data expiration still valid?

Here is the result I expect:

Run the program in Nov 1st, add item "A" in the database and then exit.
Run the program in Nov 3rd, add item "B" in the database and then exit.
Run the program in Nov 7th. At this time, item "A" and "B" are still existent.
Run the program in Nov 9th. At this time, only item "B" is still existent.
Run the program in Nov 11th. At this time, non of item "A" and "B" is existent.

Is it possible?

data expiration

It seems like the expiration only works as long as the program does not exit before the expiration.

If I set a key to expire in 10 seconds and the program finishes in 5 I can run it as many times as I want without the expiration ever happening. Is this intentional?

panic: runtime error: index out of range

Hi, when try to ran this code:

package main

import (
    "fmt"
    "log"
    "math/rand"
    "time"

    "github.com/tidwall/buntdb"
)

var count = 1000

func init() {
    rand.Seed(time.Now().UnixNano())
}

func main() {
    db, err := buntdb.Open("data.db")
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    db.CreateIndex("ages", "user:*:age", buntdb.IndexInt)

    for i := 0; i < 10; i++ {

        fmt.Println("iteration: ", i)

        db.Update(func(tx *buntdb.Tx) error {
            for j := 0; j < count; j++ {
                key := fmt.Sprintf("user:%d:age", j)
                val := fmt.Sprintf("%d", rand.Intn(100))
                tx.Set(key, val, nil)
            }
            return nil
        })

        db.Update(func(tx *buntdb.Tx) error {
            tx.Ascend("ages", func(key, val string) bool {
                _, err := tx.Delete(key)
                if err != nil {
                    log.Fatal(err)
                }
                return true
            })
            return nil
        })

    }

}

I've got a panic on different iteration

panic: runtime error: index out of range

goroutine 1 [running]:
panic(0x50c8a0, 0xc42000a1a0)
        /home/theaidem/.envirius/envs/develope/go/src/runtime/panic.go:500 +0x1a1
github.com/tidwall/btree.(*node).iterate(0xc4200127c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x520100, 0xc420043d40, 0x4fa280, 0xc4200161e0, ...)
        /home/theaidem/work/golang/sandbox/buntdb_ex/src/github.com/tidwall/btree/btree.go:459 +0x743
github.com/tidwall/btree.(*BTree).Ascend(0xc420014e70, 0xc420043d40)
        /home/theaidem/work/golang/sandbox/buntdb_ex/src/github.com/tidwall/btree/btree.go:637 +0x88
github.com/tidwall/buntdb.(*Tx).scan(0xc420010540, 0xc420000000, 0x52d7f4, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc420043df8, 0xc420043e08, ...)
        /home/theaidem/work/golang/sandbox/buntdb_ex/src/github.com/tidwall/buntdb/buntdb.go:1347 +0x317
github.com/tidwall/buntdb.(*Tx).Ascend(0xc420010540, 0x52d7f4, 0x4, 0xc420043df8, 0xc420043e20, 0x46f040)
        /home/theaidem/work/golang/sandbox/buntdb_ex/src/github.com/tidwall/buntdb/buntdb.go:1434 +0x80
main.main.func2(0xc420010540, 0x53c298, 0xc420010540)
        /home/theaidem/work/golang/sandbox/buntdb_ex/main.go:47 +0x5b
github.com/tidwall/buntdb.(*DB).managed(0xc42008a000, 0x1, 0x53c2b0, 0x0, 0x0)
        /home/theaidem/work/golang/sandbox/buntdb_ex/src/github.com/tidwall/buntdb/buntdb.go:869 +0xf7
github.com/tidwall/buntdb.(*DB).Update(0xc42008a000, 0x53c2b0, 0x0, 0x0)
        /home/theaidem/work/golang/sandbox/buntdb_ex/src/github.com/tidwall/buntdb/buntdb.go:892 +0x3a
main.main()
        /home/theaidem/work/golang/sandbox/buntdb_ex/main.go:49 +0x244
exit status 2

my go version is 1.7
not sure, probably the issue relate tidwall/btree package
any ideas?

Question: Search by index with equal comparison

Is it somehow possible to search by index with equal comparison (by value)? Do You planed to create a custom method for that?

Thanks!

Running speed is very slow when use the package

need use in linux?

DB server basing buntdb

Would it be a good idea to build an web wrap for buntdb for using buntdb by multiple consumers?

Q: why strings?

I'm just curious as to why keys and values are strings and not byte slices like it is more common with other Go databases/kv stores like Badger or Bolt?

Looking at the code I see you are using the first character of a new line in the db file as metadata info which I think can be easily done with bytes as well.

Is this because SummitDB and GJSON, SJSON were planned beforehand or is there a different reason for this decision?

How to save memory data to JSON

buntdb.Open(":memory:")

Now I want to save the data in memory to a JSON file

The next time you start, load the JSON data

How to set the key's index and iterator?

My key was timestamps ,and i want use the timestamps key iterator,Hao to do it?
however the Index can only be set the key of values ?

documentation mark and question

Hi,

2D point: [10 15 12]
this will be
2D point: [10 15].

Question: Is it possible/correct to use the 2D point spatial index as data time range?
For example, [startTime stopTime] UnixNano --> int64 as XY.

Can you please share performance stats ? Comparing BuntDB against the likes of elastic search..

how to save the DB

I'm unable to call save function for making my DB persistent. getting an error in io.Writer

hello, tidwall
I hoped to get the ascend results is right when I was using tx.AscendRange or tx.DescendRange fuction.
But when i hoped to get the descend results, it's nil of return.
Can you help me? thanks.

db, _ := buntdb.Open(":memory:")
	db.Update(func(tx *buntdb.Tx) error {
		tx.CreateIndex("last_name", "*", buntdb.IndexJSON("name.last"))
		
		tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":"38"}`, nil)
		tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":"47"}`, nil)
		tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":"52"}`, nil)
		tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":"28"}`, nil)
		tx.Set("5", `{"name":{"first":"Jack","last":"Ma"},"age":"20"}`, nil)
		tx.Set("6", `{"name":{"first":"Blues","last":"Li"},"age":"31"}`, nil)
		tx.Set("7", `{"name":{"first":"Pony","last":"Ma"},"age":"3"}`, nil)
		tx.Set("8", `{"name":{"first":"Robin","last":"Li"},"age":"2"}`, nil)
		tx.Set("9", `{"name":{"first":"Larry","last":"Liu"},"age":""}`, nil)
		return nil
	})

	db.View(func(tx *buntdb.Tx) error {
		fmt.Println("---------------------")
		tx.Descend("last_name", func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
		
		fmt.Println("---------------------")
		tx.DescendRange("last_name", `{"name":{"last":""}}`, `{"name":{"last":"Ma"}}`, func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
		return nil
	})

result:

---------------------
2: {"name":{"first":"Janet","last":"Prichard"},"age":"47"}
7: {"name":{"first":"Pony","last":"Ma"},"age":"3"}
5: {"name":{"first":"Jack","last":"Ma"},"age":"20"}
9: {"name":{"first":"Larry","last":"Liu"},"age":""}
8: {"name":{"first":"Robin","last":"Li"},"age":"2"}
6: {"name":{"first":"Blues","last":"Li"},"age":"31"}
1: {"name":{"first":"Tom","last":"Johnson"},"age":"38"}
4: {"name":{"first":"Alan","last":"Cooper"},"age":"28"}
3: {"name":{"first":"Carol","last":"Anderson"},"age":"52"}
---------------------

tx.CreateIndex("last_name", "*", buntdb.IndexJSON("name.last")
replace
tx.CreateIndex("last_name", "*", buntdb.Desc(buntdb.IndexJSON("name.last")))

result:

---------------------
3: {"name":{"first":"Carol","last":"Anderson"},"age":"52"}
4: {"name":{"first":"Alan","last":"Cooper"},"age":"28"}
1: {"name":{"first":"Tom","last":"Johnson"},"age":"38"}
8: {"name":{"first":"Robin","last":"Li"},"age":"2"}
6: {"name":{"first":"Blues","last":"Li"},"age":"31"}
9: {"name":{"first":"Larry","last":"Liu"},"age":""}
7: {"name":{"first":"Pony","last":"Ma"},"age":"3"}
5: {"name":{"first":"Jack","last":"Ma"},"age":"20"}
2: {"name":{"first":"Janet","last":"Prichard"},"age":"47"}
---------------------
3: {"name":{"first":"Carol","last":"Anderson"},"age":"52"}
4: {"name":{"first":"Alan","last":"Cooper"},"age":"28"}
1: {"name":{"first":"Tom","last":"Johnson"},"age":"38"}
8: {"name":{"first":"Robin","last":"Li"},"age":"2"}
6: {"name":{"first":"Blues","last":"Li"},"age":"31"}
9: {"name":{"first":"Larry","last":"Liu"},"age":""}

How to use "pivot" value in AscendGreaterOrEqual func

I wrote this part of code, please take a look.
I created a custom index, on json field, and trying to iterate over it.
I expect to get exactly two values of that iteration, but always get three.

There is the code for test case:
https://gist.github.com/e5488c8efd08092673f2553bc83d2e83

How to retrieve record on basis of value without knowing the key?

Suppose,
{"userId": "VSCODE", "password": "CODEVS","name":"abcd"}
this is my value with key "VSCODE".
i want to fetch the record on name == "abcd" condition
How can i achieve this?

Delete function

Not able to delete records.

Advice? Slow JSON unmarshalling because of string -> []bytes

I wonder if anyone can offer any advice.

I've been doing extensive benchmarking of different Go KV stores for an ERP application. I am saving millions of 'transactions' serialised with JSON, reading them back, deserialising and running some aggregation operations.

I was comparing Bunt to BadgerDB and finding that Badger was 4x faster for my benchmark for reading and aggregating, which seemed odd to me.

After a little digging, I realised that if I removed the JSON unmarshalling step, Bunt became 2x faster than Badger. Obviously I am using the same JSON unmarshalling technique in both cases.

The problem lies in the fact that Bunt returns strings whereas Badger returns []byte. This introduces another step in the deserialisation for Bunt json.Unmarshal([]byte(value), &order) which is what is causing the slowdown on the Bunt benchmark.

Ive Googled about a bit and understand that the string -> []byte conversion is costly, and have come across some solutions involving unsafe, but haven't been able to get them to work for me.

Has anyone tackled this issue and come up with a good solution. BadgerDB is really nice, but Bunt's indexing functionality is just what I'm looking for and would really love to choose it for this project. I can see that the raw iteration speed is faster than Badger - its just this string issue thats causing the slower aggregation.

Is There any way to operate DB with password?

when I operate with buntdb in local, I may want to encrypt the data. and how to operate data with a password?

Collate index invalid

hi , I used the collate index, but the result is not what I expected.
can anyone help me? Thanks！

db, _ := buntdb.Open(":memory:")
	db.Update(func(tx *buntdb.Tx) error {
		tx.CreateIndex("last_name", "*", collate.IndexJSON("ENGLISH_CS", "name.last"))
		tx.Set("1", `{"name":{"first":"Tom","last":"Johnson"},"age":"38"}`, nil)
		tx.Set("2", `{"name":{"first":"Janet","last":"Prichard"},"age":"47"}`, nil)
		tx.Set("3", `{"name":{"first":"Carol","last":"Anderson"},"age":"52"}`, nil)
		tx.Set("4", `{"name":{"first":"Alan","last":"Cooper"},"age":"28"}`, nil)
		tx.Set("5", `{"name":{"first":"Jack","last":"Ma"},"age":"20"}`, nil)
		tx.Set("6", `{"name":{"first":"Blues","last":"Li"},"age":"31"}`, nil)
		tx.Set("7", `{"name":{"first":"Pony","last":"Ma"},"age":"3"}`, nil)
		tx.Set("8", `{"name":{"first":"Robin","last":"Li"},"age":"2"}`, nil)
		tx.Set("9", `{"name":{"first":"Larry","last":"Liu"},"age":""}`, nil)
		return nil
	})
	db.View(func(tx *buntdb.Tx) error {
		fmt.Println("---------------------")
		tx.Ascend("last_name", func(key, value string) bool {
			fmt.Printf("%s: %s\n", key, value)
			return true
		})
		return nil
	})

result：

-------------------------
4: {"name":{"first":"Alan","last":"Cooper"},"age":"28"}
6: {"name":{"first":"Blues","last":"Li"},"age":"31"}
3: {"name":{"first":"Carol","last":"Anderson"},"age":"52"}
5: {"name":{"first":"Jack","last":"Ma"},"age":"20"}
2: {"name":{"first":"Janet","last":"Prichard"},"age":"47"}
9: {"name":{"first":"Larry","last":"Liu"},"age":""}
7: {"name":{"first":"Pony","last":"Ma"},"age":"3"}
8: {"name":{"first":"Robin","last":"Li"},"age":"2"}
1: {"name":{"first":"Tom","last":"Johnson"},"age":"38"}

How to prevent the duplicated data with the same key?

I'm playing around with BuntDB with this code (and executed it few times):

db.Update(func(tx *buntdb.Tx) error {
    _, _, err := tx.Set("token:ABCDEF", "Test", nil)
    return err
})

And I opened the data.db file I saw this.

*3
$3
set
$38
token:ABCDEF
$31
Test
*3
$3
set
$38
token:ABCDEF
$31
Test
*3
$3
set
$38
token:ABCDEF
$31
Test
*3
$3
set
$38
token:ABCDEF
$31
Test

Why would something like this happened? I thought BuntDB will overwrite the key, value but it just added another record.

Do I have to check the key does exist or not before I Set the key? If so, should I use Get to check it before I Set it?

compare with Aerospike?

Is there a another way to avoid with deprecate warning ?

Thanks in advance..
I'm new to buntdb. but while Testng example code, I got a message about deprecated method.
for example, below ....
db.CreateIndex("ages", "user:*:age", buntdb.IndexInt)

readme.md is too old ? if not, (in source code) comment is misspelled ?

Does pattern del disk val supported?

look source code, i find if delete btree value.program append del value in disk. long time! disk size very big!I think use pattern handle update and del.

Performance on large datasets

How does buntdb perform on data too large to fit into ram? would i have good performance with a dataset of over 5gb (mongodb dump), on a 500mb or 1gig ram server?

I'm currently evaluating buntdb over boltdb. As replacement for a system that currently runs on mongodb. I intend to use bleve search as well for indexing and search. Can I query with an array of keys? or get each key from the array individually?

Question: Intersects with polygon

Is it somehow possible to intersect queries with polygons? Do i have to create a custom indexer for that?

Thanks!

Save index operations

THE AOF saves the set and del operations. However, the index creation operations are not saved. When a program restarts it's hard to reproduce the previous indexes.

Key not found error even when the key is present in the corresponding file

Hi, is there any reason why a stored key-value pair is not found even though the key is present in the DB file? My use case is as follows:

A server writing key-value pairs to this file
Go tests updating the file

Any help would be appreciated.
Thanks.

can

Specific documentation v.v. overhead

We have billions of geospacial items across the planet what's our minimum overhead?

Error checking

Hi Josh, Nice project.
I noticed that you didn't check some errors that are returned from your functions
Example:

Is there a reason why you didn't check most errors?
(I am trying to learn more about the best practices in Go language)

NEARBY

I see that tile38 supports a NEARBY query, is that possible at all with buntdb?

http://tile38.com/commands/nearby/

Case insensitive match

Hi, is it possible to match key+"*" in case insensitive mode? Thanks

can you limit memory usage?

can you limit memory usage or set max disk usage?

how large document count support？

sorry for bad english
how large data can index？
i have 128GB data for index
and will support stastics query?

How to save one key with multiple values?

Hello,

I have an XML of this structure and I want to transfer its data to Bunt DB

<MonitoringApps>
    <VD Id="XYZ">
        <ClientId>123456789</ClientId>
        <ClientSecret>0987654321</ClientSecret>
    </VD>
</MonitoringApps>

As it is shown per VD id we have one client id and one secret. Is there any way to save the VD id as key and others as 2 values of it in the DB.

Delete inside iterator

tx.AscendKeys("object:*", func(k, v string) bool {
	if someCondition(k) == true {
		if _, err = tx.Delete(k); err != nil {

                            // got: "tx is iterating" error here

			return false // break
		}
	}
	return true // continue
})

Is there a way to Delete inside iterator?

Export db.Begin function

This one easy to do and allows controlling transaction inside application.

Can't get AscendGreaterOrEqual to work.

I have created an index with:
db.CreateIndex("idx", "idx:*", buntdb.IndexInt)
Then I set values with something like:

idx:=1
idxs := fmt.Sprintf("%d", idx)
_, _, err = tx.Set("idx:"+idxs, msg, nil)

Finally I try to retrieve values with:

since:= 1
err = dht.db.View(func(tx *buntdb.Tx) error {
	err = tx.AscendGreaterOrEqual("idx", string(since), func(key, value string) bool {
		log.Debugf("Got key:%s  and val:%s", key, value)
		return true
	})
	return err
})

if since = 1 or 0 then I get back all entries, but if since is > 1 I still get back all values! What am I doing wrong? Thanks! I.e. it doesn't actually seem to retrieve a subset.

Why does invalid database appear?

Hello, I found that my database is broken. There is only \r at the end of a line, so an invalid database error is reported. I would like to ask why this is the case. Will it happen when multiple processes are open? Thank you!

Attempting to use AscendEqual

I'm trying to use the AscendEqual function to iterate through entries, but it's not returning the results I'm expecting. I've narrowed the issue down to the following large-ish (sorry) test case:

func TestNameEquals(t *testing.T) {
	iter := func(key, value string) bool {
		i, _ := strconv.Atoi(key)
		fmt.Printf("%2d - %v\n", i, value)
		return true
	}

	j := func(v string, a int) string {
		return fmt.Sprintf("{\"name\":\"%v\", \"age\":%v}", v, a)
	}

	name := func(v string) string {
		return fmt.Sprintf("{\"name\":\"%v\"}", v)
	}

	db, _ := buntdb.Open(":memory:")
	db.CreateIndex("name", "*",
		buntdb.IndexJSON("name"),
		buntdb.IndexJSON("age"),
	)
	db.Update(func(tx *buntdb.Tx) error {
		tx.Set("1", j("bob", 42), nil)
		tx.Set("2", j("james", 55), nil)
		tx.Set("4", j("logan", 14), nil)
		tx.Set("11", j("james", 11), nil)
		tx.Set("3", j("clark", 33), nil)
		tx.Set("5", j("tommy", 15), nil)
		tx.Set("6", j("zane", 19), nil)
		return nil
	})

	db.View(func(tx *buntdb.Tx) error {
		fmt.Println(" -- Greater than d")
		tx.AscendGreaterOrEqual("name", name("d"), iter)

		fmt.Println(" -- Equals James")
		tx.AscendEqual("name", name("james"), iter)

		return nil
	})
}

The output from running the test case is:

=== RUN   TestNameEquals
 -- Greater than d
11 - {"name":"james", "age":11}
 2 - {"name":"james", "age":55}
 4 - {"name":"logan", "age":14}
 5 - {"name":"tommy", "age":15}
 6 - {"name":"zane", "age":19}
 -- Equals James
--- PASS: TestNameEquals (0.00s)

AscendGreaterOrEqual works exactly as expected, but I was hoping that AscendEqual would allow me to iterate only the james entries sorted by the age field. Have I misunderstood the purpose of the function?

fsync just once a second is a very bad idea.

You should at least mention in a README, that user can loose whole second of changes.

Tests fail with Go 1.14 race detector.

Context: https://golang.org/doc/go1.14#compiler

Test output:

go test -v -race ./...
=== RUN   TestBackgroudOperations
--- PASS: TestBackgroudOperations (3.46s)
=== RUN   TestSaveLoad
--- PASS: TestSaveLoad (0.00s)
=== RUN   TestMutatingIterator
--- PASS: TestMutatingIterator (0.19s)
=== RUN   TestCaseInsensitiveIndex
--- PASS: TestCaseInsensitiveIndex (0.03s)
=== RUN   TestIndexTransaction
--- PASS: TestIndexTransaction (0.01s)
=== RUN   TestDeleteAll
--- PASS: TestDeleteAll (0.02s)
=== RUN   TestAscendEqual
--- PASS: TestAscendEqual (0.02s)
=== RUN   TestDescendEqual
--- PASS: TestDescendEqual (0.02s)
=== RUN   TestVariousTx
fatal error: checkptr: unsafe pointer conversion

goroutine 27 [running]:
runtime.throw(0x12c2d5f, 0x23)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc0001372e8 sp=0xc0001372b8 pc=0x1070f52
runtime.checkptrAlignment(0xc0001692c0, 0x12a37a0, 0x1)
	/usr/local/go/src/runtime/checkptr.go:20 +0xc9 fp=0xc000137318 sp=0xc0001372e8 pc=0x10443d9
github.com/tidwall/rtree/base.(*treeItem).unsafeNode(...)
	/Users/olone/go/pkg/mod/github.com/tidwall/[email protected]/base/rtree.go:187
github.com/tidwall/rtree/base.(*RTree).insert(0xc00024c000, 0xc00007c6c0, 0x1296c60, 0xc0002483c0, 0x0, 0xc000058c00)
	/Users/olone/go/pkg/mod/github.com/tidwall/[email protected]/base/rtree.go:242 +0x2bc fp=0xc000137408 sp=0xc000137318 pc=0x12179cc
github.com/tidwall/rtree/base.(*RTree).Insert(0xc00024c000, 0xc000234990, 0x2, 0x2, 0xc0002349a0, 0x2, 0x2, 0x1296c60, 0xc0002483c0)
	/Users/olone/go/pkg/mod/github.com/tidwall/[email protected]/base/rtree.go:227 +0x1cd fp=0xc0001374b8 sp=0xc000137408 pc=0x121765d
github.com/tidwall/rtree.(*RTree).Insert(0xc0001691c0, 0x12fef00, 0xc0002483c0)
	/Users/olone/go/pkg/mod/github.com/tidwall/[email protected]/rtree.go:57 +0x2f7 fp=0xc000137598 sp=0xc0001374b8 pc=0x121b9a7
github.com/tidwall/buntdb.(*DB).insertIntoDatabase(0xc00012a160, 0xc0002483c0, 0xc0001376f8)
	/Users/olone/playground/buntdb/buntdb.go:494 +0x456 fp=0xc0001376a8 sp=0xc000137598 pc=0x12236a6
github.com/tidwall/buntdb.(*Tx).Set(0xc00000e700, 0x12bb6d9, 0x6, 0x12bd469, 0xf, 0x0, 0x1008710, 0xc000137868, 0x104a0bc, 0x1007970, ...)
	/Users/olone/playground/buntdb/buntdb.go:1378 +0x1cc fp=0xc000137780 sp=0xc0001376a8 pc=0x122b89c
github.com/tidwall/buntdb.TestVariousTx.func13(0xc00000e700, 0xc000137801, 0xc00000e700)
	/Users/olone/playground/buntdb/buntdb_test.go:950 +0x515 fp=0xc000137940 sp=0xc000137780 pc=0x1254245
github.com/tidwall/buntdb.(*DB).managed(0xc00012a160, 0x12bb101, 0xc000137e40, 0x0, 0x0)
	/Users/olone/playground/buntdb/buntdb.go:949 +0x153 fp=0xc000137a30 sp=0xc000137940 pc=0x1226f33
github.com/tidwall/buntdb.(*DB).Update(...)
	/Users/olone/playground/buntdb/buntdb.go:972
github.com/tidwall/buntdb.TestVariousTx(0xc00017e240)
	/Users/olone/playground/buntdb/buntdb_test.go:928 +0x171f fp=0xc000137ed0 sp=0xc000137a30 pc=0x123778f
testing.tRunner(0xc00017e240, 0x12c7870)
	/usr/local/go/src/testing/testing.go:991 +0x1ec fp=0xc000137fd0 sp=0xc000137ed0 pc=0x11721bc
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000137fd8 sp=0xc000137fd0 pc=0x10a4851
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1042 +0x661

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000158000, 0x12bce1d, 0xd, 0x12c7870, 0x1)
	/usr/local/go/src/testing/testing.go:1043 +0x699
testing.runTests.func1(0xc000158000)
	/usr/local/go/src/testing/testing.go:1284 +0xa7
testing.tRunner(0xc000158000, 0xc000135d50)
	/usr/local/go/src/testing/testing.go:991 +0x1ec
testing.runTests(0xc000138020, 0x14697a0, 0x20, 0x20, 0x0)
	/usr/local/go/src/testing/testing.go:1282 +0x528
testing.(*M).Run(0xc00014e080, 0x0)
	/usr/local/go/src/testing/testing.go:1199 +0x300
main.main()
	_testmain.go:168 +0x224

goroutine 40 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000142210)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 14 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0001422c0)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 16 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000142370)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 50 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000142420)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 52 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0001424d0)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 54 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc000142580)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 42 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0004c4000)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 43 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc0004c40b0)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 24 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00012a000)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 26 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00012a0b0)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640

goroutine 28 [chan receive]:
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc00012a160)
	/Users/olone/playground/buntdb/buntdb.go:536 +0x2c7
created by github.com/tidwall/buntdb.Open
	/Users/olone/playground/buntdb/buntdb.go:171 +0x640
FAIL	github.com/tidwall/buntdb	5.555s
FAIL

Sliding Expiration and Callback

This is a feature request - if it's proper to add them.

Adding sliding TTL; on each get operation, if the entry has a TTL and a sliding flag is set, the absolute internal expiration time will be shifted into future.
Setting a callback like func(key, value string) that will be called (in it's own new goroutine) when an entry gets expired.

Since these are optional, they would affect performance only if they are present.

Auto Shrink dont get triggered

Hi,

I have a program that performs some crawling, store and update data in buntdb and stops, after a few minutes a crontab launch de program again.

I don't know why but the database size grows and grows, containing some keys duplicated more than 50 times (Ending in multiple GB values completely full of duplicates).

There's any way I can force BuntDB to shrink the file at the end of every execution of the program or something like this?

Shrink, panic to many open files

~40 recording per seconds
small data (max 1Kb), raw json, without indexes
25/70 write and read

init code

db, err := buntdb.Open("_workspace/_cache.db")
	if err != nil {
		return
	}
	db.SetConfig(buntdb.Config{
		SyncPolicy:         buntdb.EverySecond,
	})

log error

panic: open _workspace/_cache.db: too many open files

goroutine 39 [running]:
github.com/tidwall/buntdb.(*DB).Shrink.func4(0xc4200e26e0, 0xc420172480, 0x14, 0xab24, 0xc43805e2d8, 0xc439733920, 0x18, 0x0, 0x0)
	/go/src/github.com/tidwall/buntdb/buntdb.go:738 +0x326
github.com/tidwall/buntdb.(*DB).Shrink(0xc4200e26e0, 0x0, 0x0)
	/go/src/github.com/tidwall/buntdb/buntdb.go:746 +0x2db
github.com/tidwall/buntdb.(*DB).backgroundManager(0xc4200e26e0)
	/go/src/github.com/tidwall/buntdb/buntdb.go:602 +0x247
created by github.com/tidwall/buntdb.Open
	/go/src/github.com/tidwall/buntdb/buntdb.go:165 +0x206

real	1m46.244s
user	0m34.317s
sys	0m2.755s

Is it in-memory db?

I worked with boltdb,it's like Buntdb. As mentioned in Documentation buntdb is an in-memory database(while boltdb isn't).
So can it manage write(on-disk) operation faster than disk memory DBs?(Does it use any in-memory middle cache and write queue?)

How to store and retrieve a tree?

Hi,
I have items that are linked in a hierarchical structure. Any item can have zero or more parents(or be a parent to zero or more children).. ie. one-to-many relationships. I need to be able retrieve all parents for a child item up to the upper most item(s).

How should I set this into Bunt and how would I go about retrieving the data?

--
I asked because of the support of rtrees but I think this db is not graph-friendly.

What happens if out of disk? what happens if out of memory?

if disk backed,
What happens if out of disk?
if memory only,
what happens if out of memory?