couchbase / goforestdb Goto Github PK

View Code? Open in Web Editor NEW

This project forked from deepkaran/goforestdb

37.0 38.0 6.0 93 KB

Go bindings for ForestDB

License: Apache License 2.0

Go 99.91% C 0.09%

goforestdb's Introduction

goforestdb

Go bindings for ForestDB

Building

Obtain and build forestdb: https://github.com/couchbaselabs/forestdb (run make install to install the library)
Install header files to system location
On Ubuntu 14.04: cd <forestdb_project_dir> && mkdir /usr/local/include/libforestdb && cp include/libforestdb/* /usr/local/include/libforestdb
go get -u -v -t github.com/couchbase/goforestdb

Documentation

See godocs

Sample usage (without proper error handling):

// Open a database
db, _ := Open("test", nil)

// Close it properly when we're done
defer db.Close()

// Store the document
doc, _ := NewDoc([]byte("key"), nil, []byte("value"))
defer doc.Close()
db.Set(doc)

// Lookup the document
doc2, _ := NewDoc([]byte("key"), nil, nil)
defer doc2.Close()
db.Get(doc2)

// Delete the document
doc3, _ := NewDoc([]byte("key"), nil, nil)
defer doc3.Close()
db.Delete(doc3)

goforestdb's People

Contributors

Stargazers

Watchers

Forkers

t3rm1n4l dim pradeepchhetri developgo standardgalactic ajunlonglive

goforestdb's Issues

expose new method to seek iterator

see couchbase/forestdb@6bb679c

make custom comparators in go easier

Registering custom comparators requires passing a in pointer to a function. In cgo this must be an actual c function. It could be a gateway function which in-turn calls go code.

See https://code.google.com/p/go-wiki/wiki/cgo#Function_pointer_callbacks

Right now I've given the Config object the ability to take an unsafe.Pointer. This should allow users to write their own function (in C, possibly calling go) and register it themselves.

I'd love to have our own more generic gateway function, but I think in order to do this we would need a void *context object in the function signature, which we could use to keep track of who was calling us, and which go function did they want us to invoke.

add methods to get/set configuration struct

Currently all you can do is create the default config

Cannot build after OSX Yosemite upgrade

I just upgraded to OSX Yosemite (yeah I know, I was waiting for them to fix all the bugs), and now I'm having issues building goforestdb. In order to build, I had to do the following workarounds:

In my .bash_profile, export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include/. It seems that by default /usr/local/include is no longer in the default header search path. (or is my system messed up somehow?)
In all .go files that had an #cgo LDFLAGS directive, replace with: //#cgo LDFLAGS: -L/usr/local/lib -lforestdb to explicitly add /usr/local/lib into the library search path.

If someone else is able to build fine on OSX Yosemite, then just close this bug and I'll try to figure out how my system is messed up. Any pointers welcome.

Btw, forestdb itself still builds fine.

(I wonder if I just need to reinstall or rebuild go from source?)

error: unexpected signal during runtime execution

Any thoughts on following error? ( I'm using go1.5.1 darwin/amd64 on OSX 10.10.5 ( Yosemite ) with latest ForestDB )

package main

import (
    "fmt"

    "github.com/couchbase/goforestdb"
)

func main() {
    fmt.Println("test")

    // Open a database
    db, _ := forestdb.Open("test", nil)

    // Close it properly when we're done
    defer db.Close()
}

$ go run main.go 
test
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x0 pc=0x0]

runtime stack:
runtime.throw(0x414c320, 0x2a)
  /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.sigpanic()
  /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x4001d40, 0xc82004dc70, 0xc800000000)
  /usr/local/go/src/runtime/cgocall.go:120 +0x11b fp=0xc82004dc50 sp=0xc82004dc20
github.com/couchbase/goforestdb._Cfunc_fdb_get_default_config(0x100004000008, 0x8000000, 0x1000, 0x1, 0x105400001, 0x1e00000104000000, 0x100000, 0xf, 0xc82001c001, 0x1e, ...)
  ??:0 +0x45 fp=0xc82004dc70 sp=0xc82004dc50
github.com/couchbase/goforestdb.DefaultConfig(0xc820062270)
  gocode/src/github.com/couchbase/goforestdb/config.go:201 +0x82 fp=0xc82004de30 sp=0xc82004dc70
github.com/couchbase/goforestdb.Open(0x4123050, 0x4, 0x0, 0x0, 0x0, 0x0)
  gocode/src/github.com/couchbase/goforestdb/file.go:30 +0x51 fp=0xc82004ded0 sp=0xc82004de30
main.main()
  main.go:13 +0xf2 fp=0xc82004df50 sp=0xc82004ded0
runtime.main()
  /usr/local/go/src/runtime/proc.go:111 +0x2b0 fp=0xc82004dfa0 sp=0xc82004df50
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82004dfa8 sp=0xc82004dfa0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1
exit status 2

goforestdb tests are okay;

go test -v .
=== RUN   TestSnapshotAndRollback
--- PASS: TestSnapshotAndRollback (0.00s)
=== RUN   TestForestDBCrud
--- PASS: TestForestDBCrud (0.00s)
=== RUN   TestForestDBCompact
--- PASS: TestForestDBCompact (0.02s)
=== RUN   TestForestDBCompactUpto
--- PASS: TestForestDBCompactUpto (0.00s)
=== RUN   TestForestDBConcurrent
--- PASS: TestForestDBConcurrent (4.59s)
=== RUN   TestForestDBIterator
--- PASS: TestForestDBIterator (0.00s)
=== RUN   TestForestDBIteratorSeq
--- PASS: TestForestDBIteratorSeq (0.00s)
=== RUN   TestForestDBIteratorSeek
--- PASS: TestForestDBIteratorSeek (0.00s)
=== RUN   TestForestDBIteratorPrev
--- PASS: TestForestDBIteratorPrev (0.00s)
=== RUN   TestForestDBIteratorOnSnapshot
--- PASS: TestForestDBIteratorOnSnapshot (0.00s)
=== RUN   TestForestDBIteratorPreAlloc
--- PASS: TestForestDBIteratorPreAlloc (0.00s)
=== RUN   TestForestDBIteratorBug
--- PASS: TestForestDBIteratorBug (0.00s)
=== RUN   TestForestDBKVCrud
--- PASS: TestForestDBKVCrud (0.00s)
=== RUN   TestForestDBKVBatch
--- PASS: TestForestDBKVBatch (0.00s)
=== RUN   TestPool
--- PASS: TestPool (0.00s)
=== RUN   TestSnapshotMarkersSingleKvs
--- PASS: TestSnapshotMarkersSingleKvs (0.00s)
=== RUN   TestSnapshotMarkersMultiKvs
--- PASS: TestSnapshotMarkersMultiKvs (0.00s)
=== RUN   TestTx
--- PASS: TestTx (0.00s)
PASS
ok      github.com/couchbase/goforestdb 4.638s

add tests for commit/snapshot/rollback

Unexpected behavior when setting a doc body to nil

When I set a doc body to nil and the retrieve the doc, it's returning an empty slice (and I was expecting it to return a nil doc body)

Steps to reproduce:

Create new doc
Update doc to set doc body to nil
Get doc via kvstore.GetKV()
Expected: should return nil. Actual: empty slice is returned

Test case

func TestSetDocBodyToNil(t *testing.T) {

    rawBucket, tempDir := GetTestBucket()

    defer os.RemoveAll(tempDir)
    defer CloseBucket(rawBucket)

    bucket := rawBucket.(*forestdbBucket)

    // set "foo" key to value "bar"
    err := bucket.kvstore.SetKV([]byte("foo"), []byte("bar"))
    assert.True(t, err == nil)

    err = bucket.db.Commit(forestdb.COMMIT_NORMAL)
    assert.True(t, err == nil)

    // now set "foo" key to nil
    doc, err := forestdb.NewDoc([]byte("foo"), nil, nil)
    assert.True(t, err == nil)

    defer doc.Close()
    err = bucket.kvstore.Set(doc)
    assert.True(t, err == nil)

    err = bucket.db.Commit(forestdb.COMMIT_NORMAL)
    assert.True(t, err == nil)

    // get the value of "foo" key
    docBody, err := bucket.kvstore.GetKV([]byte("foo"))
    assert.True(t, err == nil)
    assert.True(t, docBody == nil) // fails, because it's an empty slice

}

add tests for sequence iteration

memory leak?

I am not sure if it's the C code or the go wrapper or maybe something I misunderstand altogether, but the following piece of code is an easy way to consume all available memory of your machine:

package main

import (
    "encoding/binary"
    "log"

    fdb "github.com/couchbaselabs/goforestdb"
)

func abortOn(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

func main() {
    db, err := fdb.Open("/tmp/leak.fdb", fdb.DefaultConfig())
    abortOn(err)

    for i := 0; i < 100000000; i++ {
        key := make([]byte, 8)
        binary.BigEndian.PutUint64(key, uint64(i))
        err := db.SetKV(key, []byte("BOGUSDATA"))
        abortOn(err)

        if (i+1)%1000000 == 0 {
            log.Printf("written %d keys", i+1)
        }
    }

}

It looks like a memory leak to me. I tried both, the db.Set and the db.SetKV methods (obviously closing the documents) and they both produce the same pattern of uncontrolled memory consumption growth.

Any ideas? Thanks!

Getting better stack traces?

I somehow managed to cause a panic (I'm still trying to dig into the root cause), and I'm seeing a stack trace:

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x15 pc=0x4ea78bb]

runtime stack:
runtime.gothrow(0x4847e70, 0x2a)
    /usr/local/go/src/runtime/panic.go:503 +0x8e
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:14 +0x5e

goroutine 7 [syscall, locked to thread]:
runtime.cgocall_errno(0x4002df0, 0xc20819b900, 0xc200000000)
    /usr/local/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc20819b8e0 sp=0xc20819b8b8
github.com/couchbaselabs/goforestdb._Cfunc_fdb_kvs_close(0x5500090, 0x0)
    /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/goforestdb/:251 +0x43 fp=0xc20819b900 sp=0xc20819b8e0
github.com/couchbaselabs/goforestdb.(*KVStore).Close(0xc20802bb10, 0x0, 0x0)
    /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/goforestdb/forestdb.go:26 +0x135 fp=0xc20819b978 sp=0xc20819b900
github.com/couchbaselabs/forestdb-bucket.(*forestdbBucket).Close(0xc208092480)
    /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/forestdb-bucket/forestdb_bucket.go:470 +0x71 fp=0xc20819b998 sp=0xc20819b978

it's a bit cryptic, and I was wondering if there was a way to get a better error message.

Panic when read/written concurrently

Here's a test script I've been using:

package main

import (
    "bytes"
    "encoding/binary"
    "fmt"
    "log"
    "os"
    "sync"

    fdb "github.com/couchbaselabs/goforestdb"
)

var testFile = "/tmp/test.fdb"
var testValue = bytes.Repeat([]byte{120}, 256)
var total = int(1e5)

func abortOn(err error) {
    if err != nil {
        log.Fatal(err.Error())
    }
}

func main() {
    os.RemoveAll(testFile)

    config := fdb.DefaultConfig()
    config.SetWalFlushBeforeCommit(true)

    wg := sync.WaitGroup{}
    db, err := fdb.Open(testFile, config)
    abortOn(err)
    defer db.Close()

    abortOn(write(db, 0))
    abortOn(read(db, 0))

    wg.Add(1)
    go func() {
        defer wg.Done()
        abortOn(write(db, 1))
    }()
    wg.Add(1)
    go func() {
        defer wg.Done()
        abortOn(read(db, 0))
    }()
    wg.Wait()
}

func write(db *fdb.Database, base int) error {
    fmt.Println("* writing")
    for n := 0; n < total; n++ {
        key := make([]byte, 4)
        binary.BigEndian.PutUint32(key, uint32(base*total+n))
        if err := db.SetKV(key, testValue); err != nil {
            return err
        }
    }

    return db.Commit(fdb.COMMIT_NORMAL)
}

func read(db *fdb.Database, base int) error {
    fmt.Println("* reading")
    for n := 0; n < total; n++ {
        key := make([]byte, 4)
        binary.BigEndian.PutUint32(key, uint32(base*total+n))
        if _, err := db.GetKV(key); err != nil {
            return err
        }
    }
    return nil
}

Causes:

$ go run fdb-bug.go 
* writing
* reading
* writing
* reading
*** Error in `/tmp/go-build726603444/command-line-arguments/_obj/exe/fdb-bug': double free or corruption (fasttop): 0x00000000097f6e00 ***
SIGABRT: abort
PC=0x7f1172a0ad27
signal arrived during cgo execution

goroutine 21 [syscall]:
runtime.cgocall(0x402410, 0x7f1173239e08)
  $GOROOT/src/pkg/runtime/cgocall.c:143 +0xe5 fp=0x7f1173239df0 sp=0x7f1173239da8
github.com/couchbaselabs/goforestdb._Cfunc_fdb_get_kv(0x13b62c0, 0xc208073040, 0x4, 0xc20803c758, 0xc208073048, 0x1)
  github.com/couchbaselabs/goforestdb/_obj/_cgo_defun.c:179 +0x31 fp=0x7f1173239e08 sp=0x7f1173239df0
github.com/couchbaselabs/goforestdb.(*Database).GetKV(0xc20803c028, 0xc208073040, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
  $GOPATH/src/github.com/couchbaselabs/goforestdb/kv.go:32 +0xf2 fp=0x7f1173239e88 sp=0x7f1173239e08
main.read(0xc20803c028, 0x0, 0x0, 0x0)
  fdb-bug.go:68 +0x1ee fp=0x7f1173239f68 sp=0x7f1173239e88
main.func·002()
  fdb-bug.go:45 +0x58 fp=0x7f1173239fa8 sp=0x7f1173239f68
runtime.goexit()
  $GOROOT/src/pkg/runtime/proc.c:1445 fp=0x7f1173239fb0 sp=0x7f1173239fa8
created by main.main
  fdb-bug.go:46 +0x23d

goroutine 16 [semacquire]:
sync.runtime_Semacquire(0xc208096e50)
  $GOROOT/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0xc20805a000)
  $GOROOT/src/pkg/sync/waitgroup.go:129 +0x14b
main.main()
  fdb-bug.go:47 +0x24d

goroutine 19 [finalizer wait]:
runtime.park(0x4153f0, 0x77bbd8, 0x770de9)
  $GOROOT/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x77bbd8, 0x770de9)
  $GOROOT/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
  $GOROOT/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
  $GOROOT/src/pkg/runtime/proc.c:1445

goroutine 17 [syscall]:
runtime.goexit()
  $GOROOT/src/pkg/runtime/proc.c:1445

goroutine 20 [syscall]:
github.com/couchbaselabs/goforestdb._Cfunc_fdb_set_kv(0x13b62c0, 0xc208073020, 0x4, 0xc208040000, 0x100, 0x425de3)
  github.com/couchbaselabs/goforestdb/_obj/_cgo_defun.c:305 +0x31
github.com/couchbaselabs/goforestdb.(*Database).SetKV(0xc20803c028, 0xc208073020, 0x4, 0x4, 0xc208040000, 0x100, 0x100, 0x0, 0x0)
  $GOPATH/src/github.com/couchbaselabs/goforestdb/kv.go:58 +0xa4
main.write(0xc20803c028, 0x1, 0x0, 0x0)
  fdb-bug.go:55 +0x215
main.func·001()
  fdb-bug.go:40 +0x58
created by main.main
  fdb-bug.go:41 +0x1f0

rax     0x0
rbx     0x8b
rcx     0xffffffffffffffff
rdx     0x6
rdi     0x74ef
rsi     0x74f2
rbp     0x7f1170e9ebe0
rsp     0x7f1170e9e848
r8      0x3030653666373930
r9      0x6f6974707572726f
r10     0x8
r11     0x202
r12     0x7f1170e9e9f0
r13     0x7
r14     0x8b
r15     0x7
rip     0x7f1172a0ad27
rflags  0x202
cs      0x33
fs      0x0
gs      0x0
exit status 2
godep: go exit status 1