Advice? Slow JSON unmarshalling because of string -> []bytes about buntdb HOT 11 CLOSED

For reference, I've tried converting to bytes like this, but I get a panic: runtime error: slice bounds out of range at some point in the loop:

func stringToBytes(s *string) []byte {
	return *(*[]byte)(unsafe.Pointer(s))
}

from buntdb.

I think only solution is fork or make PR with some changes to work buntdb with slice of bytes than raw strings. Cause internally it use anyway bytes as u can see here https://github.com/tidwall/buntdb/blob/master/buntdb.go#L1218

from buntdb.

Oh yeah - you're right. I wonder what the motivation behind having string in the api was. Presumably that was for a good reason.

from buntdb.

I wonder what the motivation behind having string in the api was. Presumably that was for a good reason.

Strings are used instead of bytes because the database is in-memory and strings avoids any chance of accidental mutations to the base values. For example, with []byte values, it could be possible to call val := tx.Get("mykey") and then change the byte at some arbitrary position and break the ordering of the indexing.

This is less of an issue with on-disk databases because Get operations commonly retrieve from disk and load into memory for a short period of time. Changes to the memory would likely not corrupt the value in that persists on disk.

Your stringToBytes is casting types of two different sizes. string is 2 words and []bytes is 3 words. This may work better:

func stringToBytes(s string) []byte {
	return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
		(*reflect.StringHeader)(unsafe.Pointer(&s)).Data,
		len(s), len(s),
	}))
}

from buntdb.

Thanks so much. Could I be cheeky and ask for the correct function for the inverse operation - this is a little too low level for me!!

from buntdb.

Just swap out the StringHeader/SliceHeader and string/[]byte. Also a StringHeader takes 2 words which is the Data and Len fields, while the SliceHeader has Data, Len, and Cap.

func bytesToString(b []byte) string {
    return *(*string)(unsafe.Pointer(&reflect.StringHeader{
        (*reflect.SliceHeader)(unsafe.Pointer(&b)).Data, len(b),
    }))
}

from buntdb.

Superb - looking forward to plugging this improved conversion into the benchmarks. I bet it will improve the iteration speed massively.

from buntdb.

Hello,

@jpincas you have a best result with the stringToBytes?

from buntdb.

Actually it didn't seem to make a whole lot of difference. I think it might be the case that Go optimises this under the hood.

from buntdb.

Ok interesting, thank you for your reply 👍

from buntdb.

I’m closing this issue for now. Feel free to reopen if you find a specific issue that needs addressing. Thanks.

from buntdb.