mercari / datastore Goto Github PK
View Code? Open in Web Editor NEW(AE|Cloud) Datastore Wrapper
License: MIT License
(AE|Cloud) Datastore Wrapper
License: MIT License
Hi
I'm experiencing failed on go get. Following is the command which I ran and its result.
$ go get go.mercari.io/datastore
package go.mercari.io/datastore: unrecognized import path "go.mercari.io/datastore" (https fetch: Get /datastore?go-get=1: stopped after 10 redirects)
My environments:
In current implementation of dsmiddleware/rpcretry
,
code = Aborted desc = too much contention on these datastore entities. please try again. entity group key: ...
is retried and repeat
code = InvalidArgument desc = The referenced transaction has expired or is no longer valid.
They should not be retried.
golang/appengine@9d8544a...b9aad5d
(q *Query) BatchSize(size int) *Query
type Product struct {
Price *int
}
This feature convenient when outputting JSON
Attempting to receive data in a struct with GetMulti will fail.
datastore: dst has invalid type
type Hoge struct {
ID string `datastore:"-"`
Body string
}
var rvs []*Hoge
if err := ds.GetMulti(ctx, keys, &rvs); err != nil {
t.Fatal(err)
}
try code https://github.com/sinmetal/dstest/blob/master/ds_test.go#L47
I want to write below code.
type EventCource struct {
ParentID int64 `boom:"parent,Event"
ID int64
Name string
}
Thank you for awesome package.
I tried to get embedded struct but failed. Show the cases below (ignore error).
The difference of the cases is Bar
is struct
or pointer of struct
.
Is this a bug?
The embedded Bar
is null.
type Foo struct {
_kind string `boom:"kind"`
ID string `boom:"id"`
Bar *Bar `datastore:",flatten"`
}
type Bar struct {
Name string
}
func get() {
client, _ := aedatastore.FromContext(ctx)
bm := boom.FromClient(ctx, client)
foo := Foo{ID:"1"}
bm.Get(&foo)
b, _ := json.Marshal(foo)
fmt.Print(b)
// => {"ID":"1","Bar":null}
}
type Foo struct {
_kind string `boom:"kind"`
ID string `boom:"id"`
Bar Bar `datastore:",flatten"`
}
type Bar struct {
Name string
}
func get() {
client, _ := aedatastore.FromContext(ctx)
bm := boom.FromClient(ctx, client)
foo := Foo{ID:"1"}
bm.Get(&foo)
b, _ := json.Marshal(foo)
fmt.Print(b)
// => {"ID":"1","Bar":{"Name":"bar1"}}
}
We get the entity by batch, But if it is ErrNoSuchEntity when We go error by *datastore.Batch#Exec.
I want to suppress it.
In Firestore Datastore mode, 500 write limits are no longer applied.
https://cloud.google.com/datastore/docs/release-notes?hl=en#March_29_2023
Firestore no longer limits the number of writes that can be passed to a Commit operation or performed in a transaction. Previously, the limit was 500. Limits for request size and the transaction time limit still apply.
current implementation doesn't have context argument in RunInTransaction callback function.
This makes difficult about emit beautiful trace log.
I added a rediscache middleware in front of the cloud datastore and I tried to write and read a struct with an embedded struct in it. The program successfully writes the payload in both the cache and datastore. But it panics (https://github.com/mercari/datastore/blob/master/clouddatastore/key.go#L29) when trying to write the data into the redis cahce when encountering a cache miss.
After some debugging, I think the issue was that the library wraps an Entity
around the embedded struct with a nil
key. It worked on write because the nil
key was catched at https://github.com/mercari/datastore/blob/master/clouddatastore/convert.go#L10 since the key was a {*datastore.key}{nil}
, but on read, the key was a {*datastore.keyImpl}{nil}
so the nil checking returns false and keep proceeding to the panic point.
type Item struct {
Parent datastore.Key `boom:"parent"`
ID string `boom:"id"`
Contents []string `json:"contents"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
SchemaVersion int `json:"-"`
}
func hoge() {
bm := boom.FromClient(ctx, client)
item := &Item{
Parent: nil,
ID: form.ID,
Contents: form.Contents,
}
bm.Put(item) // panic !!!
}
interface conversion: interface is nil, not datastore.Key
AEDatastore+AEMemcacheからCloudDatastore+AEMemcacheに移行しようとして気がついた。
なにかをGetして、Txに突入してPutし、Txから抜けてGetすると古いEntityが取れる… という問題がある。
https://github.com/mercari/datastore/blob/master/dsmiddleware/storagecache/storagecache.go#L313
if err == nil , sErr != nil
this line occur segv.
in
, not-in
, and !=
query operators are new features introduced in cloud.google.com/go/datastore
v1.8.0
.
https://github.com/googleapis/google-cloud-go/blob/main/datastore/CHANGES.md#180-2022-06-21
Query.Filter
is deprecated.
https://pkg.go.dev/cloud.google.com/go/datastore#Query.Filter
I want to use the query operators via boom
.
toOriginalPropertyList
causes invalid entity type
error with nested struct data types.
Here's a test code to reproduce.
Thanks.
Documentation:
Cloud Firestore Datastore mode
https://cloud.google.com/datastore/docs/concepts/transactions?hl=en#read-only_transactions
Cloud Datastore
https://cloud.google.com/datastore/docs/concepts/cloud-datastore-transactions?hl=en#uses_for_transactions
Library support(App Engine Datastore and Cloud Datastore):
https://pkg.go.dev/google.golang.org/[email protected]/datastore#TransactionOptions
https://pkg.go.dev/cloud.google.com/go/[email protected]#TransactionOption
dst has invalid type
is too ambiguous.
You cannot see what dst
means.
So we should replace dst
to more meaninful description.
When using redis and cloud.google.com/go/datastore which is after V1.4.0, if the time.Time type member of the struct is uninitialized, the struct cannot be put to datastore.
Without redis, put work well.
Using cloud.google.com/go/datastore V1.3.0, put work well.
(Few days ago I updated go.mod, then my unittest failed. datastrore V1.3.0 is OK, V1.4.0 or later is fail.)
Below is my Test code.
package main
import (
orgdatastore "cloud.google.com/go/datastore"
"context"
"fmt"
"github.com/gomodule/redigo/redis"
"go.mercari.io/datastore"
"go.mercari.io/datastore/clouddatastore"
"go.mercari.io/datastore/dsmiddleware/rediscache"
"log"
"net"
"time"
)
type Entity struct {
Tp time.Time //About cloud.google.com/go/datastore, in v1.4.0 or later, redis will result in an error unless it is explicitly initialized. No error occurs to v1.3.0
P int
}
var ename string
var pname string
var eid string
func main() {
loc, err := time.LoadLocation("Asia/Tokyo")
if err != nil {
loc = time.FixedZone("Asia/Tokyo", 9*60*60)
}
time.Local = loc
ename = "Entity"
pname = "my-project"
eid = "eid"
//pattern_originalDatastore()
pattern_mercariDatastore(true, false)
pattern_mercariDatastore(true, true)
}
func pattern_mercariDatastore(withRedis, doInitTimeMember bool) {
dbkind := fmt.Sprintf("mercari_redis_%v_timeInit_%v", withRedis, doInitTimeMember)
fail := false
c := context.Background()
client := getMercariClient(c, withRedis)
defer client.Close()
k := client.NameKey(ename, eid, nil)
e := new(Entity)
if err := client.Get(c, k, e); err != nil {
//This is first get, then err shold not be nil.
//log.Printf("%s Get Error %v\n", dbkind, err)
} else {
log.Printf("%s initialize miss\n", dbkind)
fail = true
}
e.P = 50
if doInitTimeMember { //cloud.google.com/go/datastore in v1.4.0 or later, this is necessary.
e.Tp = time.Now()
}
if _, err := client.Put(c, k, e); err != nil {
log.Printf("%s Put Error %v\n", dbkind, err)
fail = true
}
e2 := new(Entity)
if err := client.Get(c, k, e2); err != nil {
log.Printf("%s Put Fail %v\n", dbkind, err)
fail = true
}
if e2.P != 50 {
log.Printf("%s Put Fail 50, now %d\n", dbkind, e2.P)
fail = true
}
e2.P = 100
client.Put(c, k, e2)
e3 := new(Entity)
if err := client.Get(c, k, e3); err != nil {
log.Printf("%s Put Fail %v\n", dbkind, err)
fail = true
}
if e3.P != 100 {
log.Printf("%s Put100 Fail \n", dbkind)
fail = true
}
client.Delete(c, k)
if fail {
log.Printf("----- %s Fail -----", dbkind)
} else {
log.Printf("----- %s OK -----", dbkind)
}
}
func pattern_originalDatastore() {
dbkind := "original"
fail := false
c := context.Background()
client, _ := orgdatastore.NewClient(c, pname)
defer client.Close()
k := orgdatastore.NameKey(ename, eid, nil)
e := new(Entity)
if err := client.Get(c, k, e); err != nil {
//This is first get, then err shold not be nil.
//log.Printf("%s Get Error %v\n", dbkind, err)
} else {
log.Printf("%s initialize miss\n", dbkind)
fail = true
}
e.P = 50
if _, err := client.Put(c, k, e); err != nil {
log.Printf("%s Put Error %v\n", dbkind, err)
fail = true
}
e2 := new(Entity)
if err := client.Get(c, k, e2); err != nil {
log.Printf("%s Put Fail %v\n", dbkind, err)
fail = true
}
if e2.P != 50 {
log.Printf("%s Put Fail 50, now %d\n", dbkind, e2.P)
fail = true
}
e2.P = 100
client.Put(c, k, e2)
e3 := new(Entity)
if err := client.Get(c, k, e3); err != nil {
log.Printf("%s Put Fail %v\n", dbkind, err)
fail = true
}
if e3.P != 100 {
log.Printf("%s Put100 Fail \n", dbkind)
fail = true
}
client.Delete(c, k)
if fail {
log.Printf("----- %s Fail -----", dbkind)
} else {
log.Printf("----- %s OK -----", dbkind)
}
}
func getMercariClient(c context.Context, withredis bool) datastore.Client {
orgClient, _ := orgdatastore.NewClient(c, "my-project")
client, _ := clouddatastore.FromClient(c, orgClient)
dial, err := net.Dial("tcp", "localhost:6379")
if err != nil {
log.Printf("getMercariClient %v", err)
}
if withredis {
conn := redis.NewConn(dial, 100*time.Millisecond, 100*time.Millisecond)
//defer conn.Close()
mw := rediscache.New(conn,
rediscache.WithLogger(func(ctx context.Context, format string, args ...interface{}) {
log.Printf(format, args...)
}))
rediscachemw := mw
client.AppendMiddleware(rediscachemw)
}
return client
}
should returns error. don't blocking anything.
Commit, Rollback, RunInTransaction.
*datastore.Key
to datastore.Key
*datastore.Query
to datastore.Query
*datastore.Iterator
to datastore.Iterator
datastore.TransactionOptions
does not supportederr.(appengine.MultiError)
to err.(datastore.MultiError)
carefully.
appengine.BlobKey
google.golang.org/appengine/datastore.Done
to google.golang.org/api/iterator.Done
key.IntID()
to key.ID()
key.StringID()
to key.Name()
*datastore.Commit
to datastore.Commit
*goon.Goon
to *boom.Boom
goon.FromContext(ctx)
to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds)
Boom does not provide a transaction context to use taskqueue with transaction.
If we want to use taskqueue with transaction, we need transaction context.
Currently, mercari/datastore has aedatastore.TransactionContext(t datastore.Transaction) context.Context
. but boom.Transaction does not implement datastore.Transaction and boom.Transaction does not provide getter for datastore.Transaction.
So we can't use taskqueue with transaction when we use boom.
https://cloud.google.com/datastore/docs/manage-databases
You can create multiple Firestore databases per project. You can use multiple databases to set up production and testing environments, to isolate customer data, and for data regionalization.
CHANGES.md
https://github.com/googleapis/google-cloud-go/blob/main/datastore/CHANGES.md#1130-2023-07-26
Why isn't this library supporting DistinctOn
method?
// Query represents a datastore query.
type Query interface {
Ancestor(ancestor Key) Query
EventualConsistency() Query
Namespace(ns string) Query
Transaction(t Transaction) Query
Filter(filterStr string, value interface{}) Query
Order(fieldName string) Query
Project(fieldNames ...string) Query
Distinct() Query
// NOT IMPLEMENTED ON APPENGINE DistinctOn(fieldNames ...string) *Query
KeysOnly() Query
Limit(limit int) Query
Offset(offset int) Query
Start(c Cursor) Query
End(c Cursor) Query
Dump() *QueryDump
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.