maddyblue / goon Goto Github PK
View Code? Open in Web Editor NEWAutocaching interface to the app engine datastore for Go
Home Page: http://godoc.org/github.com/mjibson/goon
License: ISC License
Autocaching interface to the app engine datastore for Go
Home Page: http://godoc.org/github.com/mjibson/goon
License: ISC License
Add support for applications that use the google.golang.org/appengine packages. These packages work on App Engine Classic and managed VMs.
There are a couple of ways to do this. One is to fork Goon and replace all uses of appengine/* with google.golang.org/appengine/* and appengine.Context with the x/net Context. A second option is to write a shim layer above Goon.
I found that when we fetch slice values from cache, goon appends the values into existing slice. Instead of appending values, goon should replace the slice.
Test to reproduce:
yuichi1004@ab412e6
Test Result:
$ go test -v -run Migration ./
=== RUN TestMigration
INFO 2018-04-28 09:43:06,511 devappserver2.py:105] Skipping SDK update check.
WARNING 2018-04-28 09:43:06,511 devappserver2.py:121] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
WARNING 2018-04-28 09:43:06,566 simple_search_stub.py:1196] Could not read search indexes from /var/folders/5j/2bmf8d2d3xgb9lmttlptxvm8p49fjs/T/appengine.testapp.01099286/search_indexes
INFO 2018-04-28 09:43:06,568 api_server.py:308] Starting API server at: http://localhost:51010
INFO 2018-04-28 09:43:06,638 dispatcher.py:255] Starting module "default" running at: http://localhost:51011
INFO 2018-04-28 09:43:06,640 admin_server.py:146] Starting admin server at: http://localhost:51013
--- FAIL: TestMigration (4.48s)
goon_test.go:994: MC-false > Expected 1 car! Got: 2
FAIL
exit status 1
FAIL github.com/yuichi1004/goon 4.501s
Note that we have extra cars on the above test.
https://github.com/golang/go/wiki/Modules#modules
This repository has no tag.
I think we should add a semver tag for go modules.
We can use only commit hash designate now, It is inconvenient.
Hi,
Am trying to store a struct into datastore like below,
type Config struct {
Id int64 datastore:"id,noindex" goon:"id" json:"id"
Config map[string]float64 datastore:"config,noindex" json:"config,omitempty"
}
I have also impl PropertyLoaderSaver interface to convert the map into json string before saving to datastore, everything work fine with datastore, while fetching the json string is convert to back to map,
But when this entity is fetched from memcache am getting following error from goon.go file,
goon - goon.go:475 - goon: Failed to decode field config - extra data in buffer
Any help on how to fix this issue?
Note: Every First time fetch from datastore it works fine, only subsequent fetch (while it fetch from memcache) its throwing this error
For now I've edited goon.go:44:
func NewGoon(r *http.Request) *Goon {
return FromContext(appstats.NewContext(r))
}
I'm really enjoying goon, but something I think it needs is a .PutMemoryOnly (which puts to cache and memcache, excluding datastore).
I see a big need for this since I have applications doing a lot of .Puts to maintain a last_seen timestamp (which I'm more than happy to only write to dbstore every 30/60 seconds - However I require it to still be available "if" possible via cache or memcache).
Currently my only option is to maintain my own cache separately, so I'm caching the same thing twice :/
Do you think it would make sense to add laterGet and laterPut using "appengine/delay" ?
laterGet can also be useful if you want to pull something from memcache you might need later in another request.
PR #87 redesigned cache key generation to fix some issues, however in the process it removed the ability for people to determine the final cache key. This is something that people actually use, as evidenced by this comment. Thus the ability to know the final cache key should be reintroduced.
The conundrum is that we either have to surface it with a new function name, and thus breaking compatibility with existing code that expects MemcacheKey
to return the final key or we surface the whole thing via MemcacheKey
and not provide automatic bug fixes to people who override the function.
The following test fails on the last inequality check with:
demo_test.go:54: goonPutGet.Value should be 15 but is 0
package goon_test
import (
"testing"
"appengine/aetest"
"appengine/datastore"
"github.com/mjibson/goon"
)
type PutGet struct {
ID int64 `datastore:"-" goon:"id"`
Value int32
}
func TestPutGet(t *testing.T) {
c, err := aetest.NewContext(nil)
if err != nil {
t.Fatal(err.Error())
}
g := goon.FromContext(c)
key, err := g.Put(&PutGet{ID: 12, Value: 15})
if err != nil {
t.Fatal(err.Error())
}
if key.IntID() != 12 {
t.Fatal("ID should be 12 but is", key.IntID())
}
// Datastore Get
dsPutGet := &PutGet{}
err = datastore.Get(c,
datastore.NewKey(c, "PutGet", "", 12, nil), dsPutGet)
if err != nil {
t.Fatal(err.Error())
}
if dsPutGet.Value != 15 {
t.Fatal("dsPutGet.Value should be 15 but is",
dsPutGet.Value)
}
// Goon Get
goonPutGet := &PutGet{ID: 12}
err = g.Get(goonPutGet)
if err != nil {
t.Fatal(err.Error())
}
if goonPutGet.ID != 12 {
t.Fatal("goonPutGet.ID should be 12 but is", goonPutGet.ID)
}
if goonPutGet.Value != 15 {
t.Fatal("goonPutGet.Value should be 15 but is",
goonPutGet.Value)
}
}
Looking at the code it appears that the in memory cached value of PutGet{ID:12} is not being transferred to goonPutGet
. Note that if I create a new Goon
struct for the g.Get(goonPutGet)
everything works fine because the in memory cache is not used.
TestEmbeddedStruct fails on my environment. It seems to be failed to fetch unexported fields following.
2220 type ContainerStruct struct {
2221 Id string `datastore:"-" goon:"id"`
2222 embeddedStructA
2223 embeddedStructB `datastore:"w"`
2224 }
I think private fields are expected to be ignored by datastore API. So this behavior looks as expected to me. Am I missing something?
The test passed if I make those fields public like this.
2220 type ContainerStruct struct {
2221 Id string `datastore:"-" goon:"id"`
2222 EmbeddedStructA
2223 EmbeddedStructB `datastore:"w"`
2224 }
$ goapp test -v . -run TestEmbeddedStruct
2017/05/03 14:33:06 appengine: not running under devappserver2; using some default configuration
=== RUN TestEmbeddedStruct
INFO 2017-05-03 05:33:07,983 devappserver2.py:756] Skipping SDK update check.
WARNING 2017-05-03 05:33:07,984 devappserver2.py:772] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
WARNING 2017-05-03 05:33:08,079 simple_search_stub.py:1146] Could not read search indexes from /var/folders/h0/c79kbs8x6jj8scsd6j22c5hh01hx69/T/appengine.testapp.yuichi.murata/search_indexes
INFO 2017-05-03 05:33:08,084 api_server.py:205] Starting API server at: http://localhost:50322
INFO 2017-05-03 05:33:08,087 dispatcher.py:197] Starting module "default" running at: http://localhost:50323
INFO 2017-05-03 05:33:08,090 admin_server.py:116] Starting admin server at: http://localhost:50324
--- FAIL: TestEmbeddedStruct (4.35s)
goon_test.go:2266: #1 - Expected - 1, got 0
goon_test.go:2269: #1 - Expected - 3, got 0
goon_test.go:2272: #1 - Expected - 4, got 0
goon_test.go:2266: #2 - Expected - 1, got 0
goon_test.go:2269: #2 - Expected - 3, got 0
goon_test.go:2272: #2 - Expected - 4, got 0
FAIL
exit status 1
FAIL github.com/yuichi1004/goon 4.364s
There's a few people looking at this now, and it'd be awesome if someone wanted to implement control variance, as discussed in David Symonds' talk: http://talks.golang.org/2013/highperf.slide#26
I've seen memcache on goread take 100-200ms, both on read and write, which can triple the time of a request. If defaulted to 3ms or 5ms or something, and made configurable, I would be happy.
See #55
When you "put" an incomplete object and the Id is of type string, it's not populated and is instead left blank. This is inherently because the datastore will only generate an integer id.
I guess it would be proper to return an error back to the caller if they're trying to "Put" an incomplete key and their Id is of type String?
Similar to the request at qedus/nds#54
Would it be possible to extend this to work outside of Google App Engine?
Is it safe to add a property to an entity? or should I migrate my data first?
I noticed that goon uses a *http.Request
instead of a context.Context
when creating a *Goon
, which is only used to get a new context.Context
. I'm not clear on the reasoning behind this decision as it breaks from the common Go and AppEngine style of passing in a context.Context as the first argument in a function call. Especially since the vast majority of functions will already have a context.Context
, but it's usually only the handler function which has a *http.Request
to pass in.
I would suggest requiring a context.Context
as the argument instead of the *http.Request
which is currently used. However I'd love to get some feedback on this.
I was thinking, it would be quite useful to add a (writing this off the top of my head)
func (g *Goon) getContext() appengine.Context {
return g.context
}
I keep having to pass the Context and Goon around at the same time (logging, creating queries, accessing memcache directly, etc), and Goon already has the Context already.
Let me know if there are any objections to this, and if not, I'll fire up a pull request.
I am just wondering if there is a case when gob.Register is needed on the following lines:
https://github.com/mjibson/goon/blob/master/entity.go#L31
https://github.com/mjibson/goon/blob/master/entity.go#L41
Hi, I'd like to unit test my code that relies on datastore in a way that does not require the devserver to be started. Is it possible to use goon and test without it using aetest and starting a devserver behind the covers?
Hello,
Thank you a lot for this package. it is always a great help for us.
By the way, I encountered its behavior as this Title, when I try to use the new function IgnoreFieldMismatch
.
a3e1c61#diff-fd6485aca5f3b9fb50a56fbfc9f202b4R414
I think It's better to be improved, If it is not by design.
Thanks again
Based on discussion at http://mattjibson.com/blog/2013/06/26/go-read-open-source-google-reader-clone/#comment-944610909, memcache access is limited to 1000 objects at once, and the memcache library needs to be made aware of this limit.
I don't think this is a goon issue, but instead that Google released a bugged SDK. However it has a serious effect on goon (tests don't work), so I think it's an issue worth keeping track of here.
Basically some tests seem to deadlock, and others throw up a wide variety of threading errors. I have filed a bug report to google.
I have only tested this on my own Windows setup, and I have only seen other reports with similar errors from Windows setups. So it would be interesting to know if it's a Windows specific problem.
@mzimmerman, you run arch, right? Perhaps you could try running the goon tests with Go SDK 1.9.3 a few times and see if they always succeed without errors, like they should and have been succeeding with previous SDK versions.
I think this is so cool project. but, I can't use no license source code.
Just curious. Does it have any meaning such as GO cONtext
or any?
If I try to create an entity in a unit test :
u := &User{Name: "Joe", CName: "joe"}
k, err := n.Put(u)
if err != nil {
t.Fatal(err)
}
Parentless queries will fail because of a consistency issue (parentless puts may not be available right away.)
The problem can be avoided by sleeping for 1 second but the test becomes dependent on the machine load, etc.
// Sleep a bit to wait for the HRD emulation to get out of our way
time.Sleep(1000 * time.Millisecond)
According to this post the proper way to avoid this issue inside a test is to get the entity using its known key, right after.
My subsequent queries work perfectly if I perform a datastore.Get(c,k,u)
first.
However they fail miserably if I use n.Get(u)
even though the key is known and set:
Hi there,
New to goon / appstats & go, came across a issue where the context Goon was using was not that of appstats'. So I was not getting any RPC calls logged, Goons documentation doesn't mention to use FromContext(c) rather than NewGoon(r) (I discovered it seeing the function in the function listing).
Might be easier for newer guys to be aware to use FromContext when using appstats.
Thanks,
Mike :)
I'm happy with the recent changes to the memcache layer. However, the complexity of the serialization code prevented me from reading any of the changes. While the performance benefits are nice, I do not think they are enough to justify this complexity of code. I think it is more important that many people agree that it is good.
I propose that we remove this magic and revert back to the gob version. I definitely want to keep the new features that were added, but I want to understand how they work.
If there is strong disagreement to this proposal, I'd like to discuss it. My goal is to have the code in a state where I can understand it and have motivation to read through all changes. I thought I could ignore that, but goread broke so many times that this is clearly not possible.
Thoughts?
Currently, goon.Get calls methods as below
.. => goon.Get => datastore.Get => goon.putMemcache => ..
or (if it's cached)
.. => goon.Get => memcache.Get => goon.putMemcache => ..
I think this flow has a problem.
PropertyLoadSaver.Load is called in only datastore.Get.
And goon.putMemcache ignores fields which tagged as datastore:"-"
.
Therefore the fields changed by PropertyLoadSaver.Load aren't cached.
The return of goon.Get should be same whether it's cached or not.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.