google / go-cloud Goto Github PK
View Code? Open in Web Editor NEWThe Go Cloud Development Kit (Go CDK): A library and tools for open cloud development in Go.
Home Page: https://gocloud.dev/
License: Apache License 2.0
The Go Cloud Development Kit (Go CDK): A library and tools for open cloud development in Go.
Home Page: https://gocloud.dev/
License: Apache License 2.0
There are a number of places throughout the libraries that would like to report errors that don't really fall into typical call flow, such as during the Wire cleanup function. Typically, these should be logged somewhere, but a library should not log. To give a simple hook point that can be used to collect errors, I propose something like:
// ErrorReporter wraps the ReportError method.
//
// ReportError is called when a "background" error occurs: something that failed,
// but isn't appropriate to pass back up to a caller. It must be safe to call from multiple
// goroutines, and an implementation may retain the error past the return of ReportError.
type ErrorReporter interface {
ReportError(error)
}
This would allow us to use the replay package for integration tests.
For both Delete and NewReader, there's no way to distinguish blob-not-found from a normal error. This is useful for serving 404 instead of 500.
/cc @shantuo
Since we are still in the early stages where we can break the Goose API, it's worth pushing the API to allow a runtime-based injector down the road, if we find we need it. The intent would be to store the arguments, not to actually implement anything inside the marker function package.
AFAICT, the only change needed is to make the first argument to goose.Bind
take in a pointer to an interface, since otherwise the interface type would be lost by the time Bind
starts. The semantics of goose.Value
might be slightly weird, but the restrictions on what can be used in goose.Value
should eliminate most potential problems.
EDIT 2019-07-09 Adding more context to this issue:
Right now, a server using the Go CDK server package can configure the OpenCensus trace exporter, but not the metric exporter.
A new field on *server.Options
to initialize the metrics exporter.
There are existing tests in this package, they should be converted to use RPC replay. RPC replay already exists in the Go Cloud Client libraries.
Looks like paramstore.Watch
doesn't attempt to watch immediately, but rather waits for its wait time and then goes for a look.
A paramstore watcher
should look and return immediately on the first call to Watch
It's currently using a timer for counting between RPC calls, which is less readable than the ticker style used in paramstore.
There's a few tests here, but it would be good to write some HTTP replay tests that exercise the GCS blob API. This can probably borrow structure from the fileblob tests.
There are some example tests that don't build. Somehow they got through Travis which is odd.
$ go test -run=TestWatchReplay runtimevar/runtimeconfigurator/*.go
# command-line-arguments_test
runtimevar/runtimeconfigurator/example_test.go:32:24: undefined: runtimeconfigurator.Dial
runtimevar/runtimeconfigurator/example_test.go:37:41: multiple-value runtimeconfigurator.NewClient() in single-value context
FAIL command-line-arguments [build failed]
Can we scrub the project ID from the RPCs? Maybe the RPCs are just replayed without matching? Got to see how the RPC replay system works.
Blocking #61 .
For post-launch.
Slash paths are more conventional on buckets anyway, and are easier to reason about sanitizing.
We don't really want to commit to the API surface, as it is being used to test the project itself, not for helping others tests their apps.
This came up in trying to write a sample application that proxies blob access. For both GCS and S3, the services support setting a Content-Type
header that can be accessed later. We should extend the blob interface to allow reading and writing of metadata beyond just Size
.
Caused a failure in https://travis-ci.com/google/go-cloud/builds/75051179
Needs to be passed a flag to have it load test files but there's no way to do that with golangci-lint
so let's just drop it completely.
While testing a PR, I noticed that Travis flagged a Wire provider set as an unused variable (source):
samples/guestbook/main.go:269:5: `appSet` is unused (varcheck)
var appSet = wire.NewSet(
^
This is technically correct, but not an actual issue, and it did cause the build to fail, thus blocking submission. Based on recommendation from @cflewis, I'm disabling varcheck
as a temporary workaround, then keeping this open for solving the underlying issue.
The project name is "Go X Cloud". Our repository name and import path should reflect that. Import path: github.com/google/go-x-cloud
.
e.g. read/write/delete a file
EDIT 2019-07-09 to add additional context
If a user uses the Go CDK server
package, they currently have to manually configure sending profiler information to Stackdriver Profiler.
The sdserver
package should have an option to permit profiling information to be sent to Stackdriver Profiler.
The Stackdriver Profiler for Go can be configured to be very low-overhead so as to be turned on by default. We should turn it on at server start using the server.Options.Driver
field.
Instructions for setting up the profiling agent are at https://cloud.google.com/profiler/docs/profiling-go, and the relevant code.
Now that we're using GitHub, let's set up a Travis build. This should likely be running in replay mode for pull requests, and we may want to investigate using the real production projects for master builds.
TestMain is making it kinda gnarly to run the test. Let's remove it and make it easier to modify.
From internal design discussion, goose.Use
seemed to be confusing, so it was decided to change it to goose.Build
.
The current runtimeconfigurator.NewClient
function takes in a varargs client option, which is idiomatic for GCP clients, but is difficult to integrate into Goose. It would be good to include the client in the gcpcloud.Services
set instead of pushing responsibility onto the application.
Do the required reading and be sure this is how we want to go.
Specifically need to figure out how to deal with backlogged work/work for the future. "Future" milestone? Label?
It would be nice to have PR Travis builds only run the tests for any affected packages for the files touched, then leave the master branch build to run all tests. The benefit would be to decrease the presubmit latency, since running less tests also means less dependencies to download. I'm not sure if this has been done before, but it seems like all the information is there in the Go tool and Travis to do subsetting like this.
/cc @cflewis
$ vgo test -short ./blob/s3blob ./runtimevar/paramstore
ok github.com/google/go-cloud/blob/s3blob (cached)
ok github.com/google/go-cloud/runtimevar/paramstore 0.702s
$ git status
On branch master
Your branch is up to date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: blob/s3blob/testdata/blob.yaml
modified: runtimevar/paramstore/testdata/decoder.yaml
modified: runtimevar/paramstore/testdata/watch_change.yaml
modified: runtimevar/paramstore/testdata/watch_initial.yaml
modified: runtimevar/paramstore/testdata/write_read_delete.yaml
no changes added to commit (use "git add" and/or "git commit -a")
Differences appear to be the mu
key and some minor YAML string encoding. I would expect this to be clean on a new checkout.
$ git rev-parse HEAD
189ea15d728cc05309379d7991d126307dda3350
$ vgo list -m
MODULE VERSION
github.com/google/go-cloud -
cloud.google.com/go v0.21.0
contrib.go.opencensus.io/exporter/stackdriver v0.0.0-20180421005815-665cf5131b71
github.com/GoogleCloudPlatform/cloudsql-proxy v0.0.0-20180321230639-1e456b1c68cb
github.com/aws/aws-sdk-go v1.13.20
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
github.com/bgentry/speakeasy v0.0.0-20170417200703-4aabc24848ce
github.com/census-ecosystem/opencensus-go-exporter-aws v0.0.0-20180411051634-41633bc1ff6b
github.com/coreos/bbolt v0.0.0-20171207012708-48ea1b39c25f
github.com/coreos/etcd v0.0.0-20180424190529-fdde8705f5bf
github.com/coreos/go-semver v0.2.0
github.com/coreos/go-systemd v0.0.0-20170731111925-d21964639418
github.com/coreos/pkg v0.0.0-20160727233714-3ac0863d7acf
github.com/cpuguy83/go-md2man v0.0.0-20170603125239-23709d084719
github.com/davecgh/go-spew v1.1.0
github.com/dgrijalva/jwt-go v0.0.0-20160616191556-d2709f9f1f31
github.com/dnaeon/go-vcr v0.0.0-20180504081357-f8a7e8b9c630
github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4
github.com/fsnotify/fsnotify v1.4.7
github.com/ghodss/yaml v0.0.0-20170327235444-0ca9ea5df545
github.com/go-ini/ini v1.25.4
github.com/go-sql-driver/mysql v0.0.0-20180308100310-1a676ac6e4dc
github.com/gogo/protobuf v0.0.0-20171007142547-342cbe0a0415
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903
github.com/golang/protobuf v1.0.0
github.com/google/btree v0.0.0-20161005200959-925471ac9e21
github.com/google/go-cmp v0.2.0
github.com/googleapis/gax-go v0.0.0-20170915024731-317e0006254c
github.com/gorilla/mux v1.6.1
github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c
github.com/grpc-ecosystem/go-grpc-prometheus v0.0.0-20170826090648-0dafe0d496ea
github.com/grpc-ecosystem/grpc-gateway v0.0.0-20171103165559-8cc3a55af3bc
github.com/inconshreveable/mousetrap v0.0.0-20141017200713-76626ae9c91c
github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8
github.com/jonboulle/clockwork v0.0.0-20160615175015-2eee05ed7941
github.com/jtolds/gls v0.0.0-20170503224851-77f18212c9c7
github.com/kr/pty v0.0.0-20170307145309-2c10821df3c3
github.com/mattn/go-runewidth v0.0.0-20170201022620-9e777a8366cc
github.com/matttproud/golang_protobuf_extensions v1.0.0
github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
github.com/openzipkin/zipkin-go v0.1.0
github.com/pmezard/go-difflib v1.0.0
github.com/prometheus/client_golang v0.8.0
github.com/prometheus/client_model v0.0.0-20171117100541-99fa1f4be8e5
github.com/prometheus/common v0.0.0-20180413074202-d0f7cd64bda4
github.com/prometheus/procfs v0.0.0-20180408092902-8b1c2da0d56d
github.com/russross/blackfriday v0.0.0-20170728175326-4048872b16cc
github.com/shurcooL/sanitized_anchor_name v0.0.0-20151028001915-10ef21a441db
github.com/sirupsen/logrus v0.0.0-20170815202055-f006c2ac4710
github.com/smartystreets/assertions v0.0.0-20180301161246-7678a5452ebe
github.com/smartystreets/goconvey v0.0.0-20180222194500-ef6db91d284a
github.com/smartystreets/gunit v0.0.0-20180314194857-6f0d6275bdcd
github.com/soheilhy/cmux v0.0.0-20170814202137-bb79a8346501
github.com/spf13/cobra v0.0.0-20151124153217-1c44ec8d3f15
github.com/spf13/pflag v0.0.0-20170508184408-e57e3eeb33f7
github.com/stretchr/objx v0.0.0-20180106011353-facf9a85c22f
github.com/stretchr/testify v1.2.1
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
github.com/ugorji/go v0.0.0-20171019201919-bdcc60b419d1
github.com/urfave/cli v0.0.0-20160628053056-1efa31f08b93
github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18
go.opencensus.io v0.9.0
go.uber.org/atomic v0.0.0-20171114204401-8474b86a5a6f
go.uber.org/multierr v0.0.0-20170630175437-3c4937480c32
go.uber.org/zap v0.0.0-20170925195302-35aad584952c
golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44
golang.org/x/net v0.0.0-20180502164142-640f4622ab69
golang.org/x/oauth2 v0.0.0-20180402223937-921ae394b943
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
golang.org/x/sys v0.0.0-20180329131831-378d26f46672
golang.org/x/text v0.3.0
golang.org/x/time v0.0.0-20170420181420-c06e80d9300e
golang.org/x/tools v0.0.0-20180314180217-d853e8088c62
google.golang.org/api v0.0.0-20180413000347-fca24fcb4112
google.golang.org/appengine v1.0.0
google.golang.org/genproto v0.0.0-20180409222037-51d0944304c3
google.golang.org/grpc v1.11.3
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
gopkg.in/cheggaaa/pb.v1 v1.0.0-20160328142451-226d21d43a30
gopkg.in/yaml.v2 v2.2.1
In particular, the following packages (omitting the github.com/google/go-cloud
prefix) might be renamed.
aws/awscloud -> aws/cloud
gcp/gcpcloud -> gcp/cloud
blob/fileblob -> blob/file
blob/gcsblob -> blob/gcs
blob/s3blob -> blob/s3
health/sqlhealth -> health/sql
server/sdsserver -> server/sds
server/xrayserver -> server/xray
This rename would necessitate named imports when a user imports both aws/cloud
and gcp/cloud
for example. That seems like a reasonable tradeoff for a better package name, though.
Cf. Package Names.
Post launch. Possibly "post 1.0" of the APIs.
Generic code should import golang.org/x/cloud
.
Cloud-specific code (e.g. code written to be fed into Wire) should import additional cloud-specific repositories (to be determined) to make use of pre-written wirings for GCP, AWS, etc.
I cloned this repo and then ran the following command, which I stopped after waiting 8 minutes for it to complete. Note, I'm excluding wire tests which add ~3 minutes to the test runtime.
$ time go test $(go list ./... | grep -v wire)
? github.com/google/go-cloud/aws [no test files]
ok github.com/google/go-cloud/aws/awscloud (cached) [no tests to run]
ok github.com/google/go-cloud/blob (cached)
? github.com/google/go-cloud/blob/driver [no test files]
ok github.com/google/go-cloud/blob/fileblob (cached)
ok github.com/google/go-cloud/blob/gcsblob (cached)
^C
real 8m7.669s
user 0m4.329s
sys 0m1.283s
Compare with:
$ time go test -short $(go list ./... | grep -v wire)
? github.com/google/go-cloud/aws [no test files]
ok github.com/google/go-cloud/aws/awscloud (cached) [no tests to run]
ok github.com/google/go-cloud/blob (cached)
? github.com/google/go-cloud/blob/driver [no test files]
ok github.com/google/go-cloud/blob/fileblob (cached)
ok github.com/google/go-cloud/blob/gcsblob (cached)
ok github.com/google/go-cloud/blob/s3blob (cached)
? github.com/google/go-cloud/gcp [no test files]
ok github.com/google/go-cloud/gcp/gcpcloud (cached) [no tests to run]
ok github.com/google/go-cloud/health (cached)
ok github.com/google/go-cloud/health/sqlhealth (cached)
ok github.com/google/go-cloud/mysql/cloudmysql (cached) [no tests to run]
ok github.com/google/go-cloud/mysql/rdsmysql (cached) [no tests to run]
ok github.com/google/go-cloud/requestlog (cached)
ok github.com/google/go-cloud/runtimevar (cached)
? github.com/google/go-cloud/runtimevar/driver [no test files]
ok github.com/google/go-cloud/runtimevar/filevar (cached)
ok github.com/google/go-cloud/runtimevar/paramstore (cached)
ok github.com/google/go-cloud/runtimevar/runtimeconfigurator 0.046s
? github.com/google/go-cloud/server [no test files]
? github.com/google/go-cloud/server/sdserver [no test files]
? github.com/google/go-cloud/server/xrayserver [no test files]
? github.com/google/go-cloud/testing/replay [no test files]
? github.com/google/go-cloud/tests/gcp/app [no test files]
? github.com/google/go-cloud/tests/gcp/test-driver [no test files]
real 0m2.455s
user 0m2.254s
sys 0m0.925s
package main
import (
"context"
"log"
"time"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/google/go-cloud/runtimevar"
"github.com/google/go-cloud/runtimevar/paramstore"
)
func main() {
ctx := context.Background()
sess := session.Must(session.NewSession())
client := paramstore.NewClient(ctx, sess)
v, err := client.NewVariable(ctx, "/foo/bar", runtimevar.StringDecoder, ¶mstore.WatchOptions{
WaitTime: 5 * time.Second,
})
for {
snapshot, err := v.Watch(ctx)
if err != nil {
log.Println("Watch error:", err)
}
log.Println("Value is now:", snapshot.Value)
}
}
Run with:
$ aws ssm put-parameter --name="/foo/bar" --type=String --value="ohai from AWS"
$ go run foo.go
Value is now: ohai from AWS
Value is now: ohai from AWS
Value is now: ohai from AWS
Value is now: ohai from AWS
Something like:
Presently, go-cloud supports
From a comment on PR #33:
I wanted to err on the side of ensuring that the names would be valid and not have special meaning on the filesystem. In particular, something like colon can potentially allow
C:/foo.txt
to do bad things on Windows. I'd be happy to open a bug to support Unicode characters and fix if there's demand.
This is that bug. @shantuo
Terraform fails to provision GCP resources with the following errors:
Error: Error applying plan:
4 error(s) occurred:
* google_project_service.container: 1 error(s) occurred:
* google_project_service.container: Error enabling service: Error waiting for apis ["container.googleapis.com"] to be enabled for enocom-dev: googleapi: Error 403: The caller does not have permission, forbidden
* google_project_service.runtimeconfig: 1 error(s) occurred:
* google_project_service.runtimeconfig: Error enabling service: Error waiting for apis ["runtimeconfig.googleapis.com"] to be enabled for enocom-dev: googleapi: Error 403: The caller does not have permission, forbidden
* google_project_service.storage: 1 error(s) occurred:
* google_project_service.storage: Error enabling service: failed to issue request: googleapi: Error 403: The caller does not have permission, forbidden
* google_project_service.sql: 1 error(s) occurred:
* google_project_service.sql: Error enabling service: failed to issue request: googleapi: Error 403: The caller does not have permission, forbidden
This was apparently noted in hashicorp/terraform-provider-google#1579 and then fixed with hashicorp/terraform-provider-google#1632.
But the fix still isn't in the newest version of the Terraform GCP provider (v.0.14.0). Once the fix lands, we need to update this line to require the version with the fix.
Here is the error:
aws/provision-db.sh: Connecting to database...
docker: Error response from daemon: Mounts denied:
The path /var/folders/_h/ssstsdnj2998kj4bsmlxm17w00h7cm/T/tmp.JbxaL9nU
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.
time="2018-06-20T14:59:52-06:00" level=error msg="error waiting for container: context canceled"
aws/provision-db.sh: Removing ingress rule...
This with a vanilla install of Docker on MacOS 10.13.4.
Linters are our first and best defense against accidents like not checking return values. This is important.
I've identified one that I like a lot:
go get -u github.com/golangci/golangci-lint/cmd/golangci-lint
golangci-lint run (find . -type d ! -path "./.git*" ! -path "./tests*" ! -path "./goose*" ! -path "*_demo*")
This shows some issues we have, so I'll need to fix those first.
After running terraform apply
as part of the steps to provision infrastructure on AWS in the guestbook sample, terraform prompts me for a public SSH key:
var.ssh_public_key
A public key line in .ssh/authorized_keys format to use to authenticate to your instance. This must be added to your SSH agent for provisioning to succeed.
Enter a value:
We should add a note in the README which explains what the SSH key will be used for and how to create a key and find it.
Step 2 of #87 .
These can go in docs/
so they're easily read on GitHub.
Needs approval from @cassandroid .
What documents need to be written? Need to get approvals from @cassandraoid .
Definitely needed: Contributing document, getting started material.
Is it just as easy as https://docs.travis-ci.com/user/reference/osx/#Using-OS-X and running it as a shard?
Example failure (source):
--- FAIL: TestReturnsUpdatedSnapshotConcurrently (0.10s)
filevar_test.go:205: Snapshot.Value: {string}:
-: ""
+: `{"hello": "cloud"}`
FAIL
FAIL github.com/google/go-cloud/runtimevar/filevar 10.241s
Much like tests/gcp, add an integration test that runs on EC2 and checks that traces and logs are written.
EDIT 2019-07-09 Adding more context to this issue:
Right now, a server using the Go CDK server package will automatically capture OpenCensus traces, but will not capture request metrics like opencensus.io/http/server/request_count
. Go CDK servers should send these metrics out of the box.
A new field on *server.Options
that to initialize the metrics exporter (#13), and for the server's request handler to use ochttp.Handler
.
These two sets create convenience when writing sample app, but don't really simplify the user experience. We should consider move these into sample code from the library.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.