aws / aws-sdk-go-v2 Goto Github PK

The global Config type has several service specific configuration values. These values need to be moved to the individual service client types.

The SDK needs to be able to code generate collection (slice/map) elem types as value types not pointers.

There are several places where the v1 SDK's documentation is still exists in v2 repo. These need to be cleaned up and removed.

Related to the confusion caused in #83

SDK API operations such as s3.GetObject use and io.Reader in the response for the customer to read API response data from. This prevents the SDK from retrying failed get operations as the SDK's Send has already returned.

An alternative approach would be for the SDK to take an io.Writer builder/getter as input and write to that stream. This would cause the Send to block until the response is written to the io.Writer...

Example customer code:

var f *os.File

req := svc.GetObjectRequest(params)
resp, err := req.Send(ctx, writeTempFileBuilder(&f))
if err != nil {
    panic(err)
}
if err := f. Sync(); err != nil {
    panic(err)
}

info, err := f.Stat()
if err != nil {
    panic(err)
}

fmt.Println("File Info:", Info.Name(), info.Size())
f.Close()

Looks like there is a memory issue sometimes when building/linking the op_crawler testing utility. A unit test failed with a Go link failure.

go build github.com/aws/aws-sdk-go-v2/internal/awstesting/cmd/op_crawler: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed
Makefile:61: recipe for target 'unit' failed
make: *** [unit] Error 1

Version of AWS SDK for Go?

What's the command to determine this?
Else, going to assume 493e3ce

Version of Go (go version)?

go version go1.9.2 linux/amd64

What issue did you see?

With medialive the way to test an empty token is *next == "". on mediapackage it's next == nil. I'm puzzled why.

• https://godoc.org/github.com/aws/aws-sdk-go-v2/service/medialive#ListChannelsOutput
• https://godoc.org/github.com/aws/aws-sdk-go-v2/service/mediapackage#ListOriginEndpointsOutput

Steps to reproduce

Compare

• https://s.natalian.org/2018-01-09/live.go
• https://s.natalian.org/2018-01-09/pkgs.go

Notice if live.go uses:

    if next == nil {
        break
    }

It just goes in an infinite loop.

~$ go version ; git -C ~/go/src/github.com/aws/aws-sdk-go-v2 describe --all
go version go1.9.2 linux/amd64
tags/v2.0.0-preview.2

What issue did you see?

I find it cumbersome to maintain a mapping between my struct and dynamodb with expression.Set. Is there a way to map/marshal the whole struct to dynamodb? I can't see how that's done with https://godoc.org/github.com/aws/aws-sdk-go-v2/service/dynamodb/expression

Requiring the user to retrieve the response metadata from the request type is very awkward. The output type should provide a way to access the response metadata.

Version of AWS SDK for Go?

{
	"checksumSHA1": "VsyHSneU5fMUAaQDQ1BAqZcvrG0=",
	"path": "github.com/aws/aws-sdk-go-v2",
	"revision": "01b01bc24609090862fbae454a86704b75a0f3b3",
	"revisionTime": "2018-01-11T17:45:17Z"
},

Version of Go (go version)?

1.9.2

What issue did you see?

I was just tried to upgrade from v1 to v2. I'm using cloudformation API. I could successfully created a stack with CreateStackRequest and got the stack id. But then when I used WaitUntilStackCreateComplete, it never returned.

Steps to reproduce

	// create stack
	req := client.CreateStackRequest(&input)
	output, err := req.Send()
	if err != nil {
		return
	}
	fmt.Println(output)

	// wait the stack to be created
	describeInput := cf.DescribeStacksInput{
		StackName: output.StackId,
	}
	fmt.Printf("Waiting until the stack is created...")
	if err = client.WaitUntilStackCreateComplete(&describeInput); err != nil {
		fmt.Println(err)
		deleteStack(stackId)
		return
	}
	fmt.Println("Done.")

This is a snippet of my code. I never saw the Done printing out.

If you have have an runnable example, please include it.

~$ go version
go version go1.9.2 linux/amd64
~$ git --git-dir ~/go/src/github.com/aws/aws-sdk-go-v2/.git describe --all
tags/v2.0.0-preview.2

What happened to CredentialsChainVerboseErrors in v1?

The SDK's protocol marshalers will need to be updated to correctly marshal collection elem value types.

A major change in the v2 API is the requirement to use Request methods (#33). We currently use the iface packages to test our code using the v1 API with the strategy described in

Could you please ensure that this mocking remains possible? Perhaps the Request API can return a Request object that doesn't go out to the web and instead returns a mock Response? Other alternative mock strategies would be equally welcome.

To tack on, the suggestion in #50 to split up the iface packages would likely make mocking for us more painful as we would have to generate more interfaces.

The aws.Config pointers be converted to value` types.

#84 (comment)

• bad version of dynamodbattribute marshaling and unmarshaling
• bad version of batcherrors

The v1 SDK exposes a single interface that most errors in the SDK try to conform to. awserr.Error. This error doesn't make sense for a lot of the errors the SDK core and utilities expose. Specifically the code and message values only make sense for API response errors.

The v2 SDK should refactor awserr.Error into an APIError interface type that basically is the awserr.RequestFailure type. I think the generic awserr.Error can be replaced in favor with errors that support the github.com/pkg/errors package's Wrap and Cause pattern.

Refactoring these error types should also satisfy a goal of error checking in user applications should be based more on type assertions instead of string comparison.

V2 is an opportunity to refactor the pain points of the S3 crypto client.

Version of AWS SDK for Go?

Current master as of 2018-01-10: 14d1593

Version of Go (go version)?

% go version
go version devel +9044f018ec Tue Jan 9 01:53:38 2018 +0000 darwin/amd64

What issue did you see?

s3manager.Uploader panics when uploading an object with custom metadata.

Steps to reproduce

Code (make sure to set the bucket name): https://play.golang.org/p/l1nWAdGf_qT

Output:

% go run upload.go
panic: reflect.Set: value of type map[string]*string is not assignable to type map[string]string

goroutine 1 [running]:
reflect.Value.assignTo(0x139e400, 0xc4206c4030, 0x15, 0x14342cc, 0xb, 0x139ebe0, 0x0, 0x139e400, 0xc4206c4030, 0x15)
	/Users/fsouza/.gimme/versions/go/src/reflect/value.go:2235 +0x41d
reflect.Value.Set(0x139ebe0, 0xc4206c6090, 0x195, 0x139e400, 0xc4206c4030, 0x15)
	/Users/fsouza/.gimme/versions/go/src/reflect/value.go:1373 +0xa4
github.com/aws/aws-sdk-go-v2/internal/awsutil.rcopy(0x139ebe0, 0xc4206c6090, 0x195, 0x139e400, 0xc42012c230, 0x195, 0xc4206c6000)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:97 +0xc3f
github.com/aws/aws-sdk-go-v2/internal/awsutil.rcopy(0x142a820, 0xc4206c6000, 0x199, 0x14287a0, 0xc42012c1c0, 0x199, 0x1)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:78 +0x261
github.com/aws/aws-sdk-go-v2/internal/awsutil.rcopy(0x14194a0, 0xc4206c6000, 0x16, 0x1373820, 0xc42012c1c0, 0x16, 0x101)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:68 +0x8e7
github.com/aws/aws-sdk-go-v2/internal/awsutil.Copy(0x14194a0, 0xc4206c6000, 0x1373820, 0xc42012c1c0)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/internal/awsutil/copy.go:21 +0xb9
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.(*uploader).singlePart(0xc420177ea0, 0x14a50c0, 0xc4206c4000, 0x1, 0x14a39e0, 0xc420010050)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:558 +0x65
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.(*uploader).upload(0xc420177ea0, 0x0, 0x0, 0x0)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:458 +0x537
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.Uploader.UploadWithContext(0x500000, 0x5, 0x0, 0x2710, 0x14aca40, 0xc420011260, 0x0, 0x0, 0x0, 0x1c08210, ...)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:376 +0x184
github.com/aws/aws-sdk-go-v2/service/s3/s3manager.Uploader.Upload(0x500000, 0x5, 0x0, 0x2710, 0x14aca40, 0xc420011260, 0x0, 0x0, 0x0, 0xc42012c1c0, ...)
	/Users/fsouza/src/github.com/aws/aws-sdk-go-v2/service/s3/s3manager/upload.go:347 +0xc4
main.main()
	/Users/fsouza/upload.go:17 +0x41b
exit status 2

The SDK's protocol (un)marshaling needs to be updated for typed enums. Specifically handling typed enums being string value type aliases not pointer.

zero enum value instructs marshaler the field is unset.

Add the ability to paginate a request from the API's typed request.

One way this can be refactored is to breakout the pagination from a callback to a pattern similar to bufio's scanner.

This is an example using MediaLive's ListInput's API.

V1

err = svc.ListInputsPages(&medialive.ListInputsInput{},
    func(page *medialive.ListInputsOutput, lastPage bool) bool {
        log.Println("Received", len(page.Inputs), "inputs in page")
        Inputs = append(Inputs, page.Inputs...)
        return true
    })

Proposed V2

req := svc.ListInputsRequest(&medialive.ListInputsInput{})
pager := req.Paginate()

for pager.Next() {
    page := pager.Page()
    Inputs = append(Inputs, page.Inputs...)
}
if err := pager.Err(); err != nil {
   panic("paging failed, " + err.Error())
}

With V2, we are curious if people are using the setters in the service APIs. As this isn't too idiomatic to Go, we want to ensure that there is a benefit that people are getting other than just convinience. Should we keep setters in V2?

Refactor CrdentialsLoader and CredentialProvider into a single entity instead of requiring the user to interface with both. This moves the critical section and caching from loader to the provider if the provider needs it.

This replaces the config and client's CredentialsLoader field with a Credentials CredentialsProvider. the CredentialsProvider would stay and interface, but it would be the responsibility of the provider to manage the critical section if it needs one.

~$ go version
go version go1.9.2 linux/amd64
~$ git --git-dir ~/go/src/github.com/aws/aws-sdk-go-v2/.git describe --all
tags/v2.0.0-preview.2

Hi, is there a better pattern to use to work out where a service (in my example, medialive) is active?

http://play.golang.org/p/O4MlKjZAP5O

Functions with region endpoints.Region and cfg.Region = region.ID() looks ugly to me.

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

493e3ce

Version of Go (go version)?

go version go1.9.2 linux/amd64

What issue did you see?

main.go:13:2: cannot find package "github.com/aws/aws-sdk-go-v2/aws/session" in any of:
        /usr/lib/go/src/github.com/aws/aws-sdk-go-v2/aws/session (from $GOROOT)
        /home/hendry/go/src/github.com/aws/aws-sdk-go-v2/aws/session (from $GOPATH)

Steps to reproduce

1. Copy the example from https://godoc.org/github.com/aws/aws-sdk-go-v2#hdr-Complete_SDK_Example

2. Paste it into a main.go somewhere inside $GOPATH/src

3. Run it

Retries for throttled request do not cap at 5 minutes due to throttled boolean not being checked first.

aws/aws-sdk-go#1654

DynamoDB.UpdateItemRequest is quite painful to work with.

If my struct property might be empty, I have to take dynamodb.UpdateItemInput out and create a condition that somehow updates ExpressionAttributeNames, ExpressionAttributeValues & let's not forget UpdateExpression.

I wish it was as easy as the example at https://godoc.org/github.com/aws/aws-sdk-go-v2/service/dynamodb/dynamodbattribute when I can dynamodbattribute.MarshalMap my complete struct to item and UpdateItemRequest with simply the Item:. https://godoc.org/github.com/aws/aws-sdk-go-v2/service/dynamodb#UpdateItemInput seems to require a very complex structure for an update. :/ Unless I am missing something?

The V2 SDK's models are a little out of date and need updated to the most recent released models.

To prevent s3manager's input shapes from getting out of date, the SDK should use code generation. One solution is to use reflection on service/s3 input shape and generate from that.

In regards to #88

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?
LATEST

Version of Go (go version)?
1.9.2

Support for Cognito just the way it is supported in JavaScript SDK and Android SDK to calculate the Auth Challenge to send in RespondToAuthChallenge

Refer: aws/aws-sdk-go#1245

The SDK's aws.Request type is too generic and does too much. We want to break this apart. Specifically decouple the http.Request/Response from the the SDK's request structure. In v1 SDK having the HTTP request built early on made it difficult to extend and modify how the SDK handled requests. e.g HTTP Body value couldn't be fixed to be set to nil when there wasn't actually a request body.

One way this could be done is to abstract the Transport from the SDK's API request. This potentially could manifest as a Sender, or Transport interface that all SDK's service clients have as a member. This type could be responsible for the protocol (un)marshaling, and round tripping the request over the wire. This pattern would more easily allow the SDK to support other transport protocols. Such as TCP and MQTT.

Port aws/aws-sdk-go#1640 to v2.

SDK's Usage of EndpointResolver should be updated so that the DisableSSL and DualStack aws.Config values can be removed. Ideally the EndpointResolver is constructed with these options at setup instead of at runtime.

The service client New methods should return an error in addition to the client value. This error is to capture validation of configuration, and resolving endpoints.

Currently endpoint resolving is performed per request. This is excessive as the endpoint will always be the same across a client value.

EndpointsResolver type is duplicated between aws and enpdoints package. This needs to be normalized.

The SDK's codegen should be updated to render all API enums as typed values. With the enum being a type alias of string value type not pointer.

Remove the multiple ways an API can be called and simplify to a single request method that returns a request type specific to the API.

It would be helpful to create usage examples for an application using v1 explaining how to migrate to specific changes in v2.

Could you please provide instructions to update https://amazonlinux.com/ to a >=1.9 release?

sudo yum install go attempts to install 1.8.3-1.amzn2 currently.

Thanks!

Customers have requested that the SDK's generated iface packages generate an interface per API operation, instead of grouping all operations together in a single interface.

This feedback is both from usability study, and v1 SDK users that found the size, and breaking changes of the current iface package to be difficult to use.

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

v2

Version of Go (go version)?

1.9

What issue did you see?

Hi. So I have a test like this:

func (fc *fakeCreateCFClient) DescribeStacksRequest(input *cloudformation.DescribeStacksInput) cloudformation.DescribeStacksRequest {
	if fc.stackname == "NotEmptyStack" || fc.stackname == "DescribeStackFailed" {
		return cloudformation.DescribeStacksRequest{
			Request: &aws.Request{
				Data:  &NotEmptyStack,
				Error: fc.err,
			},
		}
	}
	return cloudformation.DescribeStacksRequest{
		Request: &aws.Request{
			Data: &cloudformation.DescribeStacksOutput{},
		},
	}
}

I was expecting it to fail with a given error in the fake client which was working perfectly fine with v1 since that particular method was returning an error. Now, I don't have a perfect way to mock Send so I guessed, that maybe a request error like the above is the way to make errors propagate through to Send. Alas, that's not the case.

Thus, my question is, how do I make error scenarios happen?

Thanks,
Gergely.

The external configuration loading package needs to be documented. Its currently very bare, and not obvious how to use its extended functionality.

https://godoc.org/github.com/aws/aws-sdk-go-v2/aws/external

NewSharedConfig should return errors if it is unable to load the profile or the files. The caller of the function should be responsible for determining what to do with this information. Currently errors are silently dropped.

aws/aws-sdk-go#1713

Need to sync v1 API models into v2.

Currently the code generation is very complex and requires knowledge of previous states to make the correct decision. A better solution would be a visitor pattern and use different walkers to generate a particular piece of code.

The v2 SDK is moving its execution flow way from the need of creating functional option and towards a more explicit configuration pattern. The S3 manager Upload and Download methods need to be refactored to not take functional options and provide a more explicit configuration story.

The embedded Request within the API typed request is mutable and incorrectly will persist data.