Git Product home page Git Product logo

go-retryablehttp's Introduction

go-retryablehttp

Build Status Go Documentation

The retryablehttp package provides a familiar HTTP client interface with automatic retries and exponential backoff. It is a thin wrapper over the standard net/http client library and exposes nearly the same public API. This makes retryablehttp very easy to drop into existing programs.

retryablehttp performs automatic retries under certain conditions. Mainly, if an error is returned by the client (connection errors, etc.), or if a 500-range response code is received (except 501), then a retry is invoked after a wait period. Otherwise, the response is returned and left to the caller to interpret.

The main difference from net/http is that requests which take a request body (POST/PUT et. al) can have the body provided in a number of ways (some more or less efficient) that allow "rewinding" the request body if the initial request fails so that the full request can be attempted again. See the godoc for more details.

Version 0.6.0 and before are compatible with Go prior to 1.12. From 0.6.1 onward, Go 1.12+ is required. From 0.6.7 onward, Go 1.13+ is required.

Example Use

Using this library should look almost identical to what you would do with net/http. The most simple example of a GET request is shown below:

resp, err := retryablehttp.Get("/foo")
if err != nil {
    panic(err)
}

The returned response object is an *http.Response, the same thing you would usually get from net/http. Had the request failed one or more times, the above call would block and retry with exponential backoff.

Getting a stdlib *http.Client with retries

It's possible to convert a *retryablehttp.Client directly to a *http.Client. This makes use of retryablehttp broadly applicable with minimal effort. Simply configure a *retryablehttp.Client as you wish, and then call StandardClient():

retryClient := retryablehttp.NewClient()
retryClient.RetryMax = 10

standardClient := retryClient.StandardClient() // *http.Client

For more usage and examples see the pkg.go.dev.

go-retryablehttp's People

Contributors

ash2k avatar claire-labry avatar dependabot[bot] avatar dolmen avatar evanphx avatar gavriel-hc avatar geototti21 avatar hashicorp-copywrite[bot] avatar hc-github-team-es-release-engineering avatar jbardin avatar jefferai avatar jen20 avatar justenwalker avatar manicminer avatar marianoasselborn avatar mark-adams avatar maximerety avatar mdeggies avatar mgwoj avatar ncabatoff avatar phinze avatar ppai-plivo avatar ryanuber avatar sebasslash avatar theckman avatar therealjt-abstract avatar tomclegg avatar vancluever avatar vbauerster avatar xonstone avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-retryablehttp's Issues

Custom `CheckRetry` apparently not working?

So.....based off this: https://github.com/hashicorp/go-retryablehttp/blob/master/client.go#L399-L405. By default, it ignores timeout errors. But I do want to force retry on those, here's what I did:

// Temporaryable is to match an error that has `.Temporary()`
type Temporaryable interface {
	Temporary() bool
}

// Timeoutable is to match an error that has `.Timeout()`
type Timeoutable interface {
	Timeout() bool
}

retryClient.CheckRetry = func(ctx context.Context, resp *http.Response, err error) (bool, error) {
	if terr, ok := err.(Temporaryable); ok && terr.Temporary() {
		return true, nil
	}

	if terr, ok := err.(Timeoutable); ok && terr.Timeout() {
		return true, nil
	}

	return retryablehttp.DefaultRetryPolicy(ctx, resp, err)
}

However, it doesn't seem to be workingโ€”with RetryMax set to 10, I can't see it getting past the 1st attempt.

Setting timeouts

As far as I can see in current API there's no way to set a request timeout, neither per request nor per client.

In a bare http client I can set client request timeout using client.Timeout Retrying client does not expose hashiClient struct member so I can't set this per client. And I'm also unaware on how can I set per request timeout?

No support for request.WithContext

Heya,

Lovely piece of code as usual, but one thing I find slightly annoying is that I cannot use request.WithContext since that returns the request struct from net/http. Any suggestions on how to get around this?

Client.Do data race on Client.HTTPClient

When Client.HTTPClient is nil and there are concurrent calls to Client.Do, a data race can occur at:

go-retryablehttp/client.go

Lines 493 to 495 in 27fc49b

if c.HTTPClient == nil {
c.HTTPClient = cleanhttp.DefaultPooledClient()
}

This can be resolved by doing the check and initialization inside a sync.Once.Do. I can open a PR if you'd like.

Timeout backoff feature

It would be great to have a feature that allows timeouts to increase or change every retry.

This could be integrated by having a TimeoutBackoff interface similar to the current Backoff option that allows users to create their own function to change the timeout after a failed attempt. If the request succeeds or runs out of retrys the timeout would be reset.

If people think this is a good idea, I can create the PR.

undefined errors.Unwrap

Hi,
Not sure if this is an issue with go-retryablehttp but everything used to work fine. Now when I run our test script (without any code changes) when it gets to the Golang part I'm seeing this error:

# github.com/hashicorp/go-retryablehttp
/go/pkg/mod/github.com/hashicorp/[email protected]/roundtripper.go:48:16: undefined: errors.Unwrap

And I can see 0.6.7 was just tagged 5 hours ago - unfortunately I don't know how I can make it use 0.6.6 again to see if that solves the issue.
There is a go.mod file which has the content

module github.com/our-org/our-sdk-go

require (
	github.com/antihax/optional v1.0.0
	github.com/hashicorp/go-retryablehttp v0.6.7
	golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a
)

but when I modify it and run the script again it seems to get re-generated?

Introduce more generic Logger interface

Hey,
I am using uber-go/zap for logger and I created a wrapper that impements the basic Logger interface

// Logger interface allows to use other loggers than
// standard log.Logger.
type Logger interface {
	Printf(string, ...interface{})
}

but I don't want to log debug logs, and in general I'd like more control.

I'd like to propose to change the Logger interface to something more generic so then every logger library can implement it

type Logger interface {
	Errorf(format string, v ...interface{})
        Infof(format string, v ...interface{})
        Debugf(format string, v ...interface{})
        Warnf(format string, v ...interface{})
}

This can be introduced either with backwards compatible way by keeping the old Logger interface or by breaking it as we are still on v0.X releases.

Another Option is to introduce an ExtendedLogger that is a generic interface that will replace the hclog.Logger interface in backwards compatible way.

// ExtendedLogger interface implements the basic methods that a logger library needs
type ExtendedLogger interface {
	Error(string, ...interface{})
	Info(string, ...interface{})
	Debug(string, ...interface{})
	Warn(string, ...interface{})
}

Please let me know what you think, more than happy to make the changes :)

Cheers

New issue when building with docker

When attempting a go get "github.com/hashicorp/go-retryablehttp" from within a docker container (e.g. RUN go get "github.com/hashicorp/go-retryablehttp") i receive the following error:

-#github.com/hashicorp/go-retryablehttp
/go/src/github.com/hashicorp/go-retryablehttp/client.go:447:17: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
/go/src/github.com/hashicorp/go-retryablehttp/client.go:507:16: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
/go/src/github.com/hashicorp/go-retryablehttp/client.go:538:16: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
/go/src/github.com/hashicorp/go-retryablehttp/client.go:545:15: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)
/go/src/github.com/hashicorp/go-retryablehttp/client.go:554:14: c.HTTPClient.CloseIdleConnections undefined (type *http.Client has no field or method CloseIdleConnections)

Thoughts as to what could be causing this? Just started happening in the past day or two.

[Question] Need http "method" in custom CheckRetry function

Hi, I've started to incorporate the use of retryablehttp into an existing Go library. One thing that I'll likely need to do is implement a custom CheckRetry function. In my case, I will need to use the http.Request's "Method" field, in conjunction with the response status code and "error", to determine if a retry is needed/wanted. For example, I want to avoid doing retries for a POST or PUT unless the error was some sort of client connection error (e.g. If a POST/PUT encounters a timeout, I don't want to retry because there's no way of knowing if the server actually did complete the request after the client timeout occurred; but if a client error occurred due to a temporary network hiccup, then a retry can be attempted).

Unfortunately, the CheckRetry function signature doesn't seem to provide the http.Request or any of the other request-specific info. Is there any advice you could provide for how to get access to the http.Request's "Method" field from within a custom CheckRetry function?

Thanks in advance!

Feature to allow set new context for retry

entry.go.txt
Purpose is to set new context in request or pass context to be considered by retryableHttp library for next usage. We are setting timeout set on context and would like to retry with new context, as once timeout happens, in below code snippet req.context().Done() completes and library does not retry.

select {
case <-req.Context().Done():
return nil, req.Context().Err()
case <-time.After(wait):
}

Sample code attached (please rename by removing ".txt" from end.

Could you add 429 Too Many Request as default policy for retry

Hi, the HTTP code 429 Too Many Requests is a retryable http error where it may be possible to parse out the Retry-After: {{num}} [time resolution] response header element. I've implemented it like this but it would be nice if it could be placed into the standard policy as well as default backoff?

Cheers,
Mario :)

func (ep *ClimatixICEndpoint) newClient() *retryablehttp.Client {
	client := retryablehttp.NewClient()

	client.Backoff = func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration {
		if resp.StatusCode == 429 {
                        if s, ok := resp.Header["Retry-After"]; ok {
				if sleep, err := strconv.ParseInt(s[0], 10, 32); err == nil {
					return time.Duration(int64(time.Second) * sleep)
				}
			}
		}

		return retryablehttp.DefaultBackoff(min, max, attemptNum, resp)
	}

	client.CheckRetry = func(c context.Context, resp *http.Response, err error) (bool, error) {
		if resp.StatusCode == 429 {
			return true, nil
		}
		// All other go to default policy
		return retryablehttp.DefaultRetryPolicy(c, resp, err)
	}

	return client
}

Using logrus as the custom logger

Is it possible to use a logrus logger as the custom logger?
When I try, it seems like the logger is not used as a LeveledLogger, which is what I was expecting

ResponseLogHook is not called if error is nil

From what I understood you are calling log hook only if response is present, which makes sense. However hooks in my perspective is something called regardless of whether response is successful or not. For sure I can use checkRetry feature to get notified when response is returned, but maybe it would be possible to add something like OnErrorLogHook to ensure http.Do is fully wrapped?

here is a piece of code I am referring to:
https://github.com/hashicorp/go-retryablehttp/blob/master/client.go#L523

Let me know if this makes sense, I can add go with a PR

Client configurable through constructor

Hi! I've just added here the options pattern so it'll be nicer to configure the Client with custom settings.
If you see it as a feature to be added to the library, I'll send the PR.

Thanks!

Mention use of cleanhttp which by default disables keepalives

It does not concern in most use cases but users should be aware that keep-alives are disabled explicitly by default. Most people would assume defaults and expect them enabled.

ie: I was planning to use it for an HTTP consumer and it definitely can re-use underlying connection. I only got to know while I was going thru source code.

What I mean is library should highlight the fact what it is doing internally, so that users are aware and can make an informed decision. Happy to make a PR :)

Support different CheckRetry per request

Currently, the CheckRetry policy is specific to the client instance and applies to all requests it makes. However, various APIs may behave differently and thus require different handling.
This forces us to create multiple client instances, each with its own CheckRetry. It would be better if we could reuse the same client instance, and provide a different policy per request.

Request using retryable returns an empty body

Hi. I've got the following request:

var reqBody []byte
req, err := http.NewRequest("GET", path, bytes.NewBuffer(reqBody))

if err != nil {
		return resp, err
	}

	req.Header.Set("Authorization", myAuth)
	req.Header.Set("Content-Type", "application/json")
	retryableReq, err := retryablehttp.FromRequest(req)
	if err != nil {
		return resp, err
	}

	resp, err = client.Do(retryableReq)

which is returning "" in the body BUT when I perform the same request using http it gets what's expected (something in the body)

Request wrapper make standard request `GetBody` property always nil

In the Go standard library, when creating a new request, the GetBody property of the request is set according to the body parameter.

This attribute will affect the client's behavior after receiving the 307, 308 response code. If it is empty, it is likely that it will not follow redirect.

if ireq.GetBody == nil && ireq.outgoingLength() != 0 {
	// We had a request body, and 307/308 require
	// re-sending it, but GetBody is not defined. So just
	// return this response to the user instead of an
	// error, like we did in Go 1.7 and earlier.
	shouldRedirect = false
}

But now when request wrapper creates a new request, the body parameter is not passed, making GetBody always nil.

func NewRequest(method, url string, rawBody interface{}) (*Request, error) {
	bodyReader, contentLength, err := getBodyReaderAndContentLength(rawBody)
	if err != nil {
		return nil, err
	}

	httpReq, err := http.NewRequest(method, url, nil)  // <=== here, body is nil
	if err != nil {
		return nil, err
	}
	httpReq.ContentLength = contentLength

	return &Request{bodyReader, httpReq}, nil
}

Library closes all connections, makes reuse impossible

As a followup to: #57 (review)

If I'm reading this correctly, the library currently seems to close all happy-case connections, which makes it unsuitable for use in a micro-service architecture where many requests go to the same few hosts. I'm currently observing this behavior in one of my services.
This is not just a connection-time issue, but in my case the closed requests eat up available sockets in TIME_WAIT state to the point of machine failure.

@jefferai Could you please revisit this review request above with this consideration in mind?
I'm happy to PR this is you concur it's an issue.

Data race in Client body handling?

We ran into a data race when using this in Vault (using v0.6.2):

https://app.circleci.com/pipelines/github/hashicorp/vault/8325/workflows/fd591032-03ba-46c7-b024-7c86545a08f9/jobs/54628/steps

WARNING: DATA RACE
Write at 0x00c00010ac40 by goroutine 65:
  github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp.(*Client).Do()
      /go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/go-retryablehttp/client.go:453 +0x1e8b
  github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/api.(*Client).RawRequestWithContext()
      /go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/api/client.go:846 +0x79c
  github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/api.(*Sys).Mount()
      /go/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/vault/api/sys_mounts.go:47 +0x22a
  github.com/hashicorp/vault/command/server/seal_test.prepareTestContainer.func2()
      /go/src/github.com/hashicorp/vault/command/server/seal/server_seal_transit_acc_test.go:171 +0x295
  github.com/hashicorp/vault/vendor/github.com/cenkalti/backoff.RetryNotify()
      /go/src/github.com/hashicorp/vault/vendor/github.com/cenkalti/backoff/retry.go:37 +0xc8
  github.com/hashicorp/vault/vendor/github.com/ory/dockertest.(*Pool).Retry()
      /go/src/github.com/hashicorp/vault/vendor/github.com/cenkalti/backoff/retry.go:24 +0xeb
  github.com/hashicorp/vault/command/server/seal_test.prepareTestContainer()
      /go/src/github.com/hashicorp/vault/command/server/seal/server_seal_transit_acc_test.go:158 +0x8c5
  github.com/hashicorp/vault/command/server/seal_test.TestTransitSeal_TokenRenewal()
      /go/src/github.com/hashicorp/vault/command/server/seal/server_seal_transit_acc_test.go:52 +0x53
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199 
 
Previous read at 0x00c00010ac40 by goroutine 14:
  net/http.(*persistConn).writeLoop()
      /usr/local/go/src/net/http/request.go:1389 +0x4d0 
 
Goroutine 65 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:960 +0x651
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1202 +0xa6
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:909 +0x199
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1200 +0x521
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1117 +0x2ff
  main.main()
      _testmain.go:48 +0x223 
 
Goroutine 14 (finished) created at:
  net/http.(*Transport).dialConn()
      /usr/local/go/src/net/http/transport.go:1581 +0xc06
  net/http.(*Transport).dialConnFor()
      /usr/local/go/src/net/http/transport.go:1313 +0x14f
==================
    testing.go:853: race detected during execution of test

I've tried reproducing this locally but haven't been successful yet. It appears to be a pretty rare occurrence as I don't recall seeing a race more than a couple of times (and I'm not sure if the previous times were the same race or from an already patched one).

I suspect this is somehow related to rewinding the request body since the write appears to be when setting the request body: req.Body = ioutil.NopCloser(body) here and the read in the net/http library: if r.Body != nil here (note: this is in go 1.13.8)

[Question] Using go-retryablehttp with gock

Hi, i'm using gock as a tool for testing.
When I switched my app to use *retryablehttp.Client instead of the standard http client, the gock tool stoped working and it is not longer possible to mock requests for test purposes.
Does retryablehttp compatible with gock? Is there a way to make it work? some additional configuration that can be set or a workaround?

I'm converting retryablehttp to http.Client as suggested in the documentation:

retryClient := retryablehttp.NewClient()
retryClient.RetryMax = 10

standardClient := retryClient.StandardClient() // *http.Client

Thanks,
Olla

http.Client compatible NewHTTPClient?

Hello! I like your functionality, but I'm in a situation where I need to adhere to the http.Client api exactly, and the Do() signature is not func (c *Client) Do(req *http.Request) (*http.Response, error).

Lots of enhanced/replacement libraries seem to implement the old api exactly so you can just swap out an import (pkg/errors vs. stdlib errors for instance). As a workaround, a wrapper is possible:

type RetryingHTTPClient struct {
    client *retryablehttp.Client
}

func (c *RetryingHTTPClient) Do(req *http.Request) (*http.Response, error) {
    wrappedReq, err := retryablehttp.FromRequest(req)
    if err != nil {
        return nil, err
    }

    return c.client.Do(wrappedReq)
}

Is there any openness to a PR adding a method like NewHTTPClient() that would return a client that conformed to something like func (c *RetryingHTTPClient) Do(req *http.Request) (*http.Response, error)?

// HTTPDoer is an interface for the one method of http.Client that is used by RetryingHTTPClient
type HTTPDoer interface {
	Do(*http.Request) (*http.Response, error)
}

Return response in error case

Right now the retryable client will return nil and the error if a request failed. It would be good to still have access to the response to for example check the status code or any error messages.

I know I can define a custom error handler to do this, but I think it should be the default behavior.

Request with context

Calling WithContext on wrapped Request, would return *http.Request, so user cannot do:

req, err := NewRequest(http.MethodGet, ts.URL, nil)
if err != nil { /* handle error */ }
ctx, cancel := context.WithCancel(req.Request.Context())
req = req.WithContext(ctx) // compile time error here

Also the lib would start retrying on context error, i.e. cancel called. Which is imo wrong. If user calls cancel, the intention is cancel, not retry.

DefaultClient and DefaultPooledClient

Hi, i was wondering why this package only use DefaultClient instead of DefaultPooledClient ?
My guess that in big production you have L4 balancer and you don't want to have tcp connection with keepalive.
But for a small production it might be useful.
Made a PR.
#14

`attemptNum` as part of `CheckRetry`

type CheckRetry func(ctx context.Context, resp *http.Response, attemptNum int, err error) (bool, error)

Came across a use case where this would've been a great help and was wondering if this was something discussed previously?

Setting timeouts (ie. deadlines)

Hello,

How would one go about setting timeouts (deadlines) for requests, specifically for GETs, but I guess it applies for all methods?

Thanks

Adding TLS transport in Client

I want to add TLSClientConfig in http Client.

transport := &http.Transport{TLSClientConfig: tlsConfig}

Is there a way to send tls config thorugh retryablehttp?

Thanks

URL in logs / errors and better control of the logging

Hi,

I'm currently using this otherwise nice package to interact with 3rd party APIs, which force me to put its API keys into the actual URL. Since the package include the URL in both its logs as well as include them in the errors it return to leak the API keys into our logs - which is unfortunate.

Additionally the logging doesn't quite allow me to do what I need of it. It seems like the package attempt to do leveled logs, but not really?

We used leveled logs (logrus) and ideally I would like the failed HTTP requests to be logged at WARN level. However the current implementation either force me to have non-warn related logs (even prefixed with debug at WARN level, failed requests logged at INFO or DEBUG or loose the information around what cause requests to fail entirely.

As I don't see adding a dependency to any leveled logging packages out there as an option, I would suggest either:

  1. Change the existing logger into a simple leveled logger interface defined in the package itself as well as provide helpers to make it as easy as possible (possibly also backwards compatible if that is a priority) to turn a log.Logger into one that implements this simple leveled logger.
  2. Specify two distinct log.Logger loggers on Client (one used for errors and one used for everything else). Given most decent leveled loggers today offer a possibility to be used as a standard log.Logger at a given level, this then, in turn, allow the user to set these using a leveled logger that is fixed to a specific level (most leveled logging packages I've seen has basic support for this).

I was wondering whether I should remake a similar package or if you would be open to a PR that:

  • Add a boolean field on Client that allow you to toggle whether to include the URL in errors/logs (with default behavior that keeps logs as is)
  • Either makes its own simple leveled logging interface or add two different log.Logger fields to Client?

Best regards,
Kasper

Examples please

Can you put some usage examples in the docs please. Nice reference but not much use in getting started

Do() should checks about CheckRetry value before calling it

I have integrated go-retryablehttp but went into an issue on segfault when Do() function calls "CheckRetry".

That was because my code create the Client struct itself using:
hc := &http.Client{}

So all extra members of the Client object were not defined (like CheckRetry).

Switching to:
hc := retryablehttp.NewClient()
Fixed the issue.

So, I think the library should check "CheckRetry" value before calling it.

Remove printing req.Method and req.Url

While using retryablehttp in my project where multiple requests are involved,fills my entire log with request urls and methods.I thing this print(on line 524) of client.go can be removed.This shouldn't be a problem as users of this package can print their request info before or after the request is done.

[Question] Guidance to use x-ray with retryable http

Hi, I'm using this excellent http tool but need to incorporate x-ray. In x-ray you wrap the http.Client using xray.Client(client) and it returns a http.Client where they have installed a RoundTripper like

// Client creates a shallow copy of the provided http client,
// defaulting to http.DefaultClient, with roundtripper wrapped
// with xray.RoundTripper.
func Client(c *http.Client) *http.Client {
	if c == nil {
		c = http.DefaultClient
	}
	transport := c.Transport
	if transport == nil {
		transport = http.DefaultTransport
	}
	return &http.Client{
		Transport:     RoundTripper(transport),
		CheckRedirect: c.CheckRedirect,
		Jar:           c.Jar,
		Timeout:       c.Timeout,
	}
}

Current version of retryablehttp seems not to allow for overloading the internal http.Client:

// NewClient creates a new Client with default settings.
func NewClient() *Client {
	return &Client{
		HTTPClient:   cleanhttp.DefaultPooledClient(),
		Logger:       defaultLogger,
		RetryWaitMin: defaultRetryWaitMin,
		RetryWaitMax: defaultRetryWaitMax,
		RetryMax:     defaultRetryMax,
		CheckRetry:   DefaultRetryPolicy,
		Backoff:      DefaultBackoff,
	}
}

Would it make sense to allow for an method like this(ish)?

func NewFromClient(client *http.Client) *Client {
	return &Client{
		HTTPClient:   client,
		Logger:       defaultLogger,
		RetryWaitMin: defaultRetryWaitMin,
		RetryWaitMax: defaultRetryWaitMax,
		RetryMax:     defaultRetryMax,
		CheckRetry:   DefaultRetryPolicy,
		Backoff:      DefaultBackoff,
	}
}

Cheers,
Mario

go-retryablehttp http.RoundTripper

Hi HashiCorp,
some 6 month ago I cloned your project and refactored Client to http.RoundTripper. The basic difference it that instead of

type Client struct {
	HTTPClient *http.Client // Internal HTTP client.
...

I'm using

type Transport struct {
	next http.RoundTripper
...

The reason why I did that is because it's much easier to integrate that to existing code bases. You can also use it effectively as part of more complicated middleware chains.

The work can be enhanced to provide both options in the retryablehttp package. Would you be interested in such a patch?

Implementation idea: there would be both retryablehttp.Client and retryablehttp.Transport types and client would be using the transport when constructed [1]. Please let me know what do you think.

ps.
Thanks for sharing this cool library, I really enjoy working with your code!

[1] https://github.com/hashicorp/go-retryablehttp/blob/master/client.go#L261

Add support for milliseconds

Hello,
This package works well except with Discord webhook endpoints.

Their reply looks like this:

{
  "global": false,
  "message": "You are being rate limited.",
  "retry_after": 555
}

The thing is, retry_after is specified in milliseconds, but go-retryablehttp will wait for more than 9 minutes because it handles it as seconds.

I looked at the doc and couldn't find a way to handle milliseconds. Did I miss something or is it a missing feature?
Thanks

Why close all idle connections when retry check is not OK?

Hi, thanks for developing such wonderful library. I was trying to use it in my project and I have a question on below line.

go-retryablehttp/client.go

Lines 575 to 581 in 4af2e4b

if !checkOK {
if checkErr != nil {
err = checkErr
}
c.HTTPClient.CloseIdleConnections()
return resp, err
}

May I know why need to do c.HTTPClient.CloseIdleConnections() when do not need retry? I found this will send a FIN to server side and start to close connection. But In my understanding, the connection can be still reused. Actually in my project, I found the connection number was keep growing after my service starts and accepts API requests. Then I removed this line in my project, the function is working and connections stop to keep growing.

rework logger

In #32 added custom logger, but it looks very strange, repeats twice the word DEBUG

{"level":"debug","time":"2019-04-04T16:45:01Z","message":"[DEBUG] POST https://`url`"}

#49 here I suggested adding a level logger

HTTP Request error should be propagated to the caller

Hello,

I'm working to add retryablehttp to a CLi I'm building for something, and noticed an interesting change in error messages as a result of doing that. Specifically when the Do() method exceeds its number of retries, the error it returns does not include the original error that was encountered:

go-retryablehttp/client.go

Lines 556 to 557 in 46f7ba5

return nil, fmt.Errorf("%s %s giving up after %d attempts",
req.Method, req.URL, c.RetryMax+1)

So if there was a connection reset error causing it, versus an HTTP 500, the error doesn't provide that context to the consumer.

You do provide an ErrorHandler, but with what you've provided to that handler it's impossible to create an error string that is identical to the one above. This is because the Do() method relies on the *http.Request being in-scope to generate the retries exceeded error string, but that's not passed to ErrorHandler.

I'm interested in seeing whether:

  1. We can enhance that line of code above to also include the original error string in the message.
  2. The ErrorHandler can be updated to receive an *http.Request so that we can create an error identical to that of Do().

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.