Sometimes generating 33 ARI requests in a single second about certmagic HOT 7 CLOSED

I think this is a thundering herd issue. We probably need https://pkg.go.dev/golang.org/x/sync/singleflight with the domain as key.

every 6 hours, if respecting the Retry-After header

Is this zoned per queried hostname or per client?

from certmagic.

Thank you for the investigation and fix! We plan to keep an intermittent eye on ARI traffic patterns for a while, so I'll let you know if I see anything else jump out at me.

from certmagic.

Oh yeah that makes sense. If you get like 33 requests within a second, then all 33 of them might be triggering ARI via on-demand maintenance. I think certmagic needs to use https://pkg.go.dev/sync#WaitGroup to make sure it only gets fired off a single time per window.

from certmagic.

Is this zoned per queried hostname or per client?

Neither, it's per certificate. A single certificate may have multiple hostnames, but also a single client may manage multiple certificates.

from certmagic.

I'll take a look into this when I'm back at my desk

from certmagic.

The ARI suggestedWindow should be cached for the duration provided by the Retry-After header in the ARI response.

CertMagic does honor the Retry-After header, if present, by calling acme.RenewalInfo.NeedsRefresh().

I do agree this is likely a thundering herd, where many calls to update ARI come in before the first one finishes, since it lacks synchronization.

We can synchronize ARI fetching using the configured storage plugin. This will prevent any more than 1 instance in a cluster from fetching ARI at the same time, and after the first one does, the others will load and use its result.

Depending on the storage plugin, it's possible that this locking will be more expensive than actually fetching ARI, but it only happens once in a while, so maybe it's OK.

from certmagic.

Thanks for the report! This should fix it but without an offending client to test with I can only guess, but it makes sense to me.

I've synchronized the ARI fetching by the ARI UniqueIdentifier.

from certmagic.