RW - wrong status code for storage.ErrTooOldSample (out of order time window feature) about prometheus HOT 4 CLOSED

You're right, 404 is not appropriate - I meant to use 400 - Bad Request

The HyperText Transfer Protocol (HTTP) 400 Bad Request response status code indicates that the server cannot or will not process the request due to something that is perceived to be a client error (for example, malformed request syntax, invalid request message framing, or deceptive request routing).

From my understanding, 403 is usually used in the context of security/RBAC - which isn't the case here, since this is a business logic restriction. Seeing 403 in logs would usually imply some auth issue which would be misleading.

from prometheus.

Why 404? I think 403 is more appropriate:

403 Forbidden
The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.

from prometheus.

Thanks.

I just handled an incident triggered by this behavior. The Prometheus server returned 500: too old sample one time to an agent configured to remote write its data. Agent then endlessly retried the same call again and again block all samples until I killed / cleaned it.

Returning 400 seems like the right to do.

from prometheus.

Looks like ErrOutOfOrderExemplar and a few others would have the same problem.

from prometheus.