Violation of Strict Consistency for HEAD-Object after DELETE-Object on buckets with Bucket Replication about minio HOT 6 CLOSED

@klauspost We only interact with the "top" object - we do not use the version id at all.

I just used your example with "mc" in a bash loop. I ran it 100 times (not parallel) and it failed 4 out of 100 times. Maybe calling "mc" from bash is a bit slower than the version I used with Typescript/Deno or with php.

One information I forgot: the tests I ran with Deno ran against a load balancer (haproxy) that points with a "leastconn" config to any of the five nodes in the main MinIO cluster.

I just ran my Deno test gain - this time on the first node of the cluster itself and I directed the MinIO requests only to this first node. Then the failure rate increase to around 74%. It seems that the faster the requests are, the higher the failure rate.

from minio.

Inserting time.Sleep(time.Second) at the start of replicateDelete reproduces it reliably.

from minio.

We just updated both clusters to the new version "2024-05-07T06-41-25Z". We can confirm that we cannot reproduce the inconsistency anymore. Thanks a lot for the very quick fix.

from minio.

Thanks for the detailed report!

Ping @poornas @krisis - I will see if I can reproduce and see where the race is happening.

from minio.

@ChristophWersal One question. Are the deletes and headobject for a specific version or just for the "top" object - meaning requests for object without version id?

I am trying mc cp 0.txt myminio/testbucket&&mc rm myminio/testbucket/0.txt&&mc stat myminio/testbucket/0.txt on a replicated bucket. This creates a delete marker and doesn't delete the older versions and from casual testing it doesn't reproduce.

from minio.

Thanks. I am not going to speculate too much. That said my guess is there is a window in the replication process where the request is proxied to the replica and the replica serves the headobject before the replication has deleted the object.

from minio.