Use s3manager.BatchDelete about s5cmd HOT 3 CLOSED

Seems like we cannot use s3manager.BatchDelete for this release.

There are two different approaches to use s3manager.BatchDelete API.

1. Create paginator and use this paginator on delete operations.

  input := &s3.ListObjectsInput{
    Bucket:  aws.String("bucket"),
    Prefix: aws.String("prefix")
  }
  iter := s3manager.NewDeleteListIterator(svc, input)

This is fast but will not work since it removes the list result directly. We also need to remove keys that match with wildcard strings (not whole listing result). As an alternative, maybe we can write custom paginator (since it is an interface) and use this paginator on batch delete operations. But this does not reduce the complexity (I prefer our current architecture).

2. Give the list of objects to DeleteObjectsIterator and use this iterator on delete operations.

batcher.Delete(ctx, &s3manager.DeleteObjectsIterator{Objects: objects})

This is super easy and clean but slow. It won't perform well since it is required to list and collect all objects first and pass collected objects to DeleteObjectsIterator. Currently, we don't block during S3 pagination (listing). We are directly sending these keys into the URL channel and it works asynchronously.

That is why I am proposing to close this issue and remove from this milestone :)

@igungor What do you think? You can close the issue if you agree.

from s5cmd.

Thank you Onur.

Not a blocker. Let's keep the issue open for a while.

from s5cmd.

s3manager.BatchDelete does not provide any improvement over our solution.

from s5cmd.