Bug Report

Describe the bug

put_object is running into a timeout, because of an indentation error in
api.py:473:

async with aiohttp.ClientSession() as session:
	response = await self._url_open(
		"GET",
		"us-east-1",
		bucket_name=bucket_name,
		query_params={"location": ""},
		session=session,
	)

element = ET.fromstring(await response.text())

await response is outside the context manager and the session already got closed, leading to a timeout after some minutes.
Indenting the line fixes the problem.
PR incoming.

Hi team,

Really like this project. I saw that in read_part_data, read() is required to be a sync function. Can we support a use case where it's async? Thanks a lot.

Feature Request

Is your feature request related to a problem? Please describe.

Minio has introduced snowball object. This type of object allows user to upload multiple files in single request. You can find more details in Minio blog and API docs

Describe the solution you'd like

This feature is already implemented in official Minio package minio under the method upload_snowball_objects. I think it is good idea to keep original naming. Source code can be found here

It's looks very cool, but is there any more detailed Useage?

I found that some code fragments do not conform to the format of pep8, which is not beautiful and easy to read. I hope I can reformat the code.

Bug Report

Describe the bug

session argument ignored

• miniopy-async/miniopy_async/api.py

  Line 412 in f1b8912

  async with aiohttp.ClientSession() as session:

Expected behavior

Don't ignore session argument

list_objects does not properly handle truncation: the while is_truncated: loop of _list_objects always exits on the first iteration.

The following will reproduce:

from miniopy_async import Minio
import asyncio

client = Minio(
    "play.min.io",
    access_key="Q3AM3UQ867SPQQA43P2F",
    secret_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
    secure=True,  # http for False, https for True
)


async def main():
    objects = await client.list_objects("many-object-bucket")
    print(len(objects))


asyncio.run(main())

The bucket has 2000 objects so this should return 2000, but in miniopy_async 1.19 will return 1000.

A PR with a proposed fix will follow shortly.

Feature Request

👋 Hi. Would you be open to distributing miniopy-async on conda-forge as well?

It looks like it would be pretty straightforward to do given that grayskull (which can build the recipe for you) passes without any missing dependencies.

$ grayskull pypi --list-missing-deps --strict-conda-forge miniopy-async



#### Initializing recipe for miniopy-async (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package miniopy-async
miniopy-async 100% Time:  0:00:00   1.8 MiB/s|############################################################################################################################################################|
Checking for pyproject.toml
pyproject.toml not found.
Recovering information from setup.py
Executing injected distutils...
ERROR: Could not find an activated virtualenv (required).
ERROR:It was not possible to install setuptools.
Command: pip install setuptools --target=/tmp/pip-dir-0i63b_4n.
Error: Command '['/home/feickert/.pixi/envs/grayskull/bin/python3.12', '-m', 'pip', 'install', 'setuptools', '--target=/tmp/pip-dir-0i63b_4n']' returned non-zero exit status 3.
Recovering metadata from setup.cfg
Checking >> urllib3 100% |#########################################################################################################################################################|[Elapsed Time: 0:00:02]
Recovering license info from spdx.org ...
INFO:Best match for license Apache2.0 was Apache-2.0.
Best matches: ['Apache-2.0']
License type: Apache-2.0
License file: ['LICENSE']
Build requirements:
  <none>
Host requirements:
  - python >=3.7
  - pip
Run requirements:
  - python >=3.7
  - certifi
  - aiofile
  - aiohttp
  - urllib3

RED: Missing packages
GREEN: Packages available on conda-forge
All dependencies are already on conda-forge.

Maintainers:
   - hlf20010508

#### Recipe generated on /tmp for miniopy-async ####

I would be willing to help walk you through adding a package to conda-forge if this is the first time you've done this.

Is your feature request related to a problem? Please describe.

No, it is motivated by ssl-hep/ServiceX_frontend#352.

Describe the solution you'd like

Add miniopy-async to https://github.com/conda-forge/staged-recipes through making a PR with the recipe generated by grayskull.

Describe alternatives you've considered

None.

Pick this example and replace key/secret/url with your minio setup:

import asyncio

from miniopy_async import Minio

client = Minio(
    "play.min.io",
    access_key="Q3AM3UQ867SPQQA43P2F",
    secret_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
    secure=True  # http for False, https for True
)


async def main():
    # Download data of an object.
    print("example one")
    await client.fget_object("my-bucket", "my-object", "my-filename")


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

the file downloaded will be empty. Some debugging shows up that this code line is never executed (fget_object):

            async with aiofile.async_open(tmp_file_path, "wb") as tmp_file:
                async for data in response.content.iter_chunked(n=1024 * 1024):
                    await tmp_file.write(data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^

The response.content here is always EOF because it has been already read at api._url_open :

 async with aiohttp.ClientSession() as session:
            response = await session.request(
                method,
                urlunsplit(url),
                data=body,
                headers=headers
            )
            await response.read()
            ^^^^^^^^^^^^^^^^^^^^

        if response.status in [200, 204, 206]:
            return response

It also looks like aiohttp.ClientSession needs to be created outside of get_object otherwise stream read from the session released upon exit of _url_open would not work either.

Hello

Bug Report

Describe the bug

As I understand from the S3 docs, when providing a "x-amz-checksum-X" header to put an object, the same header should be returned upon stat/get (HEAD/GET). It's currently not the case. Is it due to minio headers generation ? Reading the raw request in the stat does not yield any more result.

To Reproduce

upload a file via "put_object" with a "x-amz-checksum-X" header that is valid. Stat or Get the object.

Expected behavior

The header should be present

I am using https but I am using a self-signed certificate and I need to turn off ssl certificate verification.
What should I do?

get_object has required session parameter but other methods hasn't session parameter

Describe the bug

If external session is useful in some use cases, it could be optional argument. But if it's so, all other methods have to have same session parameter.

Bug Report

Describe the bug

The values of custom metadata in put_object get converted into a list which leads to TypeError: Cannot serialize non-str key ['test_value'].

To Reproduce

Run

await minio.put_object(
    bucket_name=bucket_name,
    object_name=file_id,
    data=buffer,
    content_type="application/pdf",
    length=size,
    metadata={
        'test_key': 'test_value'
    }
)

Expected behavior

The resulting headers should be:

{"X-Amz-Meta-test_key": "test_value"}

not

{"X-Amz-Meta-test_key": ["test_value"]}

Additional context

The causing part is here:

Bug Report

Describe the bug

I got this warning:

example one
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x105883820>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x1062769a0>, 0.497560541)]']
connector: <aiohttp.connector.TCPConnector object at 0x10626bfa0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10626bfd0>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x106276820>, 0.524557333)]']
connector: <aiohttp.connector.TCPConnector object at 0x10628c460>
created my-object object; etag: 5d41402abc4b2a76b9719d911017c592, version-id: 5868d1d6-5085-43b1-bb43-425fded2f2a4

To Reproduce

Steps to reproduce the behavior:
My code is like this:

import io
from miniopy_async import Minio
import asyncio

client = Minio(
        "play.min.io",
        access_key="Q3AM3UQ867SPQQA43P2F",
        secret_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
    )


async def main():
    # Upload data.
    print('example one')
    result = await client.put_object(
        "test", "my-object", io.BytesIO(b"hello"), 5,
    )
    print(
        "created {0} object; etag: {1}, version-id: {2}".format(
            result.object_name, result.etag, result.version_id,
        ),
    )


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()

Expected behavior

I think the sesson or connection should be closed automatically, or some close api to use.

Desktop:

• OS: macos m1 pro
• Browser [e.g. chrome, safari]
• Version 13.0.1

hi, thanks for this package.
do you have any plan to support this repository?
it this rep always be updated with the latest minio python driver (minio-py) ?

Bug Report

Describe the bug

aiostream is not included in the dependencies

To Reproduce

# Inside new virtualenv
python setup.py install
python tests.py

Expected behavior

tests should be run

Screenshots or Outputs

Traceback (most recent call last):
  File "/home/erevilla/svn/miniopy-async/tests.py", line 19, in <module>
    from aiostream.stream import list as alist
ModuleNotFoundError: No module named 'aiostream'

Desktop:

• OS: Ubuntu Linux, 20.04