Git Product home page Git Product logo

minio-hs's Introduction

MinIO Haskell Client SDK for Amazon S3 Compatible Cloud Storage CIHackageSlack

The MinIO Haskell Client SDK provides simple APIs to access MinIO and any Amazon S3 compatible object storage.

This guide assumes that you have a working Haskell development environment.

Installation

Add to your project

Simply add minio-hs to your project's .cabal dependencies section or if you are using hpack, to your package.yaml file as usual.

Try it out in a REPL

For a cabal based environment

Download the library source and change to the extracted directory:

$ cabal get minio-hs
$ cd minio-hs-1.6.0/ # directory name could be different

Then load the ghci REPL environment with the library and browse the available APIs:

$ cabal repl
ghci> :browse Network.Minio

For a stack based environment

From your home folder or any non-haskell project directory, just run:

stack install minio-hs

Then start an interpreter session and browse the available APIs with:

$ stack ghci
> :browse Network.Minio

Examples

The examples folder contains many examples that you can try out and use to learn and to help with developing your own projects.

Quick-Start Example - File Uploader

This example program connects to a MinIO object storage server, makes a bucket on the server and then uploads a file to the bucket.

We will use the MinIO server running at https://play.min.io in this example. Feel free to use this service for testing and development. Access credentials are present in the library and are open to the public.

FileUploader.hs

#!/usr/bin/env stack
-- stack --resolver lts-14.11 runghc --package minio-hs --package optparse-applicative --package filepath

--
-- MinIO Haskell SDK, (C) 2017-2019 MinIO, Inc.
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--


{-# LANGUAGE OverloadedStrings   #-}
{-# LANGUAGE ScopedTypeVariables #-}
import           Network.Minio

import           Data.Monoid           ((<>))
import           Data.Text             (pack)
import           Options.Applicative
import           System.FilePath.Posix
import           UnliftIO              (throwIO, try)

import           Prelude

-- | The following example uses minio's play server at
-- https://play.min.io.  The endpoint and associated
-- credentials are provided via the libary constant,
--
-- > minioPlayCI :: ConnectInfo
--

-- optparse-applicative package based command-line parsing.
fileNameArgs :: Parser FilePath
fileNameArgs = strArgument
               (metavar "FILENAME"
                <> help "Name of file to upload to AWS S3 or a MinIO server")

cmdParser = info
            (helper <*> fileNameArgs)
            (fullDesc
             <> progDesc "FileUploader"
             <> header
             "FileUploader - a simple file-uploader program using minio-hs")

main :: IO ()
main = do
  let bucket = "my-bucket"

  -- Parse command line argument
  filepath <- execParser cmdParser
  let object = pack $ takeBaseName filepath

  res <- runMinio minioPlayCI $ do
    -- Make a bucket; catch bucket already exists exception if thrown.
    bErr <- try $ makeBucket bucket Nothing

    -- If the bucket already exists, we would get a specific
    -- `ServiceErr` exception thrown.
    case bErr of
      Left BucketAlreadyOwnedByYou -> return ()
      Left e                       -> throwIO e
      Right _                      -> return ()

    -- Upload filepath to bucket; object name is derived from filepath.
    fPutObject bucket object filepath defaultPutObjectOptions

  case res of
    Left e   -> putStrLn $ "file upload failed due to " ++ show e
    Right () -> putStrLn "file upload succeeded."

Run FileUploader

./FileUploader.hs "path/to/my/file"

Contribute

Contributors Guide

Development

Download the source

$ git clone https://github.com/minio/minio-hs.git
$ cd minio-hs/

Build the package:

With cabal:

$ # Configure cabal for development enabling all optional flags defined by the package.
$ cabal configure --enable-tests --test-show-details=direct -fexamples -fdev -flive-test
$ cabal build

With stack:

$ stack build --test --no-run-tests --flag minio-hs:live-test --flag minio-hs:dev --flag minio-hs:examples

Running tests:

A section of the tests use the remote MinIO Play server at https://play.min.io by default. For library development, using this remote server maybe slow. To run the tests against a locally running MinIO live server at http://localhost:9000 with the credentials access_key=minio and secret_key=minio123, just set the environment MINIO_LOCAL to any value (and unset it to switch back to Play).

With cabal:

$ export MINIO_LOCAL=1 # to run live tests against local MinIO server
$ cabal test

With stack:

$ export MINIO_LOCAL=1 # to run live tests against local MinIO server
stack test --flag minio-hs:live-test --flag minio-hs:dev

This will run all the test suites.

Building documentation:

$ cabal haddock
$ # OR
$ stack haddock

minio-hs's People

Contributors

donatello avatar ebozduman avatar harshavardhana avatar krisis avatar mpscholten avatar qnikst avatar thomasjm avatar trodrigu avatar vst avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

minio-hs's Issues

Support for Yandex Object Storage

I tried using minio-hs with Yandex Object Storage
and got an error:

uncaught exception: MinioErr
MErrHTTP (HttpExceptionRequest Request {
  host                 = "storage.yandexcloud.net"
  port                 = 443
  secure               = True
  requestHeaders       = [("Host","storage.yandexcloud.net"),("Authorization","<REDACTED>"),("X-Amz-Date","20200518T171120Z"),("x-amz-content-sha256","UNSIGNED-PAYLOAD")]
  path                 = "/bucket-test/piPeo"
  queryString          = "uploads"
  method               = "POST"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (StatusCodeException (Response {responseStatus = Status {statusCode = 403, statusMessage = "Forbidden"}, responseVersion = HTTP/1.1, responseHeaders = [("Server","nginx"),("Date","Mon, 18 May 2020 17:11:21 GMT"),("Content-Type","application/xml; charset=UTF-8"),("Transfer-Encoding","chunked"),("Connection","keep-alive"),("Keep-Alive","timeout=60"),("X-Amz-Request-Id","5ca10399c0567052"),("Content-Encoding","gzip")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "Response {responseStatus = Status {statusCode = 403, statusMessage = \"Forbidden\"}, responseVersion = HTTP/1.1, responseHeaders = [(\"Server\",\"nginx\"),(\"Date\",\"Mon, 18 May 2020 17:11:21 GMT\"),(\"Content-Type\",\"application/xml; charset=UTF-8\"),(\"Transfer-Encoding\",\"chunked\"),(\"Connection\",\"keep-alive\"),(\"Keep-Alive\",\"timeout=60\"),(\"X-Amz-Request-Id\",\"5ca10399c0567052\"),(\"Content-Encoding\",\"gzip\")], responseBody = \"<?xml version='1.0' encoding='UTF-8'?>\\n<Error><Code>AccessDenied</Code><Message>There were headers present in the request which were not signed</Message><Resource>/bucket-test/piPeo</Resource><RequestId>5ca10399c0567052</RequestId><HeadersNotSigned>x-amz-content-sha256</HeadersNotSigned></Error>\", responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}"))

Key message here is:

There were headers present in the request which were not signed
x-amz-content-sha256

I believe this can be fixed by adding the variable sha256Hdr to the list headersToSign

putObject don't support Conduit sink way

i'm writing an general conduit data tool for sink data to minio and es.
so basically, i use the following template.
but, currently putObject don't offer this kind of interface

  runConduitRes $ do
    (lift dataSourceChan >>= sourcTBMChan)
    .| C.concat
    .| toJSON
    .| getZipSink ((,) <$> ZipSink sinkES <*> ZipSink sinkMinIO)

Parallelize downloading objects

[ Inspired from https://github.com/minio/minio-go/issues/797]

Downloading an object can be parallelized using multiple GET request with range-headers. This can lead to significant download performance improvement for large files.

The strategy can be as follows:

  1. Perform a HEAD request on the object to retrieve its size and ETag.
  2. Choose a suitable split-size, and make a list of range headers to be used in subsequent requests.
  3. Perform the GET requests with range-headers and write them to appropriate offsets into the on-disk file. In each GET request use the If-Match header and set the ETag retrieved in step 1. This is to make sure that any changes to the object during the download may be detected.

Refactor LiveServer tests

Each functional test needs to be in its own function. This solves two issues:

  1. We can reduce and eliminate the number of compile time warnings the file generates.
  2. Improved readability.

bucket notify event support ?

i have less knowledge about minio Bucket Notification.
i want to get notify when some file created.

it seems it can use webSocket protocol for MQTT protocol or Webhooks.
what i should do by using this client, or setup myself WebSocket to do it.

listObjects lack retry mechanism

i fetch huge file size from server, it will contains 3 * 10^5 file number.

when i run listObjects, it report Request Timeout.

minio-hs-0.3.2 build fails with latest resourcet

As seen on the Stackage build server:

[ 6 of 16] Compiling Network.Minio.Data ( src/Network/Minio/Data.hs, dist/build/Network/Minio/Data.o )

src/Network/Minio/Data.hs:437:7: error:
    • No instance for (MonadBase IO (ResourceT IO))
        arising from the 'deriving' clause of a data type declaration
      Possible fix:
        use a standalone 'deriving instance' declaration,
          so you can specify the instance context yourself
    • When deriving the instance for (MonadBase IO Minio)
    |
437 |     , MonadBase IO
    |       ^^^^^^^^^^^^

I was able to reproduce the failure locally like so:

stack unpack minio-hs-0.3.2 && cd minio-hs-0.3.2
edit stack.yaml # add the following stack.yaml
stack build
# stack.yaml
resolver: nightly-2018-03-10
extra-deps:
- resourcet-1.2.0
- conduit-1.3.0
- conduit-extra-1.3.0
- http-conduit-2.3.0
- xml-conduit-1.8.0
- conduit-combinators-1.3.0

Make MinioT monad transformer

Hello!

It would be great, if Minio monad was implemented as a monad transformer, probably, like so:

newtype MinioT m a = Minio { unMinio :: ReaderT MinioConn (ResourceT m) a }

runMinioWith :: (MonadIO m) => MinioConn -> Minio m a -> m (Either MinioErr a)
... and so on

I already forked this library, and I will let you know if I get anything working.

minio-hs-1.2.0 build error: No instance for (MonadThrow Minio)

Building library for minio-hs-1.2.0..
[ 1 of 20] Compiling Lib.Prelude      ( src/Lib/Prelude.hs, dist/build/Lib/Prelude.o )                                                                                    
[ 2 of 20] Compiling Network.Minio.Data.ByteString ( src/Network/Minio/Data/ByteString.hs, dist/build/Network/Minio/Data/ByteString.o )                                   
[ 3 of 20] Compiling Network.Minio.Data.Crypto ( src/Network/Minio/Data/Crypto.hs, dist/build/Network/Minio/Data/Crypto.o )                                               
[ 4 of 20] Compiling Network.Minio.Data.Time ( src/Network/Minio/Data/Time.hs, dist/build/Network/Minio/Data/Time.o )                                                     
[ 5 of 20] Compiling Network.Minio.Errors ( src/Network/Minio/Errors.hs, dist/build/Network/Minio/Errors.o )                                                              
[ 6 of 20] Compiling Network.Minio.Data ( src/Network/Minio/Data.hs, dist/build/Network/Minio/Data.o )                                                                    

src/Network/Minio/Data.hs:570:7: error:
    • No instance for (MonadThrow Minio)
        arising from the 'deriving' clause of a data type declaration
      Possible fix:
        use a standalone 'deriving instance' declaration,
          so you can specify the instance context yourself
    • When deriving the instance for (MonadResource Minio)
    |
570 |     , MonadResource

Tests start failing with unordered-containers 0.2.16

cabal v2-test --constraint 'unordered-containers >= 0.2.16, probably harmless failure, unless the XML order is important?

Running 1 test suites...
Test suite minio-hs-test: RUNNING...
Tests
  Properties
    (checked by QuickCheck)
      selectPartSizes::                          OK (0.50s)
        +++ OK, passed 100 tests.
      selectCopyRanges::                         OK (0.42s)
        +++ OK, passed 100 tests.
      mkSSECKey::                                OK (0.08s)
        +++ OK, passed 100 tests.
  Unit tests
    XML Generator Tests
      Test mkCreateBucketConfig:                 OK
      Test mkCompleteMultipartUploadRequest:     OK
      Test mkPutNotificationRequest:             OK (0.04s)
      Test mkSelectRequest:                      FAIL (0.04s)
        test/Network/Minio/XmlGenerator/Test.hs:122:
        selectRequest XML should match: 
        expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><SelectRequest><Expression>Select * from S3Object</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><CompressionType>GZIP</CompressionType><CSV><QuoteCharacter>&#34;</QuoteCharacter><RecordDelimiter>\n</RecordDelimiter><FileHeaderInfo>IGNORE</FileHeaderInfo><QuoteEscapeCharacter>&#34;</QuoteEscapeCharacter><FieldDelimiter>,</FieldDelimiter></CSV></InputSerialization><OutputSerialization><CSV><QuoteCharacter>&#34;</QuoteCharacter><QuoteFields>ASNEEDED</QuoteFields><RecordDelimiter>\n</RecordDelimiter><QuoteEscapeCharacter>&#34;</QuoteEscapeCharacter><FieldDelimiter>,</FieldDelimiter></CSV></OutputSerialization><RequestProgress><Enabled>FALSE</Enabled></RequestProgress></SelectRequest>"
         but got: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><SelectRequest><Expression>Select * from S3Object</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><CompressionType>GZIP</CompressionType><CSV><QuoteCharacter>&#34;</QuoteCharacter><RecordDelimiter>\n</RecordDelimiter><QuoteEscapeCharacter>&#34;</QuoteEscapeCharacter><FieldDelimiter>,</FieldDelimiter><FileHeaderInfo>IGNORE</FileHeaderInfo></CSV></InputSerialization><OutputSerialization><CSV><QuoteCharacter>&#34;</QuoteCharacter><RecordDelimiter>\n</RecordDelimiter><QuoteEscapeCharacter>&#34;</QuoteEscapeCharacter><FieldDelimiter>,</FieldDelimiter><QuoteFields>ASNEEDED</QuoteFields></CSV></OutputSerialization><RequestProgress><Enabled>FALSE</Enabled></RequestProgress></SelectRequest>"
        Use -p '/Test mkSelectRequest/' to rerun this test only.
    XML Parser Tests
      Test parseLocation:                        OK
      Test parseNewMultipartUpload:              OK
      Test parseListObjectsResponse:             OK
      Test parseListObjectsV1Response:           OK (0.04s)
      Test parseListUploadsresponse:             OK
      Test parseCompleteMultipartUploadResponse: OK
      Test parseListPartsResponse:               OK
      Test parseCopyObjectResponse:              OK
      Test parseNotification:                    OK (0.04s)
      Test parseSelectProgress:                  OK
    Bucket Name Validity Tests
      Too short 1:                               OK
      Too short 2:                               OK
      Too long 1:                                OK
      Has upper case:                            OK
      Has punctuation:                           OK
      Has hyphen at end:                         OK (0.04s)
      Has consecutive dot:                       OK
      Looks like IP:                             OK
      Valid bucket name 1:                       OK
      Valid bucket name 2:                       OK
      Valid bucket name 3:                       OK
    Object Name Validity Tests
      Empty name:                                OK
      Has unicode characters:                    OK
    Parse MinIO Admin API ServerInfo JSON test
      FSBackend:                                 OK
      Erasure Backend:                           OK
      Unknown Backend:                           OK
    Parse MinIO Admin API HealStatus JSON test
      Good:                                      OK (0.04s)
      Corrupted:                                 OK
      Incorrect Value:                           OK
    Parse MinIO Admin API HealStartResp JSON test
      Good:                                      OK
      Missing Token:                             OK
    limitedMapConcurrently Tests
      Test with various thread counts:           OK (0.27s)

1 out of 39 tests failed (0.51s)
Test suite minio-hs-test: FAIL
Test suite logged to: dist/test/minio-hs-1.5.3-minio-hs-test.log
0 of 1 test suites (0 of 1 test cases) passed.

presignedGetObjectUrl url is not accepted by S3 (permanentRedirect)

Hi,

First of all thanks for making this library, I've successfully used it to push data to S3.

Problem I have is with presignedGetObjectUrl, it generates an URL like this:

https://s3.amazonaws.com/<bucket>/<path>?<auth>

But I’m getting a permanent redirect from AWS saying it should be

https://<bucket>.s3.amazonaws.com/<path>?<auth>

Am I using it wrong or is this a bug?

Thanks in advance!

minio putObject always fail!

i write a simple scrip to upload streaming data.
but it always fail.

  runResourceT $ do
    let (ci, accessKey, secretKey, bucket)
          = ( "http://10.129.35.175:9000"
            , "BY2BFHISRTPNY36IR4TD"
            , "ZB66/2jxW0bXkiEU0kufFT0ni1tOut9QJG8v1hb7"
            , "larluo")
    conn <- liftIO . M.connect $
                ci & M.setCreds (M.Credentials accessKey secretKey)
                   & M.setRegion "us-east-1"


    flip runReaderT conn . M.unMinio $ do
      bExist <- M.bucketExists "larluo"
      when (not bExist) $ void $ M.makeBucket "larluo" Nothing
      {--
      sourceTBMChan chan
              .| C.concat
              .| C.take 3
              .| C.map (cs . J.encode)
              .| C.iterM (liftIO . print)
              & \s -> M.putObject bucket "postgresql.txt" s Nothing M.defaultPutObjectOptions
      --}
      M.putObject bucket "postgresql.txt" (C.yieldMany ["aaa", "bbb"]) Nothing M.defaultPutObjectOptions
    liftIO $ putStrLn "finished"
Ok, two modules loaded.
*Main Network.Minio Data.Conduit Data.Function> repl
*** Exception: HttpExceptionRequest Request {
  host                 = "10.129.35.175"
  port                 = 9000
  secure               = False
  requestHeaders       = [("Authorization","<REDACTED>"),("content-encoding","aws-chunked"),("X-Amz-Date","20190701T074337Z"),("Host","10.129.35.175:9000"),("x-amz-decoded-content-length","6"),("content-length","178"),("x-amz-content-sha256","STREAMING-AWS4-HMAC-SHA256-PAYLOAD")]
  path                 = "/larluo/postgresql.txt"
  queryString          = "uploadId=6a48a93e-defd-4213-92d2-b937b6741a86&partNumber=1"
  method               = "PUT"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (StatusCodeException (Response {responseStatus = Status {statusCode = 400, statusMessage = "Bad Request"}, responseVersion = HTTP/1.1, responseHeaders = [("Content-Type","text/plain"),("Connection","close")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "Response {responseStatus = Status {statusCode = 400, statusMessage = \"Bad Request\"}, responseVersion = HTTP/1.1, responseHeaders = [(\"Content-Type\",\"text/plain\"),(\"Connection\",\"close\")], responseBody = \"400 Bad Request\", responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}")

Minio constructor is hidden for runResourceT intergration.

i have some ResourceT code block.
so i want to integrate with Minio monad with following code method:
but the unMinio is not exposed.
so i only can use liftResourceT to lift my resourceT to Minio monad.
it's seems a little too heavy.

runSourceT $
  (reg, chan) <- allocate ...
  _ <- async $ liftIO $ do
    ... network fetch
    atomically $ writeTBMChan chan rows
  flip runReaderT con . unMinio $
    runConduit $ sourceTBMChan chan ...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.