project-monai / monai-deploy-storage Goto Github PK

View Code? Open in Web Editor NEW

5.0 5.0 3.0 40.3 MB

Storage layer for MONAI Deploy clinical data pipelines system

License: Apache License 2.0

C# 99.35% Shell 0.65%

monai-deploy-storage's People

Contributors

Stargazers

Watchers

Forkers

chadkowski bridgecrew-perf6

monai-deploy-storage's Issues

Allow admin health check to be disabled

Is your feature request related to a problem? Please describe.
For IG, mc is not used and not required so it doesn't need the admin health check.

Describe the solution you'd like
An option to not enable admin health check.

Integration test for minio

Is your feature request related to a problem? Please describe.

In order to discover bugs similar to #18, some type of test should be done with the actual storage service against the plug-in.

For #18, the issue can only be discovered with a large amount of data transferred with the application+plug-in and the storage service.

Describe the solution you'd like
Integration test that brings up the actual service, e.g. minio, and tests all APIs with a large amount of data.

Describe alternatives you've considered
NA

Additional context

The issue discovered in #18 was when over 20 files were downloaded from Minio, so we should consider the number of data and the size of data per API call.

[MinIO] GetObjectAsync throws System.ObjectDisposedException

Description

With MinIO plug-in, the GetObjectAsync call throws the following exceptions:

Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
   at System.Net.Sockets.NetworkStream.<ThrowIfDisposed>g__ThrowObjectDisposedException|63_0()
   at System.Net.Sockets.NetworkStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, Boolean async, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at Monai.Deploy.Storage.MinIO.MinIoStorageService.<>c__DisplayClass9_0.<<GetObjectAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Unhandled exception. System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Minio.MinioClient.ExecuteTaskCoreAsync(IEnumerable`1 errorHandlers, HttpRequestMessageBuilder requestMessageBuilder, CancellationToken cancellationToken, Boolean isSts) in /root/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 526
   at Minio.MinioClient.StatObjectAsync(StatObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 1093
   at Minio.MinioClient.getObjectHelper(GetObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/Helper/OperationsHelper.cs:line 50
   at Monai.Deploy.Storage.MinIO.MinIoStorageService.GetObjectUsingClient(MinioClient client, String bucketName, String objectName, Action`1 callback, CancellationToken cancellationToken) in /home/vicchang/sc/monai/monai-deploy-storage/src/Plugins/MinIO/MinIoStorageService.cs:line 305
   at Monai.Deploy.Storage.MinIO.MinIoStorageService.GetObjectAsync(String bucketName, String objectName, CancellationToken cancellationToken) in /home/vicchang/sc/monai/monai-deploy-storage/src/Plugins/MinIO/MinIoStorageService.cs:line 73
   at Program.Main(String[] args) in /home/vicchang/sc/monai/monai-deploy-storage/src/test/Program.cs:line 29
   at Program.<Main>(String[] args)

Steps to reproduce

Please share a clear and concise description of the problem.

Call the GetObjectAsync method multiple times, >20 and it will throw an exception.

Expected behavior

GetObjectAsync shall be able to return the stream.

Actual behavior

Throws exception.

Configuration

MONAI Deploy Storage Library version/commit: 0.1.0
OS and version (distro if applicable): ubuntu 20.04
Kubernetes version (if applicable): NA
Docker version (if applicable): NA
Installation source (NGC, Dockerhub, or something else): NuGet & source clone

Regression?

Yes, before the API change.

Add Mino Admin Health Checks

Is your feature request related to a problem? Please describe.
Add Mino Admin Health Checks

Describe the solution you'd like
Add Mino Admin Health Checks

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Support for seaweedfs

Is your feature request related to a problem? Please describe.

Given that minio is AGPLv3 and requires paid license for commercial use. Please consider support for seaweeds.

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Describe the solution you'd like

Or support other open-source storage systems.

VerifyObjectExistsAsync bug

Description

VerifyObjectExistsAsync is supposed to validate file and path but doesn't use key which represents the file. so doesn't verify the objects at all.
reward function to make it work correctly

Description

Argo error with Access Denied when attempting to download data from Minio.

Steps to reproduce

Start a new workflow using Argo task plug-in.
...

Expected behavior

The Argo workflow executes the entire pipeline.

Actual behavior

Argo fails to download data from Minio with an access denied error.

Configuration

MONAI Deploy Storage Library version/commit: 0.2.0

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.