Git Product home page Git Product logo

imaging-ingestion's Introduction

Alvearie Imaging Ingestion

The Imaging Ingestion component brings a number of capabilities to the Alvearie ecosystem. It provides all of the necessary capability to extend the reach of existing enterprise medical Imaging systems to a modern hybrid cloud. This component leverages a number of Cloud Native Computing Foundation (CNCF) projects to cloud-enable standards-based medical imaging application interfaces such as DICOM Message Service Element (DIMSE) and DICOMweb. Being composed of a number of flexible subcomponents, it is possible to extend enterprise imaging systems to the cloud with custom, fit for purpose deployments. This is possible at speed, low cost, and little or zero integration effort.

Some potential usages:

  • Perform lightweight secure bridging of DICOM from the enterprise imaging zone to a cloud availability zone.
  • Utilize cloud-based, fine-grained, secure storage spaces for segregating DICOM data to be used for clinical or scientific purposes.
  • Distribute DICOM data to a number of different medical imaging subscribers
  • Generate FHIRv4 ImagingStudy resources from ingested content and publish to a FHIRv4 based patient logitudinal record.
  • Distribute imaging study arrival and update notifications to imaging study subscribers

Imaging Ingestion Approach

Medical Imaging clinical workflow is inherently event-driven, and solutions must be real-time. Starting with image acquisition, DICOM instances are elements of an ordered sequence of images captured by an imaging modality. Collectively, this sequence makes up an imaging modality series. There may be many imaging modality series, from any number of imaging modalities, within an imaging study. Conceptually, each DICOM instance is not unlike an observation from an IoT device. Each DICOM instance is an atomic observation (fact) that collectively constitute an imaging study (behavior).

The design approach of the imaging ingestion component is to allow for the real-time ingestion and distribution of DICOM instances (facts) to a set of instance subscribers. It aggregates DICOM instances into revisions of DICOM studies (behaviors) that are distributed to study subscribers. In these regards, it supports a number of imaging standards including DICOMweb, DIMSE, and FHIRv4. Additionally, a DIMSE (OSI level 4) proxy is provided for both unidirectional and bidirectional (OSI level 7) secure communication between the cloud and enterprise imaging systems. The proxy uses NATS as the messaging protocol.

From a technology and architecture perspective, the approach is to deliver a number of small subcomponents that interact with one another in a request-response pattern at the edges, and an event-first pattern in the core. Every subcomponent is designed and developed to be small, fast, and portable. Each subcomponent is written in either Golang or Java with all of the Java authored components using Quarkus to natively optimize runtime behavior. Each subcomponent is exposed as a Kubernetes Custom Resource Definition (CRD) for a cloud-native operating model. A Kubernetes Operator is provided to both manage the Custom Resource (CR) objects, as well as configure the event-driven and messaging frameworks that the subcomponents participate within. The operator provides support for the Knative Serving and Eventing event-driven framework.

Underlying support for DICOM standards is consumed from the dcm4che project. The imaging ingestion subcomponents wrap and extend these capabilities to provide Quarkus runtime components that are operated in a cloud-native model.

Our initial goal is to provide enough capability to deliver a rapid proof-of-value. Alvearie Imaging ingestion is designed for flexible deployment options and implemented to run small, run fast, and run anywhere.

Component Architecture

AlvearieImagingIngestionArchitecture

The imaging ingestion component is currently comprised of six (6) subcomponents:

  1. DIMSE Proxy enables a DIMSE Application Entity (AE) point-of-presence in the enterprise imaging zone and/or within the Kubernetes cluster.
  2. DIMSE Ingestion Service provides a proxied DIMSE Application Entity (AE) in the Kubernetes cluster for C-STORE and C-FIND operations on a storage space.
  3. DICOMweb Ingestion Service for DICOMweb QIDO-RS, WADO-RS, and STOW-RS access to a storage space.
  4. DICOM Event Driven Ingestion maintains a manifest of all DICOM data across all ingestion storage spaces.
  5. DICOM Instance Binding for fan-out notification of DICOM instance data.
  6. DICOM Study Binding for fan-out notification of DICOM studies.

The DICOM Instance Binding and DICOM Study Binding components are intended to enable consistent deployment and management of a number of different implementations. Implementations may range from DICOMweb, to FHIRv4, to DIMSE, to proprietary. Our development roadmap provides details on planned support for all medical imaging standard protocols.

Deployment Dependencies

The preferred deployment has the following deployment dependencies beyond Kubernetes core. There a number of ways in which these dependencies can be added to the cluster; depending upon the target platform. The preferred approach is to use Kubernetes operators whenever possible.

Dependency Function Open Source Operator Openshift OpenSource Available in OperatorHub.io Deployment Example
Certificate Manager Certificate Management cert-manager certified yes
Istio Service mesh for Knative and service security overlay Istio Operator OpenShift Service Mesh yes
Knative Serving and Eventing Serving elastic scalable microservices and Event-first interaction patterns between microservices Knative Operator OpenShift Serverless Operator yes
PostgreSQL Default backing store for imaging manifest data Crunchy Data PostgreSQL Operator certified yes
Any S3 compliant object storage DICOM storage spaces MinIO OpenShift Container Storage yes
NATS Messaging DIMSE to/from the enterprise imaging security zone and Kubernetes. This is optional and only needed for DIMSE support. NATS Operator Not certified no example

Installation and Deployment

The preferred deployment model for image ingestion is to use the provided Alvearie Imaging Ingestion Operator for installation and management of the subcomponents that are hosted within the Kubernetes cluster. The operator extends the Kubernetes API to provide a Custom Resource Definition for each of the imaging ingestion subcomponents. The documentation page of each component provide details and examples for the Custom Resources, Secret, and ConfigMap objects.

Installation and Deployment on Kubernetes procedure:

  1. Install the Alvearie Imaging Ingestion Operator
  2. Create a database for the imaging manifest data
  3. Declare a DICOM Event Driven Ingestion, configuring it to use the provided database
  4. Create a S3 Object bucket for each storage space needed
  5. Declare a DICOMweb Ingestion Service for each storage space
  6. Declare DICOM Instance Binding and DICOM Study Binding as needed for distribution of DICOM instances and DICOM study notifications.

Enabling DIMSE ingestion from the enterprise imaging zone to the storage space created in step 4 above In order to ingest DIMSE from the enterprise imaging zone to a imaging ingestion storage space, a proxy will need to be deployed within the enterprise imaging zone. This proxy will be used to project the DIMSE Ingestion Service into the zone. The proxy will use NATS to forward the C-STORE operations to the DIMSE Ingestion Service.

  1. Configure a NATS for a secure subject for the two subcomponents to communicate.
  2. Declare a DIMSE Ingestion Service in Kubernetes providing references to the NATS service and subject. If the subject is secured using NATS accounts, the JWT tokens for accessing subject will also be need to be provided as a Secret.
  3. Install a DIMSE Proxy in the enterprise imaging zone.
  4. Configure the DIMSE Proxy to use the NATS subject and tokens created by the operator in step 1.

imaging-ingestion's People

Contributors

coriaedu avatar eggebraa avatar mudm avatar pbhallam avatar rduggan-ibm avatar wlingley avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

imaging-ingestion's Issues

Can't change domain for DICOM Web services

When applying the example configuration mentioned in README.md of Alvearie DICOMweb services configuration the domain applied is always "example.com" so that external endpoint always looks like that: "wadoServiceExternalEndpoint: http://img-ingest-wado.imaging-ingestion.example.com"

The domain in the Istio Virtual Service is set accordingly.

To reproduce just apply configuration mentioned in the README of this repo.

I would also like to change protocol from http to https but that doesn't seem to possible.
Another observation I've made is that there is no external exposure for STOW, only WADO. Is that intended?

Knative 0.26 ( sources.knative.dev/v1 ) not supported

When using Knative v0.26 Imaging Operator deployment fails with error message:

controller-runtime.source if kind is a CRD, it should be installed before calling Start {"kind": "SinkBinding.sources.knative.dev", "error": "no matches for kind "SinkBinding" in version "sources.knative.dev/v1alpha2""}
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/source/source.go:117
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:167
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:681
2021-10-15T20:42:49.960Z ERROR controller-runtime.manager error received after stop sequence was engaged {"error": "no matches for kind "SinkBinding" in version "sources.knative.dev/v1alpha2""}

due to version change in source.knative.dev towards v1.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.