The main objective of the Lab Results Adaptor is to hide complex legacy standards and instead present a simple and consistent interface aligned to current NHSD national standards. The adaptor removes the requirement for a GP System to handle the complexities of EDIFACT and MESH messaging.
At a high level, the Lab Results Adaptor exposes a queue (AMQP 1.0) from which the GP System can consume Pathology and Screening medical reports in FHIR STU3 format.
See the Resources section for links to the underlying services and standards.
Adaptor handles the following types of EDIFACT messages:
- Pathology
NHS003
- Screening
NHS004
and replies with an NHS001
(NHSACK) message if requested.
Adaptor downloads EDIFACT messages from MESH mailbox (PATH_MEDRPT_V3
and SCRN_BCS_MEDRPT_V4
MESH workflows) using MESH Client component and puts them on an internal Inbound MESH Queue
. The Message Translator component fetches messages off the queue, translates the EDIFACT into FHIR format and puts the result onto the Outbound GP Queue
where other GP Supplier apps can consume medical reports from.
In the end, if the EDIFACT Interchange Header has requested such, an NHSACK confirmation will be sent back to the laboratory MESH mailbox.
Adaptor's EDIFACT to FHIR translation steps are built for high performance and resiliency by applying horizontal scaling techniques. However, the MESH consumer's step performance cannot be increased this way. Due to the nature of MESH, if there are multiple adaptor instances, only one randomly chosen instance will download messages and put them on the Inbound MESH Queue
.
EDIFACT messages are downloaded from MESH using MESH Client component. Due to specifics on how MESH works, the following design has been implemented.
The MESH Client component is triggered at a given interval (that is specified in the app configuration) where it downloads all messages matching Pathology and Screening workflows and sends back an infrastructure ACK telling MESH to delete downloaded EDIFACT messages. Even if there are multiple instances of the adaptor running, only one of them will be downloading messages at a given time.
When an EDIFACT message is downloaded from MESH using the MESH Client component, its structure is not verified at this step. The message is stored on the internal Inbound MESH Queue
and MESH is instructed that the file can be deleted since the message has been successfully downloaded.
Two error scenarios are possible at this stage:
-
Message has been downloaded but the adaptor was unable to put it on the
Inbound MESH Queue
Message has not been put on the queue, nor MESH has been instructed to delete it. Apart from error log entry, there is no side effect. The operation is retried on the next MESH scan cycle.
-
Message has been downloaded and placed on
Inbound MESH Queue
but an error occurred while notifying MESH that the message can be deletedMessage has been put on the queue, but MESH has not been instructed to delete it. There are no side effects apart from error log entry. The message has not been deleted from MESH, so it will be fetched again on the next MESH scan cycle. Since the message has already been put on
Inbound MESH Queue
, it will be processed again causing duplicates to appear on theOutbound GP Queue
. Handling duplicates is described in the duplicates section.
Once the message is on the Inbound MESH Queue
the Message Translator component picks each one up and performs the following steps:
-
Initial verification
This step parses the EDIFACT message and verifies the general structure of the message and can result in two types of errors:
-
Interchange Header (UNB) is missing mandatory fields (sender, recipient, sequence number, NHSACK request flag).
Since the header can't be parsed, an NHSACK can't be produced to inform the laboratory about the error. This is a critical error and manual intervention is required. Message is rejected causing it to be placed back on the
Inbound MESH Queue
. It is picked up again for retry until the MQ max retry value is reached. After that, message is placed on the Dead Letter Queue and manual intervention is required. -
Interchange Trailer (UNZ) message count or Message Header (UNH) or Message Trailer (UNT) can't be parsed.
Since the Interchange Header was parsed, all the information required to produce an NHSACK is available. If Interchange Header has the NHSACK request flag set, an NHSACK is produced and sent back to the laboratory MESH mailbox.
-
-
Translation
At this step an EDIFACT has already been parsed and split into messages. Each message is translated into a FHIR Bundle. Any error occurring at this step is logged, and if an NHSACK is requested, appropriate information is set in it. Because of the error, no FHIR Bundle is created for that single message. Other messages are not affected and can still be processed. At the end of this step, a summary NHSACK is generated containing information about all the successes and failures of EDIFACT messages processing.
-
Sending to
Outbound GP Queue
All individual EDIFACT messages that have been successfully translated into FHIR are sent to
Outbound GP Queue
. If an error occurs at this step, the whole EDIFACT is rejected and it returns to theInbound MESH Queue
to be picked up again until MQ max retry value is reached. After that, the EDIFACT is placed on the Dead Letter Queue and manual intervention is required. -
Sending NHSACK to
Outbound MESH Queue
All individual FHIR messages have been sent to the
Outbound GP Queue
. If an NHSACK was requested, it has been generated and sent toOutbound MESH Queue
. If an error occurs at this step, the whole EDIFACT is rejected and it will be returned to theInbound MESH Queue
to be picked up again until MQ max retry value is reached. After that, the EDIFACT is placed on the Dead Letter Queue and manual intervention is required. Since the message has been already put onInbound MESH Queue
, it will be processed again causing duplicates to appear on theOutbound GP Queue
. Handling duplicates is described in the duplicates section.
Adaptor provides detailed logs while processing each EDIFACT. Each EDIFACT that goes through the adapter is given a new randomly generated CorrelationId
. This value is available in every log line that is created while processing that EDIFACT and also as an AMQP header (check here) on the Outbound GP Queue
.
In rare cases, the adaptor can produce duplicate messages on the Outbound GP Queue
. It is GP Supplier's responsibility to identify and handle duplicates when fetching messages from the queue. The AMQP message Checksum
header should be used to identify duplicates (see Outbound GP Queue specification).
When using the adaptor, the GP Supplier will receive medical reports on the Outbound GP Queue
(AMQP 1.0) in the form of a FHIR STU3 Bundle resources. Examples can be found here
The Outbound GP Queue
from where the FHIR-formatted medical reports can be fetched from is an AMQP 1.0 queue. The payload of the queue message is the FHIR Bundle resource accompanied with custom headers:
CorrelationId
- a unique identifier assigned for a given inbound EDIFACT message that can be used to trace down message processing and issues in adaptor logs. Keep in mind that a single EDIFACT can contain multiple messages, which will produce multiple queue messages having the sameCorrelationId
. Example:83035310E0B54283A5242C8EC9CA5FD6
Checksum
- an MD5 checksum of the inbound EDIFACT message. This value should be used to handle duplicates on the GP Supplier Applications end.MessageType
- one of [Pathology
,Screening
] indicating the source EDIFACT message type
Examples of both Pathology and Screening messages are provided as part of the adaptor's User Acceptance Tests. Each example is built from 2 files:
<example-id>.edifact.dat
- EDIFACT message provided by the laboratory<example-id>.fhir.json
- message translated to FHIR format
Example input EDIFACT files can be found here with the corresponding FHIR translations found here
Pathology examples can be found here Screening examples can be found here
Pathology Messaging - EDIFACT API
Bowel Cancer Screening - EDIFACT API
Pathology & Diagnostics Information Standards Collaboration Space
National Pathology FHIR Messaging Specifications
All configuration options can be found in the application.yml file. The adaptor reads its configuration from environment variables to override the defaults. The following sections describe the environment variables used to configure the adaptor.
Variables without a default value and not marked optional MUST be defined for the adaptor to run.
Environment Variable | Default | Description |
---|---|---|
LAB_RESULTS_OUTBOUND_SERVER_PORT | 8080 | The port on which the adaptor management endpoints will run |
LAB_RESULTS_LOGGING_LEVEL | INFO | Application logging level. One of: DEBUG, INFO, WARN, ERROR. The level DEBUG MUST NOT be used when handling live patient data. |
LAB_RESULTS_LOGGING_FORMAT | (*) | Defines how to format log events on stdout |
(*) The adaptor uses logback (http://logback.qos.ch/). The built-in logback.xml
defines the default log format. This value can be overridden using the LAB_RESULTS_LOGGING_FORMAT
environment variable.
You can provide an external logback.xml
file using the -Dlogback.configurationFile
JVM parameter.
Environment Variable | Default | Description |
---|---|---|
LAB_RESULTS_MESH_OUTBOUND_QUEUE_NAME | lab_results_mesh_outbound | The name of the internal outbound (to MESH) message queue |
LAB_RESULTS_MESH_INBOUND_QUEUE_NAME | lab_results_mesh_inbound | The name of the internal inbound (from MESH) message queue |
LAB_RESULTS_GP_OUTBOUND_QUEUE_NAME | lab_results_gp_outbound | The name of the outbound (to GP System) message queue |
LAB_RESULTS_AMQP_BROKERS | amqp://localhost:5672 | A comma-separated list of URLs to AMQP brokers (*) |
LAB_RESULTS_AMQP_USERNAME | (Optional) username for the AMQP server | |
LAB_RESULTS_AMQP_PASSWORD | (Optional) password for the AMQP server | |
LAB_RESULTS_AMQP_MAX_REDELIVERIES | 3 | The number of times a message will be retried to be delivered to consumer. After exhausting all retries, it will be put on DLQ.<queue_name> dead-letter queue. The default JMS configuration is (-1) disabled. |
(*) Active/Standby: The first broker in the list is always used unless there is an error, in which case the other URLs will be used. At least one URL is required.
The adaptor configuration for MongoDB can be configured two ways, using a connection string or providing individual properties. This is to accommodate differences in the capabilities of deployment automation frameworks and varying environments.
Option 1: If LAB_RESULTS_MONGO_HOST
is defined then the adaptor forms a connection string from the following properties:
Environment Variable | Default | Description |
---|---|---|
LAB_RESULTS_MONGO_DATABASE_NAME | labresults | Database name for MongoDB |
LAB_RESULTS_MONGO_HOST | MongoDB host | |
LAB_RESULTS_MONGO_PORT | MongoDB port | |
LAB_RESULTS_MONGO_USERNAME | (Optional) MongoDB username. If set then password must also be set. | |
LAB_RESULTS_MONGO_PASSWORD | (Optional) MongoDB password | |
LAB_RESULTS_MONGO_OPTIONS | (Optional) MongoDB URL encoded parameters for the connection string without a leading ? | |
LAB_RESULTS_MONGO_TTL | P30D | (Optional) Time-to-live value |
Option 2: If LAB_RESULTS_MONGO_HOST
is undefined then the adaptor uses the connection string provided:
Environment Variable | Default | Description |
---|---|---|
LAB_RESULTS_MONGO_DATABASE_NAME | labresults | Database name for MongoDB |
LAB_RESULTS_MONGO_URI | mongodb://localhost:27017 | MongoDB connection string |
Trust Store Configuration
These optional properties configure a trust store with private CA certificates. This trust store does not replace Java's default trust store. At runtime the application adds these additional certificates to the default trust store. See [OPERATING.md - AWS DocumentDB TLS configuration](OPERATING.md#AWS DocumentDB TLS configuration) for more information.
Environment Variable | Default | Description |
---|---|---|
LAB_RESULTS_SSL_TRUST_STORE_URL | (Optional) URL of the trust store JKS. The only scheme currently supported is s3:// |
|
LAB_RESULTS_SSL_TRUST_STORE_PASSWORD | (Optional) Password used to access the trust store |
Configure the MESH API connection using the following environment variables:
Environment Variable | Default | Description |
---|---|---|
LAB_RESULTS_MESH_MAILBOX_ID | The mailbox id used by the adaptor to send and receive messages. This is the sender of outbound messages and the mailbox where inbound messages are received. | |
LAB_RESULTS_MESH_MAILBOX_PASSWORD | The password for LAB_RESULTS_MESH_MAILBOX_ID | |
LAB_RESULTS_MESH_SHARED_KEY | A shared key used to generate auth token and provided by MESH operator (OpenTest, PTL, etc) | |
LAB_RESULTS_MESH_HOST | The Complete URL with trailing slash of the MESH service. For example: https://msg.int.spine2.ncrs.nhs.uk/messageexchange/ | |
LAB_RESULTS_MESH_CERT_VALIDATION | true | "false" to disable certificate validation of SSL connections |
LAB_RESULTS_MESH_ENDPOINT_CERT | The content of the PEM-formatted client endpoint certificate | |
LAB_RESULTS_MESH_ENDPOINT_PRIVATE_KEY | The content of the PEM-formatted client private key | |
LAB_RESULTS_MESH_SUB_CA | The content of the PEM-formatted certificate of the issuing Sub CA. Empty if LAB_RESULTS_MESH_CERT_VALIDATION is false | |
LAB_RESULTS_MESH_RECIPIENT_MAILBOX_ID_MAPPINGS | (*) The mapping between each NHSACK recipient to its corresponding MESH Mailbox ID mapping. There is one mapping per line and an equals sign (=) separates the code and mailbox id. For example: "000000004400001=A6840385\000000024600002=A0047392" | |
LAB_RESULTS_SCHEDULER_ENABLED | true | Enables/disables automatic MESH message downloads |
(*) 15 digit NHS Sender ID required for successful NHSACK dispatch
The following three variables control how often the adaptor performs a MESH polling cycle. During a polling cycle the adaptor will download and acknowledge up to "the first 500 messages" (a MESH API limit).
Important: If the MESH mailbox uses workflows other than PATH_MEDRPT_V3
and SCRN_BCS_MEDRPT_V4
, then these messages must be downloaded and acknowledged by some other means in a timely manner.
The adaptor will skip messages with other workflow ids leaving them in the inbox. If more than 500 "other" messages
accumulate the adaptor wil no longer receive new inbound GP Links messages.
Environment Variable | Default | Description |
---|---|---|
LAB_RESULTS_MESH_CLIENT_WAKEUP_INTERVAL_IN_MILLISECONDS | 60000 | The time period (in milliseconds) between when each adaptor instance "wakes up" and attempts to obtain the lock to start a polling cycle |
LAB_RESULTS_MESH_POLLING_CYCLE_MINIMUM_INTERVAL_IN_SECONDS | 300 | The minimum time period (in seconds) between MESH polling cycles |
LAB_RESULTS_MESH_POLLING_CYCLE_DURATION_IN_SECONDS | 285 | The duration (in seconds) fo the MESH polling cycle |
The MESH API specifies that a MESH mailbox should be checked "a maximum of once every five minutes". The variable
LAB_RESULTS_MESH_POLLING_CYCLE_MINIMUM_INTERVAL_IN_SECONDS
controls how often the adaptor will check its mailbox for new
messages. This should not be set to less than 300 seconds. A time lock in the database prevents the polling cycle from
running more often than this minimum interval. Each adaptor instance will wake up every
LAB_RESULTS_MESH_CLIENT_WAKEUP_INTERVAL_IN_MILLISECONDS
to try this time lock. Therefore, the maximum polling cycle interval
is the sum of these two values.
Only one instance of the adaptor runs the polling cycle at any given time to prevent duplicate processing. The value
LAB_RESULTS_MESH_POLLING_CYCLE_DURATION_IN_SECONDS
prevents one polling cycle from overrunning into the next time interval.
This value must always be less than LAB_RESULTS_MESH_POLLING_CYCLE_MINIMUM_INTERVAL_IN_SECONDS
.
Refer to OPERATING.md for tips about how to operate the adaptor in the production environment.
The following sections provide the necessary information to develop the Lab Results adaptor.
The adaptor configuration has sensible defaults for local development. Some overrides might be required where the "secure by default" principle takes precedence:
LAB_RESULTS_MESH_CERT_VALIDATION: "false"
- if using fake-mesh then certificate validation must be disabledLAB_RESULTS_LOGGING_LEVEL: "DEBUG"
- consider using DEBUG logging while developing
- Install a Java JDK 11. AdoptOpenJdk is recommended.
- Install IntelliJ
- Install the Lombok plugin. Intellij should prompt you to enable annotation processing, ensure you enable this.
- Install Docker
-
Clone this repository
-
Open the cloned
integration-adaptor-lab-results
folder -
Click pop-up that appears: (import gradle daemon)
-
Verify the project structure
Project structure -> SDKs -> add new SDK -> select adoptopenjdk-11.jdk/Contents/Home (or alternative location) -> Project SDK -> Java 11 (11.0.9) -> Module SDK -> Java 11 (11.0.9)
- mongo: MongoDB Docker images
- rmohr/activemq: ActiveMQ Docker images
- nhsdev/fake-mesh: fake-mesh (mock MESH API server) Docker images
Run docker-compose up mongodb activemq fake-mesh
From IntelliJ
Navigate to: IntegrationAdapterLabResultsApplication -> right click -> Run
Inside a container
docker-compose build lab-results
docker-compose up lab-results
All quality checks
./gradlew check -x test -x integrationTest
This runs Spotbugs and Checkstyle to perform a static analysis to find potential bugs and checks to see if the code style conforms to the Java Coding standards.
Checkstyle checks
./gradlew checkstyleIntTest checkstyleMain checkstyleTest
Spotbugs checks
./gradlew spotbugsMain
SpotbugsMain is the only Spotbugs task we run when executing ./gradlew check
.
All Tests
./gradlew check -x spotbugsMain -x spotbugsIntTest -x spotbugsTest -x checkstyleMain -x checkstyleIntTest -x checkstyleTest
Unit Tests
This will run all tests inside the src/test folder.
./gradlew test
Integration Tests
A separate source folder src/intTest contains integration tests. To run the integration tests use:
./gradlew integrationTest
All Tests and Checks
This command will run all tests (unit & integration) and all static analysis and code style checks.
The --continue
flag ensures that all tests and checks will run.
./gradlew check --continue
Run lab-results
, mongodb
, activemq
and fake-mesh
containers:
docker-compose build
docker-compose up
Run test script:
cd ./release/tests
./send_message.sh
that will send an example EDIFACT message to the fake-mesh
container. The lab-results
container app will fetch this message and put the translated to FHIR format result on the activemq
container's lab_results_gp_outbound
queue.
Check ActiveMQ admin console (user: admin, password: admin) to download the result which will match the example FHIR message. Additionally mesh/mesh.sh
can be used to fetch the generated NHSACK from fake-mesh
.
Notice that depending on the lab-result
configuration, it can take several seconds until fake-mesh
is scanned for new messages.
To view data in MongoDB:
- Download Robo 3T
- Open Robo 3T -> Create new connection with details as below:
- Type: Direct Connection
- Name: labresults
- Address: localhost : 27017
- View adaptor collections by navigating to labresults -> collections -> (select any collection)
To view messages in the ActiveMQ Web Console:
- Open browser and navigate to: http://localhost:8161/
- Username: admin
- Password: admin
- Click manage ActiveMQ broker
- Click Queues tab
- Select desired queue
- Select a message ID to display information of message
A mesh.sh
bash script exists for testing or debugging MESH. For more information see: mesh/README.md
The release/tests/send_message.sh
is a test script that sends an example message to MESH mailbox.
A mock implementation of the MESH API is available for local development. The latest version is in Github at mattd-kainos/fake-mesh. It is a fork of jamespic/fake-mesh.
The nhsdev Docker Hub hosts released fake-mesh images.
Ensure that you follow the agreed Java Coding standards on the project when developing and code reviewing the adaptor. Feel free to update the documentation if you feel anything is incorrect or missing.