Comments (15)
Thank you for reporting this bug.
The errors in the first upload are caused by Snowstorm attempting to expand value sets as they are imported in an attempt to validate them. This does not always work well because some value sets refer to other nested value sets that are not yet imported. I intend to remove this validation so that the first import completes cleanly.
It looks like the error in the second upload may also be caused by an issue with that validation. I will look into this.
from snowstorm.
This will not be fixed quickly. I estimate the end of July.
from snowstorm.
@kaicode The version is just that:
------------------------------------------------------------
? HAPI FHIR 6.4.4 - Command Line Tool
------------------------------------------------------------
"We have detected a problem because we need to perform the upload remotely and it fails with an exception related to the maximum size of a JSON request."
To reproduce the issue yo must try with a "remote" file:
.\hapi-fhir-cli upload-terminology -s 1GB -d Loinc_2.76.zip -v r4 -t http://192.168.0.26:8888/snowstorm/fhir -u http://loinc.org/
instead of a local reference:
hapi-fhir-cli upload-terminology -d Loinc_2.76.zip -v r4 -t http://localhost:8080/fhir -u http://loinc.org
(" File size is greater than 10 MB - Going to use a local file reference instead of a direct HTTP transfer. Note that this will only work when executing this command on the same server as the FHIR server itself.")
We have investigated the issue amd it seems that is a problem with the HAPI-FHIR version 6.6.2 included with snowstorm-x that is resolved with the 6.8.5 release or later,
We have tested it successfully by rebuilding Snowstorm-X 9.0.1 with the HAPI-FHIR dependencies updated to 6.8.5. However, we are unsure of the impact of this on the rest of the FHIR functionality.
from snowstorm.
Hi.
Can you give us a timeframe of the version of Snowstorm that fixes this Loinc import?
No strings attached of course, It's just to plan the priorities of our development.
Thanks in advance.
from snowstorm.
The tech team, who are in charge of the Snowstorm release schedule, plan to make a public release at the start of August and another at the end of September.
I raised a PR for this issue about a month ago but unfortunately it did not make the August release candidate that's just gone into testing. It should make it into the August release.
@FJGL how pressing are your timelines for this fix? Creating a fix release in the SnowstormX project could be a short term workaround.
from snowstorm.
from snowstorm.
Hi.
Just to confirm that this bug is fixed in the public release at the end of September (any scheduled date?).
Thanks.
from snowstorm.
A Snowstorm version 9.0.0 release candidate has been created in the release-candidate branch including the LOINC package loading fixes. The release candidate will go through the usual testing and is scheduled to be deployed to our production environments on the 27th September. This release has breaking changes because it requires JDK 17 (Snowstorm previously required JDK 11).
You have the option of testing the release candidate directly by checking out the release-candidate
branch and building the project. The fastest way is to build offline without tests: mvn -o clean package -Dmaven.test.skip=true
.
from snowstorm.
We have built the release candidate 9.0.0 and it now successfully loads the Loinc_2.72 distribution without errors in the log the first or subsequent times, but it does not load the last version of LOINC 2.76.
2023-09-21 08:08:33.25 [main] ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
"resourceType": "OperationOutcome",
"issue": [ {
"severity": "error",
"code": "processing",
"diagnostics": "HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/MultiAxialHierarchy/MultiAxialHierarchy.csv]"
} ]
}
2023-09-21 08:08:33.26 [main] ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.UnprocessableEntityException: HTTP 422 : HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/MultiAxialHierarchy/MultiAxialHierarchy.csv]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:310)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:370)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:541)
at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1320)
at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:230)
at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:134)
at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:266)
at ca.uhn.fhir.cli.App.main(App.java:43)
2023-09-21 08:08:33.26 [Thread-0] INFO ca.uhn.fhir.cli.App HAPI FHIR is shutting down...
We assume it's because of the comment in using-the-fhir-api.md: "N.B. The new filename since Loinc 2.73 doesn't work yet."
Any idea when versions of Loinc later than 2.72 can be used?
TIA
from snowstorm.
The release candidate will go through the usual testing and is scheduled to be deployed to our production environments on the 27th September.
Any news about the deployment?. Has everything gone well or will be necessary a bugfix 9.0.1 ?
Just to have an estimate of the release date of the Release Candidate.
Regards.
from snowstorm.
I have just published SnowstormX 9.0.1 with the fix for loading LOINC 2.73 and above.
https://github.com/IHTSDO/snowstorm-x/releases/tag/9.0.1
Please let us know if this works for you and we will get this change into a future Snowstorm release.
from snowstorm.
Hello.
Yes, it works! and we are very interested in having that change included in the next Snowstorm release.
( It would be nice to have an approximate date :) )
We have loaded Loinc versions 2.74 and 2.76 in a clean Elasticsearch installation without problems.
Just for the sake of completeness if we load version 2.72 there is an exception:
ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
"resourceType": "OperationOutcome",
"issue": [ {
"severity": "error",
"code": "processing",
"diagnostics": "HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/ComponentHierarchyBySystem/ComponentHierarchyBySystem.csv]"
} ]
}
Error during execution:
ca.uhn.fhir.rest.server.exceptions.UnprocessableEntityException: HTTP 422 : HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/ComponentHierarchyBySystem/ComponentHierarchyBySystem.csv]
....
We understand that it is no longer compatible with Loinc versions < 2.73.
EDIT: But...
We have detected a problem because we need to perform the upload remotely and it fails with an exception related to the maximum size of a JSON request.
.\hapi-fhir-cli upload-terminology -s 1GB -d Loinc_2.76.zip -v r4 -t http://192.168.0.26:8888/snowstorm/fhir -u http://loinc.org
...
c.u.f.c.UploadTerminologyCommand Adding ZIP file: Loinc_2.76.zip
c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
"resourceType": "OperationOutcome",
"issue": [ {
"severity": "error",
"code": "processing",
"diagnostics": "HAPI-0450: Failed to parse request body as JSON resource. Error was: HAPI-1861: Failed to parse JSON encoded FHIR content: String length (20046491) exceeds the maximum length (20000000)"
} ]
}
ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 : HAPI-0450: Failed to parse request body as JSON resource. Error was: HAPI-1861: Failed to parse JSON encoded FHIR content: String length (20046491) exceeds the maximum length (20000000)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:310)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:370)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:541)
at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1320)
at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:230)
at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:134)
at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:266)
at ca.uhn.fhir.cli.App.main(App.java:43)
from snowstorm.
@FJGL what version of the hapi-fhir-cli tool you are using?
I am using version 6.4.4 and it seems to upload without an issue. It looks like 6.10.0 is the latest on the homebrew package manager. Perhaps I would need to upgrade in order to reproduce the issue you are seeing?
This is my output:
$ hapi-fhir-cli upload-terminology -d Loinc_2.76.zip -v r4 -t http://localhost:8080/fhir -u http://loinc.org
------------------------------------------------------------
🔥 HAPI FHIR 6.4.4 - Command Line Tool
------------------------------------------------------------
Process ID : [email protected]
Max configured JVM memory (Xmx) : 4.0GB
Detected Java version : 19.0.2
------------------------------------------------------------
2023-12-18 17:52:03.79 [main] INFO c.u.f.c.LogbackUtil Logging configuration set from file /logback-cli-on.xml
2023-12-18 17:52:04.33 [main] INFO c.u.f.c.UploadTerminologyCommand Reading /Users/kai/Downloads/Loinc_2.76.zip
2023-12-18 17:52:04.33 [main] INFO c.u.f.c.UploadTerminologyCommand Adding ZIP file: Loinc_2.76.zip
2023-12-18 17:52:04.45 [main] INFO c.u.f.c.UploadTerminologyCommand File size is greater than 10 MB - Going to use a local file reference instead of a direct HTTP transfer. Note that this will only work when executing this command on the same server as the FHIR server itself.
2023-12-18 17:52:04.51 [main] INFO c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2023-12-18 17:59:48.37 [main] INFO c.u.f.c.UploadTerminologyCommand Upload complete!
2023-12-18 17:59:48.42 [Thread-0] INFO ca.uhn.fhir.cli.App HAPI FHIR is shutting down...
from snowstorm.
@FJGL thank you for debugging this and finding a fix that should work.
The upcoming Snowstorm version 10 uses HAPI version 6.9.6 or later. It sounds like this will resolve the upload issue.
@CoderMChu please could you let us know when Snowstorm 10 will be published as a public release?
from snowstorm.
This is fixed in Snowstorm version 10.2.1, released today:
https://github.com/IHTSDO/snowstorm/releases/tag/10.2.1
from snowstorm.
Related Issues (20)
- Difficulty Integrating SNOMED CT APIs into Healthcare Application HOT 2
- Application fails to start: No setter found for property: index-prefix HOT 2
- Is it possible for a concept to exist without having ancestors? HOT 2
- Default module id not working HOT 4
- MRCM Range search - wrong language HOT 8
- ECL Bug, Zero In-Group Cardinality Matches Nothing HOT 1
- Snowstorm API SearchMode HOT 5
- Webhooks HOT 3
- Error installing Release 10.2.1 using docker-compose HOT 4
- Refset bulk update HOT 2
- Unable to load SNOMED CT into Docker instance HOT 3
- HTTPS Configuration for Apache Server and Swagger HOT 6
- CORS issue with Snowstorm despite updating nginx setup and application.properties HOT 1
- Loading data: Failed RF2 SNAPSHOT import on branch MAIN HOT 2
- SNOMED CT Browser makes API request without branch HOT 3
- Request for Clarification on Unlimited Pagination for {branch}/concepts Endpoint HOT 2
- Issue with updating DependantVersionEffectiveTime in CodeSystem Version HOT 4
- Own extension in FHIR CodeSystem HOT 2
- Clean install on Kubernetes failes HOT 13
- Missing @ApiResponse causes issues when trying to generate client HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snowstorm.