Git Product home page Git Product logo

Comments (15)

kaicode avatar kaicode commented on August 10, 2024 1

Thank you for reporting this bug.

The errors in the first upload are caused by Snowstorm attempting to expand value sets as they are imported in an attempt to validate them. This does not always work well because some value sets refer to other nested value sets that are not yet imported. I intend to remove this validation so that the first import completes cleanly.

It looks like the error in the second upload may also be caused by an issue with that validation. I will look into this.

from snowstorm.

kaicode avatar kaicode commented on August 10, 2024 1

This will not be fixed quickly. I estimate the end of July.

from snowstorm.

FJGL avatar FJGL commented on August 10, 2024 1

@kaicode The version is just that:

  ------------------------------------------------------------
  ?  HAPI FHIR 6.4.4 - Command Line Tool
  ------------------------------------------------------------

"We have detected a problem because we need to perform the upload remotely and it fails with an exception related to the maximum size of a JSON request."

To reproduce the issue yo must try with a "remote" file:

.\hapi-fhir-cli upload-terminology -s 1GB -d Loinc_2.76.zip -v r4 -t http://192.168.0.26:8888/snowstorm/fhir -u http://loinc.org/

instead of a local reference:

hapi-fhir-cli upload-terminology -d Loinc_2.76.zip -v r4 -t http://localhost:8080/fhir -u http://loinc.org
(" File size is greater than 10 MB - Going to use a local file reference instead of a direct HTTP transfer. Note that this will only work when executing this command on the same server as the FHIR server itself.")

We have investigated the issue amd it seems that is a problem with the HAPI-FHIR version 6.6.2 included with snowstorm-x that is resolved with the 6.8.5 release or later,

hapifhir/hapi-fhir#5204

We have tested it successfully by rebuilding Snowstorm-X 9.0.1 with the HAPI-FHIR dependencies updated to 6.8.5. However, we are unsure of the impact of this on the rest of the FHIR functionality.

from snowstorm.

FJGL avatar FJGL commented on August 10, 2024

Hi.
Can you give us a timeframe of the version of Snowstorm that fixes this Loinc import?
No strings attached of course, It's just to plan the priorities of our development.

Thanks in advance.

from snowstorm.

kaicode avatar kaicode commented on August 10, 2024

The tech team, who are in charge of the Snowstorm release schedule, plan to make a public release at the start of August and another at the end of September.
I raised a PR for this issue about a month ago but unfortunately it did not make the August release candidate that's just gone into testing. It should make it into the August release.

@FJGL how pressing are your timelines for this fix? Creating a fix release in the SnowstormX project could be a short term workaround.

from snowstorm.

FJGL avatar FJGL commented on August 10, 2024

from snowstorm.

FJGL avatar FJGL commented on August 10, 2024

Hi.
Just to confirm that this bug is fixed in the public release at the end of September (any scheduled date?).

Thanks.

from snowstorm.

kaicode avatar kaicode commented on August 10, 2024

A Snowstorm version 9.0.0 release candidate has been created in the release-candidate branch including the LOINC package loading fixes. The release candidate will go through the usual testing and is scheduled to be deployed to our production environments on the 27th September. This release has breaking changes because it requires JDK 17 (Snowstorm previously required JDK 11).

You have the option of testing the release candidate directly by checking out the release-candidate branch and building the project. The fastest way is to build offline without tests: mvn -o clean package -Dmaven.test.skip=true.

from snowstorm.

FJGL avatar FJGL commented on August 10, 2024

We have built the release candidate 9.0.0 and it now successfully loads the Loinc_2.72 distribution without errors in the log the first or subsequent times, but it does not load the last version of LOINC 2.76.

2023-09-21 08:08:33.25 [main] ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
"resourceType": "OperationOutcome",
"issue": [ {
"severity": "error",
"code": "processing",
"diagnostics": "HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/MultiAxialHierarchy/MultiAxialHierarchy.csv]"
} ]
}
2023-09-21 08:08:33.26 [main] ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.UnprocessableEntityException: HTTP 422 : HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/MultiAxialHierarchy/MultiAxialHierarchy.csv]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:310)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:370)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:541)
at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1320)
at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:230)
at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:134)
at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:266)
at ca.uhn.fhir.cli.App.main(App.java:43)
2023-09-21 08:08:33.26 [Thread-0] INFO ca.uhn.fhir.cli.App HAPI FHIR is shutting down...

We assume it's because of the comment in using-the-fhir-api.md: "N.B. The new filename since Loinc 2.73 doesn't work yet."

Any idea when versions of Loinc later than 2.72 can be used?

TIA

from snowstorm.

FJGL avatar FJGL commented on August 10, 2024

The release candidate will go through the usual testing and is scheduled to be deployed to our production environments on the 27th September.

Any news about the deployment?. Has everything gone well or will be necessary a bugfix 9.0.1 ?

Just to have an estimate of the release date of the Release Candidate.

Regards.

from snowstorm.

kaicode avatar kaicode commented on August 10, 2024

I have just published SnowstormX 9.0.1 with the fix for loading LOINC 2.73 and above.
https://github.com/IHTSDO/snowstorm-x/releases/tag/9.0.1

Please let us know if this works for you and we will get this change into a future Snowstorm release.

from snowstorm.

FJGL avatar FJGL commented on August 10, 2024

Hello.

Yes, it works! and we are very interested in having that change included in the next Snowstorm release.
( It would be nice to have an approximate date :) )

We have loaded Loinc versions 2.74 and 2.76 in a clean Elasticsearch installation without problems.

Just for the sake of completeness if we load version 2.72 there is an exception:

ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
"resourceType": "OperationOutcome",
"issue": [ {
"severity": "error",
"code": "processing",
"diagnostics": "HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/ComponentHierarchyBySystem/ComponentHierarchyBySystem.csv]"
} ]
}
Error during execution:
ca.uhn.fhir.rest.server.exceptions.UnprocessableEntityException: HTTP 422 : HAPI-0862: Could not find the following mandatory files in input: [AccessoryFiles/ComponentHierarchyBySystem/ComponentHierarchyBySystem.csv]
....

We understand that it is no longer compatible with Loinc versions < 2.73.

EDIT: But...

We have detected a problem because we need to perform the upload remotely and it fails with an exception related to the maximum size of a JSON request.

.\hapi-fhir-cli upload-terminology -s 1GB -d Loinc_2.76.zip -v r4 -t http://192.168.0.26:8888/snowstorm/fhir -u http://loinc.org
...

c.u.f.c.UploadTerminologyCommand Adding ZIP file: Loinc_2.76.zip
c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...

ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
"resourceType": "OperationOutcome",
"issue": [ {
"severity": "error",
"code": "processing",
"diagnostics": "HAPI-0450: Failed to parse request body as JSON resource. Error was: HAPI-1861: Failed to parse JSON encoded FHIR content: String length (20046491) exceeds the maximum length (20000000)"
} ]
}

ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 : HAPI-0450: Failed to parse request body as JSON resource. Error was: HAPI-1861: Failed to parse JSON encoded FHIR content: String length (20046491) exceeds the maximum length (20000000)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:310)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:370)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:541)
at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1320)
at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:230)
at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:134)
at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:266)
at ca.uhn.fhir.cli.App.main(App.java:43)

from snowstorm.

kaicode avatar kaicode commented on August 10, 2024

@FJGL what version of the hapi-fhir-cli tool you are using?
I am using version 6.4.4 and it seems to upload without an issue. It looks like 6.10.0 is the latest on the homebrew package manager. Perhaps I would need to upgrade in order to reproduce the issue you are seeing?

This is my output:

$ hapi-fhir-cli upload-terminology -d Loinc_2.76.zip -v r4 -t http://localhost:8080/fhir -u http://loinc.org
------------------------------------------------------------
🔥  HAPI FHIR 6.4.4 - Command Line Tool
------------------------------------------------------------
Process ID                      : [email protected]
Max configured JVM memory (Xmx) : 4.0GB
Detected Java version           : 19.0.2
------------------------------------------------------------
2023-12-18 17:52:03.79 [main] INFO  c.u.f.c.LogbackUtil Logging configuration set from file /logback-cli-on.xml
2023-12-18 17:52:04.33 [main] INFO  c.u.f.c.UploadTerminologyCommand Reading /Users/kai/Downloads/Loinc_2.76.zip
2023-12-18 17:52:04.33 [main] INFO  c.u.f.c.UploadTerminologyCommand Adding ZIP file: Loinc_2.76.zip
2023-12-18 17:52:04.45 [main] INFO  c.u.f.c.UploadTerminologyCommand File size is greater than 10 MB - Going to use a local file reference instead of a direct HTTP transfer. Note that this will only work when executing this command on the same server as the FHIR server itself.
2023-12-18 17:52:04.51 [main] INFO  c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2023-12-18 17:59:48.37 [main] INFO  c.u.f.c.UploadTerminologyCommand Upload complete!
2023-12-18 17:59:48.42 [Thread-0] INFO  ca.uhn.fhir.cli.App HAPI FHIR is shutting down...

from snowstorm.

kaicode avatar kaicode commented on August 10, 2024

@FJGL thank you for debugging this and finding a fix that should work.
The upcoming Snowstorm version 10 uses HAPI version 6.9.6 or later. It sounds like this will resolve the upload issue.
@CoderMChu please could you let us know when Snowstorm 10 will be published as a public release?

from snowstorm.

kaicode avatar kaicode commented on August 10, 2024

This is fixed in Snowstorm version 10.2.1, released today:
https://github.com/IHTSDO/snowstorm/releases/tag/10.2.1

from snowstorm.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.