metadatacenter / cedar-cadsr-tools Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 2.14 MB

Ingestor to transform caDSR XML-based CDEs into CEDAR metadata

License: Other

Java 94.59% Perl 5.41%

cedar-cadsr-tools's People

Contributors

Watchers

cedar-cadsr-tools's Issues

Missing the skos:notation definition when generating CEDAR template field

CADSR-VS value sets not updating nightly in BioPortal

No error being reported but CADSR-VS value sets are not updated nightly.

Handle updated caDSR XML

As per #1 NCI will update their XML representation of CDEs to include the term source for controlled term values. Update CDE ingestor to handle this XML and generate value sets accordingly.

Requires #1

Selecting incorrect concept code for answers for CDEs generated from caDSR system

For CDEs, we are selecting the last entry for the values in DataElement/VALUEDOMAIN/PermissableValues/PermissableValues_ITEM/MEANINGCONCEPTS as the concept code for each answer to a CDE-based question. We should be selecting the first as the concepts get more general and less specific later in the list. The first element is the most specific.

Dealing with compound controlled terms in caDSR CDE value sets

Some CDEs have a complex controlled term that is made of several atomic concepts joined together.

As an example the label "Frozen Section Cell Pellet Formation" has a concept code C49338,C12508,C45813 where each code corresponds to a particular concept in NCIT:

C49338: Frozen Section Diagnosis
C12508: Cell
C45813: Pellet Formation

Dealing with non-NCIT terms in value sets

The caDSR XML does not indicate the source of controlled terms. Mostly are from NCIT but not all are.

Denise talked about potentially extending their XML to provide this information.

Add caDSR reader to release

The CDE reader repo is not currently part a CEDAR release.

Adding it will be a little tricky since Java 8 is required to build it.

Requires update to Java 10,which is waiting on update of Maven plugin it uses (#3).

Requires #3

Establish a production process for updating caDSR CDEs in CEDAR

Do not generate "required: [@id]" when transforming CDEs to CEDAR fields

Update caDSR reader to Java 11

Currently the JAXB2 Maven plugin only works with Java 8. The update of the plugin to support Java 10 is ongoing:

mojohaus/jaxb2-maven-plugin#43 (comment)

When it is completed we can upgrade reader to use Java 11.

Requires metadatacenter/cedar-project#772

Update caDSR script to add new fields

Support schema:identifier, skos:prefLabel, skos:altLabel fields in generated CDE fields.

Error updating CDEs

Investigate error with CDE import.

02:04:47.919 INFO  o.m.c.ingestor.util.CedarServices - 11400/70886 CDEs retrieved.
02:04:48.878 INFO  o.m.c.ingestor.util.CedarServices - 11500/70886 CDEs retrieved.
02:04:49.746 INFO  o.m.c.ingestor.util.CedarServices - 11600/70886 CDEs retrieved.
02:04:50.663 INFO  o.m.c.ingestor.util.CedarServices - 11700/70886 CDEs retrieved.
02:04:51.775 ERROR o.m.c.i.t.c.CadsrCategoriesAndCdesUpdaterTool - Error: Server returned HTTP response code: 500 for URL: https://resource.metadatacenter.org/folders/https%3A%2F%2Frepo.metadatacenter.org%2Ffolders%2F1ee5ef41-0605-4c18-9054-b01eb4290339/contents-extract?resource_types=field&field_names=schema:identifier,pav:version,sourceHash,categories&offset=11700&limit=100
java.io.IOException: Server returned HTTP response code: 500 for URL: https://resource.metadatacenter.org/folders/https%3A%2F%2Frepo.metadatacenter.org%2Ffolders%2F1ee5ef41-0605-4c18-9054-b01eb4290339/contents-extract?resource_types=field&field_names=schema:identifier,pav:version,sourceHash,categories&offset=11700&limit=100
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1974)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1969)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1968)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
	at org.metadatacenter.cadsr.ingestor.util.CedarServices.findCdeSummariesInFolder(CedarServices.java:123)
	at org.metadatacenter.cadsr.ingestor.util.CdeUtil.getExistingCedarCdeSummaries(CdeUtil.java:133)
	at org.metadatacenter.cadsr.ingestor.tools.cde.CadsrCategoriesAndCdesUpdaterTool.main(CadsrCategoriesAndCdesUpdaterTool.java:172)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:254)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: https://resource.metadatacenter.org/folders/https%3A%2F%2Frepo.metadatacenter.org%2Ffolders%2F1ee5ef41-0605-4c18-9054-b01eb4290339/contents-extract?resource_types=field&field_names=schema:identifier,pav:version,sourceHash,categories&offset=11700&limit=100
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1924)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
	at org.metadatacenter.cadsr.ingestor.util.CedarServices.findCdeSummariesInFolder(CedarServices.java:106)
	... 4 more

Revisit bypassing SSL certificate when uploading CDEs to CEDAR server

The current solution is to ignore the CEDAR certificate when running the Java upload code. This is not really secure and it needs a better solution. Below is the error message received when executing the code:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
unable to find valid certification path to requested target

The implemented solution is based on this post: https://stackoverflow.com/questions/13626965/how-to-ignore-pkix-path-building-failed-sun-security-provider-certpath-suncertp

Support all ~200 caDSR CDE datatypes

We currently support the 10 most common caDSR CDE datatypes (CHARACTER, NUMBER, java.lang.String, DATE, java.lang.Long, java.lang.Integer, java.lang.Double, java.util.Date, ALPHANUMERIC, ISO21090CDv1.0).

We do not support the remaining ~190. Some are pretty obscure (e.g., ISO21090TELURLv1.0, ISO21090.TEL.URL.v1.0, ISO21090BLv1.0) and are used in only a few CDEs. However, we cannot handle ~10% of caDSR CDEs due to not handling these types.

Supporting some of these datatypes may be non trivial if we assume that we also validate that supplied values conform to the specified datatype.

metadatacenter / cedar-cadsr-tools Goto Github PK

cedar-cadsr-tools's People

Contributors

Watchers

cedar-cadsr-tools's Issues

Recommend Projects

Recommend Topics

Recommend Org