sentineldatahub / dhus-distribution Goto Github PK

Good day!

During user sign-in, email with verification url was received. But it starts with http:\localhost:8081...

What changes should be made and in which config files?

P.S. In DHuS Administration Manual I didn't find this information.

Good day!

In AJS GUI / Upload Management tried to upload product S2A_MSIL1C_20160116T085312_N0201_R107_T36UUA_20160116T085621.zip

Got message: "Product Upload Succeeded "The file was created successfully.""

But product didn't appear at "Collections Management".

(DHuS deployment mode: single instance)

In dhus.log there is error :
[1.0.1-15-osf][2018-12-19 11:11:37,374][INFO ] Store product S2A_MSIL1C_20160116T085312_N0201_R107_T36UUA_20160116T085621 into datastores. (Ingester.java:104 - taskExecutor-3)
[1.0.1-15-osf][2018-12-19 11:11:37,375][WARN ] Unrecoverable error happen during ingestion of null (removing from database) (Ingester.java:160 - taskExecutor-3)
java.lang.NullPointerException
at fr.gael.dhus.datastore.Ingester.ingest(Ingester.java:116)
….

In hsqldb.log new record created and deleted at one time:
/C1115/SET SCHEMA PUBLIC
INSERT INTO PRODUCTS VALUES(10,'2018-12-19 11:11:36.934000','2018-12-19 11:11:36.931000',NULL,NULL,NULL,FALSE,NULL,'file:/home/dhus/local_dhus/tempos/S2A_MSIL1C_20160116T085312_N0201_R107_T36UUA_20160116T085621.zip',FALSE,NULL,'83b41c2a-30e1-409f-a5bc-c3f8eb93aadd',NULL,NULL,-1,'application/octet-stream',NULL,NULL,NULL,NULL,NULL,NULL,'7ca663b0-18fd-45f4-b324-c3dec64a2db4')
INSERT INTO PRODUCT_USER_AUTH VALUES(10,'7ca663b0-18fd-45f4-b324-c3dec64a2db4')
COMMIT
/C1116/SET SCHEMA PUBLIC
DELETE FROM PRODUCT_USER_AUTH WHERE PRODUCTS_ID=10 AND USERS_UUID='7ca663b0-18fd-45f4-b324-c3dec64a2db4'
DELETE FROM PRODUCTS WHERE ID=10
COMMIT

What may cause such error?

P.s. sorry for my English

dhus.log
hsqldb.log
dhus(xml).txt

I need to know if there are products generated by Copernicus about Sentinel 5P for biweekly or monthly periods and where to find them

Im using ceph with swift api for my object storage. In configuring the datastore in dhus.xml, there are 3 apis to set for the provider, i set my provider to openstack-swift. But i noticed that in the logs there is a reference to openstack-keystone. Is the provider for the datastore openstack-keystone? (even though i mention in the conf settings that it should use openstack-swift)
[ERROR] Command not considered safe to retry because request method is POST: [method=org.jclouds.openstack.keystone.v2_0.AuthenticationApi.public abstract org.jclouds.openstack.keystone.v2_0.domain.Access org.jclouds.openstack.keystone.v2_0.AuthenticationApi.authenticateWithTenantNameAndCredentials(java.lang.String,org.jclouds.openstack.keystone.v2_0.domain.PasswordCredentials

We are trying to setup the whole DHuS (2.0.0) infrastructure in a containerized(docker) environment, but we are running into a problem with Solr. We are following the installation steps in the DHuS external DB Installation and Configuration Manual.

Zookeeper is configured with a chRoot "/solr", Solr configured in cloud mode with a collection named "mycollection". Zookeeper and Solr are running 1-1 instance.
Settings in dhus.xml: <search:solrCloud zkHosts="zookeeper:2181/solr/mycollection"/>
The Solr and Zookeeper containers are using the official images with the specified versions mentioned in the docs.

With this configuration when the application tries to connect to the Solr instance, it gives the following error:
Cannot connect to cluster at zookeeper:2181/solr cluster not found/not ready

Changing the Solr-core version in the dhus-core project from 5.5.3 to 7.2.1 resolves the problem, but it would be better if we wouldn't need to change anyting.

• Do you have any experience setting up the whole infrastructure (Solr cloud mode, zookeeper, postgres) in docker environment?

• Did you tried to use newer versions of Solr and Zookeeper?

• Does the dhus application have an event log of user actions like searches, product downloads, etc. with user information in them? We would need it for a usage report by domain, usage, country. If not, processing connections would be a good way to create a report like this?

The docs mentions that the Connections are accessible via odata/v1/Users('name')/Connections, and its referencing the RD6 document (DHuS OData and Open Search Interface Control Document, COPE-SERCO-IF-17-0018, v2.0). Could you give us a hint where we can find this doc?

Attempting to run DHuS 2.7.8-osf with a recent JRE 8 (e.g. build 1.8.0_302-8u302-b08-0ubuntu2-b08 as packaged for Ubuntu 21.10 Impish Indri) fails immediately due to an exception during the version check:

[2.7.8-osf][2021-12-07 19:25:16,508][INFO ] Data Hub Service initialization... (DHuS.java:260 - main) [2.7.8-osf][2021-12-07 19:25:16,520][INFO ] dhus-distribution 2.7.8-osf (DHuS.java:261 - main) [2.7.8-osf][2021-12-07 19:25:16,520][INFO ] dhus-core 1.11.1-osf (DHuS.java:262 - main) [2.7.8-osf][2021-12-07 19:25:16,521][INFO ] DRB 2.5.10 (DHuS.java:263 - main) [2.7.8-osf][2021-12-07 19:25:16,854][ERROR] Uncaught exception (UncaughtExceptionHandler.java:37 - main) java.lang.ExceptionInInitializerError at com.ibm.icu.impl.UPropertyAliases.<clinit>(UPropertyAliases.java:87) at com.ibm.icu.lang.UCharacter.<clinit>(UCharacter.java:5680) at com.hp.hpl.jena.rdf.arp.impl.CharacterModel.isComposingChar(CharacterModel.java:134) at com.hp.hpl.jena.rdf.arp.impl.ParserSupport.checkComposingChar(ParserSupport.java:140) at com.hp.hpl.jena.rdf.arp.states.AbsWantLiteralValueOrDescription.characters(AbsWantLiteralValueOrDescription.java:45) at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.characters(XMLHandler.java:156) at org.apache.xerces.parsers.AbstractSAXParser.characters(Unknown Source) at org.apache.xerces.impl.XMLNamespaceBinder.characters(Unknown Source) at org.apache.xerces.impl.dtd.XMLDTDValidator.characters(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source) at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at com.hp.hpl.jena.rdf.arp.impl.RDFXMLParser.parse(RDFXMLParser.java:106) at com.hp.hpl.jena.rdf.arp.JenaReader.read(JenaReader.java:215) at com.hp.hpl.jena.rdf.arp.JenaReader.read(JenaReader.java:202) at com.hp.hpl.jena.rdf.arp.JenaReader.read(JenaReader.java:272) at com.hp.hpl.jena.rdf.model.impl.ModelCom.read(ModelCom.java:230) at com.hp.hpl.jena.ontology.OntDocumentManager.findMetadata(OntDocumentManager.java:1010) at com.hp.hpl.jena.ontology.OntDocumentManager.initialiseMetadata(OntDocumentManager.java:968) at com.hp.hpl.jena.ontology.OntDocumentManager.<init>(OntDocumentManager.java:205) at com.hp.hpl.jena.ontology.OntDocumentManager.<init>(OntDocumentManager.java:187) at com.hp.hpl.jena.ontology.OntDocumentManager.<init>(OntDocumentManager.java:171) at com.hp.hpl.jena.ontology.OntDocumentManager.getInstance(OntDocumentManager.java:249) at com.hp.hpl.jena.ontology.OntModelSpec.getDocumentManager(OntModelSpec.java:389) at com.hp.hpl.jena.ontology.impl.OntModelImpl.getDocumentManager(OntModelImpl.java:181) at com.hp.hpl.jena.ontology.impl.OntModelImpl.<init>(OntModelImpl.java:152) at com.hp.hpl.jena.ontology.impl.OntModelImpl.<init>(OntModelImpl.java:124) at com.hp.hpl.jena.rdf.model.ModelFactory.createOntologyModel(ModelFactory.java:402) at com.hp.hpl.jena.rdf.model.ModelFactory.createOntologyModel(ModelFactory.java:361) at com.hp.hpl.jena.rdf.model.ModelFactory.createOntologyModel(ModelFactory.java:344) at fr.gael.cortex.CortexModel.loadDefaultOntModel(CortexModel.java:216) at fr.gael.cortex.CortexModel.<init>(CortexModel.java:351) at fr.gael.cortex.CortexModel.getDefaultModel(CortexModel.java:195) at fr.gael.drbx.cortex.DrbCortexModel.<init>(DrbCortexModel.java:439) at fr.gael.drbx.cortex.DrbCortexModel.getDefaultModel(DrbCortexModel.java:179) at fr.gael.dhus.DHuS.start(DHuS.java:154) at fr.gael.dhus.DHuS.main(DHuS.java:292) Caused by: java.lang.IllegalArgumentException: Invalid version number: Version number may be negative or greater than 255 at com.ibm.icu.util.VersionInfo.getInstance(VersionInfo.java:188) at com.ibm.icu.impl.ICUDebug.getInstanceLenient(ICUDebug.java:65) at com.ibm.icu.impl.ICUDebug.<clinit>(ICUDebug.java:69) ... 41 more [2.7.8-osf][2021-12-07 19:25:16,864][ERROR] Uncaught exception (UncaughtExceptionHandler.java:37 - Thread-1) java.lang.NullPointerException at fr.gael.dhus.spring.context.ApplicationContextProvider.getBean(ApplicationContextProvider.java:54) at fr.gael.dhus.DHuS$1.run(DHuS.java:130) at java.lang.Thread.run(Thread.java:748)

Replacing icu4j-3.4.4.jar with icu4j-69_1.jar from the official website allows the init process to proceed past this point (it then fails later apparently due to an unrelated bug).

As far as I am aware, an opensearch endpoint must serve the description.xml files with application/opensearchdescription+xml MIME type.

How/Where can I get it?

Version 2.7.8-osf uses log4j 2.12, which is affected by the recent Log4Shell 0-day. Besides an update to 2.15, which by default closes the vulnerability, the following mitigations are offered:

Mitigation: In releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases >=2.7 and <=2.14.1, all PatternLayout patterns can be modified to specify the message converter as %m{nolookups} instead of just %m.

I am designing a research for master thesis and I need to download the Sentinel 5P time series of a calendar year. Can someone help me please? It is very difficult to download the images of a year individually