opengeospatial / ets-cat30 Goto Github PK

If online TEAM Engine [1] is used with a reference implementation [2], eight tests fail with following message:

org.opengis.cite.validation.SchematronValidator.validate(Ljavax/xml/transform/Source;)Ljavax/xml/transform/dom/DOMResult;

Following Tests are failing:

• get Capabilities By Section
• get Capabilities From Base U R L
• get Capabilities From Base U R L As XML
• get Full Capabilities Accept Version3
• valid Open Search Description
• bounding Box Query
• get Resource By Id
• single Keyword Search

This error does not occur if TEAM Engine is setup locally via Docker [3].

[1] http://cite.opengeospatial.org/teamengine/
[2] http://demo.pycsw.org/cite/csw?service=CSW&version=3.0.0&request=GetCapabilities
[3] https://github.com/opengeospatial/teamengine-docker

This includes:

• ets-common
• TEAM Engine dependencies
• geomatics-geotk
• testng
• jcommander

According to the OpenSearch "Parameter" extension, it is possible to provide additional information for any Url element in the following manner:

<Url xmlns:parameters="http://a9.com/-/spec/opensearch/extensions/parameters/1.0/"
 type="text/html"
 template="http://example.com/search"
 parameters:method="POST"
 parameters:enctype="application/x-www-form-urlencoded">
   <parameters:Parameter name="q" value="{searchTerms}"/>
   <parameters:Parameter name="count" value="{itemsPerPage}" minimum="0"/>
   <parameters:Parameter name="start" value="{startIndex}" minimum="0"/>
</Url>

Such structure causes schema validation to fail.

The documentation inside the site folder should be update with the correct version of the specs.
The site folder content should also be converted to markdown, similar to other tests. See for example: /Users/lbermudez/Documents/Dropbox/github/ets-csw202. Correct name of files are;

• relnotes.md
• index.md

These are the correct specifications, to be referred to in the index.md and readme.md (at the root folder)

Image reference from the index file should also be update with correct versions

• OGC Catalogue Services 3.0 Specification - HTTP Protocol Binding
  (OGC 12-176r7)
• Catalog Services Specification 3.0 - General Model
  (OGC 12-168r6)
• OGC Catalogue Services 3.0 Specification - HTTP Protocol Binding - Abstract Test Suite
  (OGC 14-014r3)
• OGC OpenSearch Geo and Time Extensions
  (OGC 10-032r8)
• OpenSearch 1.1 (Draft 5)
• The Atom Syndication Format (RFC 4287)

OpenSearchGeoTests.getResourceById() substitutes {geo:uid} parameter in the URL template with the actual uid. If that uid has a form of UUID starting and ending with curly braces, then an exception is thrown from within OpenSearchTemplateUtils.buildRequestURI() at the line:

return URI.create(template);

with the message logged into the report like this:

I would suggest to URLEncode an identifier before passing to the template.

There are several tests which belong to some sort of general family of tests which I would call: "term search". It could be a single term search or multiple term search and it may apply to either CSW GetRecords request with "q" parameter or OpenSearch request using "searchTerm" parameter. Basically, all such tests are created in such a way that search term (or terms) isused to search the catalog, then a term (or terms) is expected to be present in the body of each returned record. There are several reasons why, in my opinion, this expectation is wrong:

1. Response format
   Let's imagine a metadata document which happen to have a particular term present in it's body, although it is buried deep down in the structure tree - enough to be indexed by some full text search engine like Apache Lucene. Any search request will yield that record, however the final Atom response won't have that particular element in it's body.
2. Stemmers
   Some catalogs might have linguistic stemmers deployed. In any such case, search for "rivers" (plural) will also yield records with "river" (singular). Test will fail because it won't be able to find "rivers".
3. Tags
   Some indexes might have added tags or some sort of information coming from somewhere else than metadata. Some catalogs may allow users to categorize stored metadata, for example one can declare a record to belong to "live data" category. Then search for "live data" will bring all such records even they don't have "live data" term explicitly embedded within metadata document.
4. Multi term
   It is expected that in case of multi term search, all terms from the input request will be present in the records. Not sure where this requirement comes from, but looking how the Apache Lucene works or Google Search works (both without additional search modifiers), this requirement seem to be to far fetched.In particular, I didn't find any requirement like that on OpenSearch specification. Maybe I am wrong, but a general (de facto) understanding is that in case of a multiple terms searches all records matching at least one term will be returned.

Test run argument information is missing from the rest endpoint page

Given c4348ab, I'm now getting a failure when testing against pycsw:

<![CDATA[XPath expression cannot be evaluated: not(exists(csw:SummaryRecord[dc:type or dc:subject or dc:format or ows:BoundingBox])). {1}]]>

Request: http://demo.pycsw.org/cite/csw?elementName=tns:title&request=GetRecords&service=CSW&typeNames=Record&namespace=xmlns%28tns%3Dhttp://purl.org/dc/elements/1.1/%29&version=3.0.0

What is the rationale of this assertion?

cc @rjmartell

When running:

 java -jar target/ets-cat30-0.8-SNAPSHOT-aio.jar mycsw-props.xml

The /tmp directory gets populated with files like entity-*.tmp and records-*.xml.

Not sure if this is happening in this test suite or teamengine overall. Or whether these are reused within the process or they are some leftover debugging. If they are intentional, it would be useful to allow for a setting/argument to clean up these files (or not) before the process completes.

Running mvn install when using OpenJDK11:

java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

mvn -version
Apache Maven 3.5.2
Maven home: /usr/share/maven
Java version: 11.0.1, vendor: Oracle Corporation
Java home: /java/openjdk-11.0.1
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.15.0-39-generic", arch: "amd64", family: "unix"

results in errors like

java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException

This is caused by the fact that on Java11, certain classes have been removed from the JDK. The fix for this is trivial. Just add the following dependencies to the pom.xml file.

    <dependency>
      <groupId>javax.xml.bind</groupId>
      <artifactId>jaxb-api</artifactId>
      <version>2.3.0</version>
    </dependency>
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-core</artifactId>
      <version>2.3.0</version>
    </dependency>
    <dependency>
      <groupId>com.sun.xml.bind</groupId>
      <artifactId>jaxb-impl</artifactId>
      <version>2.3.0</version>
    </dependency>
    <dependency>
      <groupId>javax.activation</groupId>
      <artifactId>activation</artifactId>
      <version>1.1.1</version>
    </dependency>

That is however not sufficient to be able to run the tests. Doing mvn install with that fix results in

java.lang.IllegalArgumentException: org.opengis.util.NameFactory is not an ImageIO SPI class
	at java.desktop/javax.imageio.spi.ServiceRegistry.checkClassAllowed(ServiceRegistry.java:722)
	at java.desktop/javax.imageio.spi.ServiceRegistry.<init>(ServiceRegistry.java:117)
	at org.geotoolkit.factory.FactoryRegistry.<init>(FactoryRegistry.java:165)
	at org.geotoolkit.factory.FactoryRegistry.<init>(FactoryRegistry.java:156)

This is caused by https://bugs.openjdk.java.net/browse/JDK-8068749. The geotools bugtracker indicates this is fixed: https://osgeo-org.atlassian.net/browse/GEOT-5289. However, the libraries on which this repo depends look like a very old version of geotools of which the latest version was published on Maven in 2015.

Running output via command line via java -jar ets-cat30-${version}-aio.jar my-csw.xml always writes out test outputs to a directory under $HOME/testng/. Can this be modified via command line argument to write to a user specified directory?

Instead, behaviour as described in opengeospatial/teamengine#251 can be observed.

Can be reproduced on http://cite.opengeospatial.org/te2/.

Probably related to opengeospatial/ets-gpkg12#17.

the maxRecords parameter has numeric values or 'unlimited'. what's the purpose of 'unlimited'. it raises some concerns when considering catalogs that have 100,000s or millions of items. denial of service attack waiting to happen.

for data listings as required by Data.gov, a common practice is to 'cache' this list on a daily basis server-side, that Data.gov can fetch. These lists are easily 100s of MB sized files without even containing the full metadata.

we suggest the test (the spec even) should not include this 'unlimited' option.

the scope and semantics of csw:AnyText syntax in CSW 3.0 needs to be defined:

• phrase vs keyword
• grouping of phrases
• boolean logic
• ...

When building the test suite the VerifyTestNGController test suite is expecting two errors yet only encounters one, causing an error. I believe this is because the default test props file does not reference a geopackage. As it is this does mean it's impossible to cleanly build from source without modification.

More to the point, though, when testing real geopackages I see that there are indeed two errors, every time. If that is the expected case, then why are these tests written to fail? The fact that they fail by design mean that it is difficult to incorporate these tests into build workflow. Do these test need to be ignored?

The fact that the test package itself passes only when there are two test failures would normally indicate that the tests should be rewritten to be positive tests which pass when the expected behaviour. is encountered. There are lots of ways to fail a test, including ways that are not expected by the test suite.

If there is a good reason to keep the tests around even though they fail, that would also be useful information. Thanks for producing the test suite at all, though!

Running mvn install on a new machine results in

[ERROR] Failed to execute goal on project ets-cat30: Could not resolve dependencies for project org.opengis.cite:ets-cat30:jar:1.3-SNAPSHOT: The following artifacts could not be resolved: org.geotoolkit.pending:geotk-xml-gml:jar:3.21, org.geotoolkit.pending:geotk-jtswrapper:jar:3.21, org.geotoolkit:geotk-epsg:jar:3.21: Failure to find org.geotoolkit.pending:geotk-xml-gml:jar:3.21 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Looking at the rest of the log, it shows the following 3 warnings:

[WARNING] The POM for org.geotoolkit.pending:geotk-xml-gml:jar:3.21 is missing, no dependency information available
[WARNING] The POM for org.geotoolkit.pending:geotk-jtswrapper:jar:3.21 is missing, no dependency information available
[WARNING] The POM for org.geotoolkit:geotk-epsg:jar:3.21 is missing, no dependency information available

Only way I found to fix this was copy all those files from the maven cache of another machine, and adding them on the new machine.

When the development of the test started the following specifications were used:

• OGC 12-176r5 OGC Catalogue Services 3.0 Specification – HTTP Protocol Binding
• OGC 12-168r5 Catalog Services Specification 3.0 – General Model
• OGC 14-014r3 OGC Catalogue Services 3.0 Specification – HTTP Protocol Binding – Abstract Test Suite
• OGC 10-032r8 OGC OpenSearch Geo and Time Extensions, Version 1.0

The above references are the ones we used in the description of the test (welcome page).

I see we have now:

• 12-176r7
• 12-168r6

Is the latest revision of the test aligned with the latest revision of the standard?

And, we need to confirm that the test is implementing the following "implementable" standard:
OGC Catalogue Services 3.0 Specification - HTTP Protocol Binding

In BasicGetRecordsTests.java, in function presentTitleProperty() for resolve Xpath expression

ETSAssert.assertXPath(
                "not(csw:SummaryRecord[dc:type or subject or dc:format or ows:BoundingBox])",
                results, null);

I need to pass a Map with namespaceBinding

Map<String, String> namespaceBindings = new HashMap<>();
namespaceBindings.put(Namespaces.DCMES,"dc");
namespaceBindings.put(Namespaces.CSW,"csw");
namespaceBindings.put(Namespaces.OWS,"ows");

ETSAssert.assertXPath(
                "not(csw:SummaryRecord[dc:type or subject or dc:format or ows:BoundingBox])",
                results, namespaceBindings);

I wonder if the problem come from my xml response or if we can enhance the test with this hashmap ?

Thanks

Please see: How to create Docker Images of test suites

Reference from: opengeospatial/cite#52

These changes should be done after all PR's are merged. https://github.com/opengeospatial/geomatics-geotk/pulls

Both BasicSearchTests.singleTermTextSearch() and OpenSearchCoreTests.singleKeywordSearch() are failing when the search term is a string with multi-byte characters, like: 소셜 미디어(사용 중지됨).

My recommendation would be to use URIUtils.getPercentEncodedString() to encode search term just like in each multipleKeywordSearch(), so:

BasicSearchTests.java:361
qryParams.put(CAT3.Q, URIUtils.getPercentEncodedString(keyword));

OpenSearchCoreTests.java:175
values.put(SEARCH_TERMS_PARAM, URIUtils.getPercentEncodedString(searchTerm));

The test is: "count(csw:SummaryRecord[dc:subject]) = ./@numberOfRecordsReturned".
However, considering that it might be more than one dc:subject in the record, shouldn't it rather be:
"count(csw:SummaryRecord[dc:subject]) >= ./@numberOfRecordsReturned" ?

Please see: https://github.com/opengeospatial/cite/wiki/How-to-run-tests-for-test-suites-with-SoapUI#integration-tests

Applies to integration and system tests.

SoapUI project must be updated as version was removed from URI to test suites of REST API (also see opengeospatial/teamengine#261).
Please also update Maven configuration and set version of SoapUI Maven Plugin to 5.5.0.

See updated documentation: https://github.com/opengeospatial/cite/wiki/How-to-run-tests-for-test-suites-with-SoapUI

Example, how this was solved for TEAM Engine tests: opengeospatial/teamengine#400

The test needs to be updated to better provide information about the grouping of the conformance class and determination about which ones are the basic conformance classes needed for certification.

Changes that are required are described here:
https://github.com/opengeospatial/cite/wiki/Encoding-Conformance-Classes

Is open search conformance class ready? If not please document what needs to be developed. Also document the other ones that you think are required to be developed.

cc @Schpidi

As per http://docs.opengeospatial.org/is/12-176r7/12-176r7.html, Requirement 004 states:

If a CSW request includes a parameter (e.g. outputFormat) that performs a function that is similar to an HTTP message header (e.g. Accept) and both are included in a request then their values must agree otherwise an InvalidParameterValue exception shall be raised as specified in Subclause 6.7.

In pycsw, we do a strict comparison of the HTTP Accept header and outputFormat, so the below works:

curl "http://demo.pycsw.org/cite/csw?mode=opensearch&service=CSW&version=3.0.0&request=GetRecords&elementsetname=full&typenames=csw:Record&resulttype=results&q=&bbox=-23.17822265625,6.62353515625,49.90283203125,71.44287109375&time=2006-08-15T00:00:00.000Z/2006-08-22T23:59:59.000Z&outputformat=application/atom+xml&&startposition=1&maxrecords=500" --header "Accept: application/atom+xml"

However, if the user specifies Accept: */*, like:

curl "http://demo.pycsw.org/cite/csw?mode=opensearch&service=CSW&version=3.0.0&request=GetRecords&elementsetname=full&typenames=csw:Record&resulttype=results&q=&bbox=-23.17822265625,6.62353515625,49.90283203125,71.44287109375&time=2006-08-15T00:00:00.000Z/2006-08-22T23:59:59.000Z&outputformat=application/atom+xml&&startposition=1&maxrecords=500" --header "Accept: */*"

Then pycsw will complain because it not an exact match.

Should Accept: */* and outputformat=application/atom+xml, for example, constitute passing Requirement 004?

When running mvn org.apache.maven.plugins:maven-dependency-plugin:analyze, I get following warnings:

[WARNING] Used undeclared dependencies found:
[WARNING]    org.testng:testng:jar:6.8.8:compile
[WARNING]    org.geotoolkit:geotk-referencing:jar:3.21:compile
[WARNING]    org.opengis:geoapi-pending:jar:3.1-M04:compile
[WARNING]    org.opengis.cite.saxon:saxon9:jar:9.0.0.8:compile
[WARNING]    com.beust:jcommander:jar:1.27:compile
[WARNING]    org.geotoolkit:geotk-utility:jar:3.21:compile
[WARNING]    xml-apis:xml-apis:jar:1.4.01:compile
[WARNING]    xerces:xercesImpl:jar:2.11.0:compile
[WARNING]    javax.ws.rs:jsr311-api:jar:1.1.1:compile
[WARNING] Unused declared dependencies found:
[WARNING]    com.google.inject:guice:jar:3.0:test

All used undeclared compile dependencies shall be declared in Maven pom file.
All unused declared dependencies shall be removed from Maven pom file.

See opengeospatial/ets-wms13#56 for instructions.

Please use following service implementation: http://demo.pycsw.org/cite/csw?service=CSW&version=3.0.0&request=GetCapabilities

Please see opengeospatial/cite#56

Error printed in Main test:

Error in call to extension function {public java.lang.Object com.occamlab.te.TECore.callFunction(net.sf.saxon.expr.XPathContext,java.lang.String,java.lang.String,net.sf.saxon.om.NodeInfo) throws java.lang.Exception}: Exception in extension function net.sf.saxon.s9api.SaxonApiException: Error in call to extension function {public javax.xml.transform.Source org.opengis.cite.cat30.TestNGController.doTestRun(org.w3c.dom.Document) throws java.lang.Exception}: Exception in extension function org.testng.TestNGException: 
Cannot instantiate class org.opengis.cite.cat30.opensearch.OpenSearchPreconditions
Result: Failed

Several tests from OpenSearchCoreTests iterate through the list of Urls from the OpenSearchDescription, pick all which have a type ending with "xml" and use it to obtain list of record elements. This works fine for CSW response or Atom response with record elements marked accordingly as 'Record' or 'entry'. However, any Url of type "application/rss+xml" will fail because RSS returns items not records nor entries.

Example of such is executeExampleQueries(), which logs message like this:

"Expected one or more {http://www.opengis.net/cat/csw/3.0}Record elements in search results. expected [true] but found [false]"

It comes with the request like this:

Method: GET
Target URI: http://host:port/context?searchText=land&start=1&max=&bbox=&f=georss
Headers: {Accept=[application/rss+xml]}