w3c / web-annotation-tests Goto Github PK

Even after #5 fixes the JSON syntax, the tests incorrectly fail the model's example anno1.json, despite being valid.

(Using python's jsonschema implementation, rather than ajv)

But common/id.json only has a warning if the test fails.

If the thinking is to use common/id.json to test for ids on any resource, then the bug is actually in the title, description and errorMessage fields.

That said ... it is valid to send an annotation without an ID in a POST via the protocol to create a new annotation. Then there isn't any existing URI for it, as the server must assign one anyway. Worth flagging such cases in the test json?

Looking at the existing test manifests (e.g., https://github.com/w3c/web-annotation-tests/blob/c9d3b8b084c509ef9fa7e8ed66e8f9a931348908/framework/annotations/model.test), the form is somewhat ad-hoc. In w3c/web-annotation#290 (comment) I was asked to help create the reports, which as mentioned there, uses the earl-report gem. This expects to find one or more test manifests, which are generally like those described for the RDF tests, for example, the Turtle test manifest. That is, there is an entry of type mf:Manifest, with an entries property referencing a list of entries; the built-in SPARQL query used to access the manifest is the following:

PREFIX mf: <http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

SELECT ?uri ?testAction ?manUri
WHERE {
  ?uri mf:action ?testAction .
  OPTIONAL {
    ?manUri a mf:Manifest; mf:entries ?lh .
    ?lh rdf:first ?uri .
  }
}

But, this can query be overridden if there is some reason to use a different structure. But, to get name and description information for each test, each test entry is expected to have rdfs:label and rdfs:comment entries, in addition to the action. Note, that the names in a JSON-LD version of the manifest can be arbitrary, as long as the context maps them to rdfs:label and rdfs:comment. The mf:action is used to retrieve the test source, and mf:result used to get the expected results, but these aren't strictly necessary, and the query could access some other entry from each test to enumerate all tests.

The earl-report software reads in the manifest and turns it into framed JSON-LD, which is then available to the template.haml which iterates over that to create output for each test. The template will read each mf:action and mf:result file to include in the output, but this isn't necessary. However, it would probably be good to include some information for each test in addition to label and comment, for example we could show each assertion.

The main thing I'm calling for here is to get the test manifests in some normalized form consistent with our reporting needs include a manifest entry with an ordered list for each test including a label and comment using some properties such that this can be transformed to RDF and queried.

ERROR: Permission to w3c/web-annotation-tests.git denied to BigBlueHat.
fatal: Could not read from remote repository.

In the meantime, I'll send a PR. 😉

This may be implementation dependent as to whether an IRI will match the format=uri rule or not.
Example from @tkanai: w3c/web-annotation#183 (comment)

If it is implementation dependent, we could either:

1. be less restrictive and just check for strings
2. devise a slower regular expression to test for IRIs
3. implement custom format checkers for format=iri

@awwright mentioned over on
json-schema/json-schema#167 (comment)
that they'd be interested in helping write some tests! 😍

This is the repo where we keep them.

We're currently targeting Web Platform Tests integration in this repo:
https://github.com/Spec-Ops/web-platform-tests/

Contributions here are governed by this doc (just FYI):
https://github.com/w3c/web-annotation-tests/blob/master/contributing.md

Much thanks for the offer to help!
🎩