A Standard Schema for Accessibility Testing Results about equalify HOT 13 CLOSED

As mentioned earlier, EARL seems to me to be a vocabulary, not a format, and its vocabulary seems to be silent on the important properties that accessibility testing tools report values of. It seems to abbreviate the entirety of test results into a single TestResult class, with no subordinate structure, so a description of a test result is nothing other than a string of text, such as “failed”.

So, the value of trying to use EARL as the basis, or as an inspiration, for a standard format is not yet clear to me.

from equalify.

@jrpool and @mgifford, tagging you here because I know you're both deep into automated testing. We're at the point with Equalify Version 2 developer where we're going to finalize our data architecture. Does this schema anticipate what you've seen with automated scan results?

from equalify.

I started a separate repo for this because I can see it being used for way more applications than just Equalify. Here it is: https://github.com/EqualifyEverything/equalify-formatter

from equalify.

EARL takes care of this: https://www.w3.org/TR/EARL10-Schema/

But we have to see if EARL is indeed viable for Equalify.

from equalify.

@jrpool- i'm kind of curious why you didn't just use EARL as a testaro standard?

from equalify.

The outline above seems mainly to reflect what most tools output, with some qualifications:

1. One tool, Nu Html Validator, has no rule IDs. Testaro fabricates rule IDs for it in the form of regular expressions.
2. Some tools provide multi-paragraph suggestions.
3. If one message cites an array of pages and also an array of elements, then it would seem the elements are on various pages, but how would the page of each element be identified? In any case, existing tools almost always test only 1 page at any time.
4. The tags property informalizes much of what I believe a practical standardized reporting format should formalize. That, it seems to me, interferes with subsequent processing of reports.
5. The element property combines an unspecified set of facts that, in my opinion, merit individual formalization.
6. Tools use various methods for identifying the locations of elements, including line number, x-y coordinates, xpath, and CSS selector, not only CSS selector.
7. The outer HTML of elements can be bulky. Some defects belong to the HTML element, so its outer HTML can be megabytes in size.

from equalify.

@jrpool- i'm kind of curious why you didn't just use EARL as a testaro standard?

EARL seems to me to be a vocabulary rather than a schema. When used in the production of reports, it seems to be unnecessarily bulky. Its vocabulary for test results seems to be primitive, relying on plain language for presentation of details. It seems to be silent on the difficult decisions in reporting, such as how to classify severity and certainty, and how to identify elements and express relationships to standards. Empirical inspection of the outputs of the 10 tools integrated by Testaro has revealed similarities to one another that organically lead to a compromise reporting format such as the standardized Testaro standard result. When I then ask how I would re-express that with the EARL vocabulary, its documentation and examples make it difficult (at least for me) to answer that question, but they suggest the result would not be parsimonious. I would be happy to have these understandings corrected.

from equalify.

Totally agree @jrpool - It's a headache to understand! It could probably be better understood as JSON. The good news is the RDF format tat EARL uses lends itself to JSON pretty easily. For example, here is an EARL output:

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:earl="http://www.w3.org/ns/earl#">

  <earl:Assertion>
    <earl:assertedBy>
      <earl:Software>
        <rdf:type rdf:resource="http://www.example.com/AccessibilityChecker"/> 
      </earl:Software>
    </earl:assertedBy>
    <earl:subject>
      <rdf:Description rdf:about="http://www.example.com/productpage.html" /> 
    </earl:subject>
    <earl:test>
      <rdf:Description rdf:resource="http://www.w3.org/TR/WCAG20/#images-must-have-alt"/> 
    </earl:test>
    <earl:result>
      <earl:outcome earl:pass />
    </earl:result>
  </earl:Assertion>

</rdf:RDF>

Here it is in JSON:

{
  "@context": {
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "earl": "http://www.w3.org/ns/earl#"
  },
  "@type": "earl:Assertion",
  "earl:assertedBy": {
    "@type": "earl:Software",
    "rdf:type": "http://www.example.com/AccessibilityChecker"
  },
  "earl:subject": {
    "@id": "http://www.example.com/productpage.html"
  },
  "earl:test": {
    "@id": "http://www.w3.org/TR/WCAG20/#images-must-have-alt"
  },
  "earl:result": {
    "earl:outcome": "earl:pass"
  }
}

Here's an even simpler JSON:

{
  "type": "Assertion",
  "assertedBy": {
    "type": "Software",
    "name": "AccessibilityChecker" 
  },
  "subject": "http://www.example.com/productpage.html",
  "test": "http://www.w3.org/TR/WCAG20/#images-must-have-alt",
  "result": "pass"
}

I'm wondering if we could work within that EARL format, using a simplified JSON schema?

from equalify.

Also might be worth mentioning that work on EARL stopped:

At this time there are not sufficient implementations to complete EARL 1.0 Schema as a W3C Recommendation.

https://www.w3.org/TR/EARL10-Schema/

from equalify.

@bbertucc is correct that writing routines to convert tool outputs into a standard format requires nontrivial effort. In the Testaro project, that work is incorporated into the standardize module located in the procs directory. it converts the outputs of the 10 tools into a single format, created as a compromise reflecting the main properties that most tools report. When another tool is to be integrated, the standardize module is revised to cover the output of that tool, too.

from equalify.

Neat what you've done in standardize @jrpool! If we had a standard schema, you would be able to quickly integrate scan results without needing to standardize most things. The conversation is progressing on A11y Slack with the folks behind axe-core and IBM's scanner. This also has a lot of implications for what the W3C team is doing with ACT standards: https://www.w3.org/WAI/standards-guidelines/act/report/testcases/

from equalify.

I'm moving forward with this issue here: https://github.com/EqualifyEverything/equalify-formatter

Once we have a standard format, I'm going to integrate it into Equalify.

from equalify.

Equalify Schema is now live, so we can close this issue.

from equalify.