a11ywatch / kayle Goto Github PK
View Code? Open in Web Editor NEWNext-gen automated web accessibility testing
Home Page: https://a11ywatch.github.io/kayle/
License: MIT License
Next-gen automated web accessibility testing
Home Page: https://a11ywatch.github.io/kayle/
License: MIT License
Using a setup similar as described in a11ywatch/a11ywatch#65, I encounter a TypeError
when scanning https://www.bolagsverket.se/ using either axe
or htmlcs
as runner.
The scan is run using the latest Docker container 'a11ywatch'. The container is launched via Podman as follows:
podman run --rm --interactive --add-host chrome:127.0.0.2 -p 3280:3280/tcp -p 50050:50050/tcp -p 50051:50051/tcp -p 50052:50052/tcp -p 50053:50053/tcp -p 50054:50054/tcp -p 9222:9222/tcp docker.io/a11ywatch/a11ywatch
Then, a scan is triggered by the following curl invocation:
curl --max-time 30 --location --request POST http://localhost:3280/api/scan -H 'Accept-Charset: utf-8' --header 'Content-Type: application/json' --data-raw '{ "url": "https://www.bolagsverket.se/", "runners": ["axe"] }'
In the terminal where the container was started (not where curl is running), the following output is generated after the start of curl:
page.evaluate: TypeError: Cannot read properties of undefined (reading 'run')
at eval (eval at evaluate (:208:30), <anonymous>:1:90)
at UtilityScript.evaluate (<anonymous>:210:17)
at UtilityScript.<anonymous> (<anonymous>:1:44)
at o (/usr/src/app/node_modules/kayle/build/kayle.js:1:354)
at s (/usr/src/app/node_modules/kayle/build/kayle.js:1:715)
at async g (/usr/src/app/node_modules/kayle/build/kayle.js:1:1685)
On the curl side, valid JSON output is returned, but no accessibility issue is reported:
{"data":{"domain":"www.bolagsverket.se","url":"https://www.bolagsverket.se/","pageLoadTime":{"duration":0,"durationFormated":"Cached/Extremely Fast"},"lastScanDate":"2023-11-17T09:15:10.162Z","issues":[],"issuesInfo":{"possibleIssuesFixedByCdn":0,"totalIssues":0,"issuesFixedByCdn":0,"errorCount":0,"warningCount":0,"noticeCount":0,"accessScore":0,"issueMeta":{"skipContentIncluded":true}},"online":true},"success":true,"code":200,"message":""}
The test in autoKayle
for the domain a11ywatch.com and some others seem to change randomly on puppeteer for the warning color-contrast. When running autoKayle
across the entire domain colorContrastMatches
adds extra warnings. This needs further investigation.
The following codes have missing 'ja' translations.
WCAG2AA.Principle1.Guideline1_3.1_3_4.
WCAG2AA.Principle1.Guideline1_4.1_4_10.C32,C31,C33,C38,SCR34,G206
WCAG2AA.Principle1.Guideline1_4.1_4_11.G195,G207,G18,G145,G174,F78
WCAG2AA.Principle1.Guideline1_4.1_4_12.C36,C35
WCAG2AA.Principle1.Guideline1_4.1_4_13.F95
WCAG2AA.Principle2.Guideline2_1.2_1_4.
WCAG2AA.Principle2.Guideline2_5.2_5_1.
WCAG2AA.Principle2.Guideline2_5.2_5_2.
WCAG2AA.Principle2.Guideline2_5.2_5_3.F96
WCAG2AA.Principle2.Guideline2_5.2_5_4.
WCAG2AA.Principle4.Guideline4_1.4_1_3.
WCAG2AA.Principle1.Guideline1_3.1_3_5.H98
Some of the notices are not being tracked for the fast_htmlcs runner.
The notice
'If this image cannot be fully described in a short text alternative, ensure a long text alternative is also available, such as in the body text or through a link.'
Now that we have ace as a runner, we need to map the errors to the scoremap.
Ace is way too slow and the impact for a re-write does not benefit the library since the runner was more for users who are used to using it. We added it failry quickly to test/experiment but, it has done more harm in production than good since theres not a label ( this is really slow and not ideal for large processing, since slow is usually tied to expense ).
The ace runner looks very promising. One of the goals for this library is performance so following the patterns of our other lib runners would help.
When running a11ywatch with [email protected]
and runner 'axe' to analyze https://www.vilhelmina.se, among other the following issue gets reported:
{
"code": "meta-viewport-large",
"type": "error",
"typeCode": 1,
"message": "Users should be able to zoom and scale the text up to 500%",
"context": "<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0, maximum-scale=2.0, user-scalable=yes\">",
"selector": ">:nth-child(1)>:nth-child(3)",
"runner": "axe",
"recurrence": 0
}
The selector starts with >
which is AFAIK invalid. It may be the case that simply html
is missing at the beginning, given that the context is about a meta
tag, probably located inside head
, but it should not be the user's task to guess here and correct invalid output.
The next step for kayle is to create our own runner in Rust that ports to wasm.
We have a high overview of the stability we can get from runs to build.
There is a method called _audit_not_ready
in kayle_innate that has a start to getting a web accessibility parser ready.
At a high level the runs should perform between 20-40ms consistently for all audits regardless. The scaling has a high top level so the larger will only affect runs a little. Having the consistency of 20-40ms stable across audits could be a big win if we pick out the important test from htmlcs
, axe, and
ace` while keeping the performance to the T.
Our rewrite of html
is really fast where it might make sense to flip the aspects of focus to increase the throughput of that runner until we get it to 10-20ms on average. The codebase allows for easy transitions to o(1) lookups and oneshot maps.
This needs to be done with axe, htmlcs, and kayle.
I'm not totally sure what the intended output is here, but i'm pretty sure the last branch of this code always panics.
Lines 24 to 30 in 7a9d96c
If I follow b.len()
through here:
b.len() > 2
is false, then this must be true b.len() <= 2
b.len() == 2
is false, then this must be true b.len() < 2
b.len()
(given its a usize) is either 0 or 1, so the index used here will I suppose have underflowed(?) and so the b[b.len() - 2]
is guaranteed to panicSince 0.4.72
I am getting a panic in the radiant_blast function through wasm, its not a super useful error but given that it started breaking after this code was introduced I am assuming this is the problem. (The url I am running against in this instance is http://localhost:4173
if it helps)
Edit: I got much nicer error out when running on github actions for some reason, it was just saying RuntimeError: unreachable
on my mac:
panicked at 'index out of bounds: the len is 1 but the index is 4294967295', src/lib.rs:29:17
This error pretty much confirms the above
First, apologies if I report this issue at the wrong place!
I am running a11ywatch as a container as pulled from hub.docker.com and then issuing request for accessibility scans via carefully crafted HTTP POST requests to http://localhost:3280/api/scan
. That is working so far and I get back a JSON document with the scan's results.
However, for some but not all issues reported under 'data' -> 'issues', the 'selector' contains invalid (or incomplete) CSS selectors, which makes locating and further processing of reported issues difficult.
For example, take the following HTML page:
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta charset="UTF-8" />
<title>Hello</title>
</head>
<body>
<p>Hello</p>
</body>
</html>
The resulting JSON document for the curl invocation of curl --output output.json --location --request POST http://localhost:3280/api/scan --header 'Content-Type: application/json' --data-raw '{ "url": "http://www.example.com/", "runners": ["htmlcs","axe","custom"] }'
(assuming the HTML code above is served from 'www.example.com') is as follows:
{
"code" : 200,
"data" : {
"domain" : "www.example.com",
"issues" : [
{
"code" : "landmark-one-main",
"context" : "<html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"en\"><head>\n<meta charset=\"UTF-8\">\n<t...</html>",
"message" : "Document should have one main landmark",
"recurrence" : 0,
"runner" : "axe",
"selector" : ">:nth-child(1)>:nth-child(1)",
"type" : "error",
"typeCode" : 1
},
{
"code" : "page-has-heading-one",
"context" : "<html xmlns=\"http://www.w3.org/1999/xhtml\" lang=\"en\"><head>\n<meta charset=\"UTF-8\">\n<t...</html>",
"message" : "Page should contain a level-one heading",
"recurrence" : 0,
"runner" : "axe",
"selector" : ">:nth-child(1)>:nth-child(1)",
"type" : "error",
"typeCode" : 1
},
{
"code" : "region",
"context" : "<p>Hello</p>",
"message" : "All page content should be contained by landmarks",
"recurrence" : 0,
"runner" : "axe",
"selector" : "body>:nth-child(1)",
"type" : "error",
"typeCode" : 1
}
],
"issuesInfo" : {
"accessScore" : 100,
"errorCount" : 3,
"issueMeta" : {
"skipContentIncluded" : true
},
"issuesFixedByCdn" : 0,
"noticeCount" : 0,
"possibleIssuesFixedByCdn" : 0,
"totalIssues" : 3,
"warningCount" : 0
},
"lastScanDate" : "2023-10-03T11:33:17.753Z",
"online" : true,
"pageLoadTime" : {
"duration" : 5,
"durationFormated" : "Cached/Extremely Fast"
},
"url" : "https://www.example.com/"
},
"message" : "",
"success" : true
}
The 'issue' with code 'region' has a valid 'selector' (body>:nth-child(1)
), but neither the issues 'page-has-heading-one' nor 'landmark-one-main' have one as those two selectors start with >
. This symbol denotes 'child combinator' and the 'parent' of this relation is missing.
Expected output: If a concrete element in the DOM tree can be identified as the culprit, the 'selector' field should unambiguously refer to it. If no such unique element exists, then either 'html', 'body', or 'head' should be stated as selector (whatever fits best for a given issue).
Part of adding the new rules from 2.2.
The 2.5.8 needs to be implemented in fast_axecore and fast_htmlcs.
We have a base rule that iterates the nodes top level for htmlcs. The htmlcs portion should include the logic to check for the elements next to each other to see if the rule fails.
fast_axecore, we do not have as nice of a setup for all top rules. We may need to just add a new rule to go through all nodes.
Being cautious of performance is critical for this rule since it touches all nodes.
We need to bind the styles from css to our node so we can properly validate attributes and prepare for our layout rendering.
Htmlcs does not have capabilities of testing gradients. Right now it would be good to ignore this and move the issue as a warning for manual testing.
In order to test a linear gradient we need to take the colors at the point of axis and apply contrast testing against the text.
The text needs to be part of the group in order to test so capturing the text width and the x-y points of where the text.
Once we have the points to map to we can apply the color matching across the gradient. We would take the contrasts for each segment and average them in unless one area fails critically ( we would need to gauge how much of the area failed with the text in it). After we have the metrics of the two a score can be done to validate the gradient.
In order to get the clip position we need to render our layout with our custom tree. The rust lib taffy
can re-create the entire tree with minimal work as long as we bind the css correctly.
This is a more advanced test that is mainly suited for manual auditing. It is possible how ever to test for the appearance across the major browsers. The most basic way to test the element would be with an anchor that overrides the style using outline none inline.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.