chanzuckerberg / axe-storybook-testing Goto Github PK

I'm trying to onboard this package to my existing Storybook, and, out-of-the-box, I noticed that most of the tests were failing. After a certain number of tests, all subsequent tests fail w/ the error:

Error: page.evaluate: Evaluation failed: TypeError: Cannot read property 'add' of undefined
     at runAxe (eval at evaluate (:303:29), <anonymous>:4:26)
     at UtilityScript.evaluate (<anonymous>:305:22)
     at UtilityScript.<anonymous> (<anonymous>:1:44)

The errors start happening for every test after a certain number of Storybooks are tested:

(this goes on for about 500 failing tests in a row)

I wasn't sure why this was happening so I decided to re-run the tests, but only with one of the failing Stories present. The error went away:

(notice that these were tests listed as failing in the previous screenshot)

Conclusion: It appears that axe-storybook-testing only works properly for 50-100 Stories, then starts failing all subsequent stories with the error above.

Currently the test timeout can be set globally

axe-storybook --timeout 5000

We could add a timeout option to the params for each story

export const SomeStory = {
  parameters: {
    axe: {
      timeout: 5000,
    },
  },
};

Allow passing axe options as a parameter on stories.

For example, something like

export const SomeStory = {
  parameters: {
    axe: {
      options: someAxeConfig,
    },
  },
}

Axe has different impacts for different rules. Impact options include minor, moderate, serious, or critical.

I think it would be nice to be able to configure what impact level to use for determining if a result should fail

One way to consider integrating this would be to use an array option in yargs with something like --failing-impact serious critical which would then not fail the test if minor or moderate issue occurred

When running my CLI command (regardless of config), the cli always hangs with the following output:

$ npm run test:a11y -- --headless false --browser firefox

> [email protected] test:a11y /Users/karrui/Projects/FormSG/frontend
> axe-storybook "--headless" "false" "--browser" "firefox"

[firefox] accessibility
Error! The suite failed to run
An error was encountered before we started testing stories. Likely this means the browser failed to open.

Error: Storybook object not found on window. Open your storybook and check the console for errors.

0 passing (0ms)
1 failing
0 pending

Right now we use import syntax because of TypeScript, but everything gets compiled to require (CommonJS).

If we convert this library to natively use ES modules (import syntax), we'll be more future proof and able to update some of the node libraries.

To do this would probably require along the lines of

• Configure TypeScript to output ES modules
• Add "type": "module" to the package.json
• Update supported node versions from >= 12 to >= 12.20 (I think. Double check this)
• Fix whatever breaks

Doing this should be safe for consumers of the library, since we now only support Node >= 12.

I had to upgrade from 7.1.1 to 7.1.2 to support Debian 12 here pass-culture/pass-culture-app-native#5550

However, I started to have this error :

browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium-1084/chrome-linux/chrome
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just installed or updated. ║
║ Please run the following command to download new browsers:              ║
║                                                                         ║
║     yarn playwright install                                             ║
║                                                                         ║
║ <3 Playwright Team                                                      ║
╚═════════════════════════════════════════════════════════════════════════╝
    at TestBrowser.create (/home/runner/work/pass-culture-app-native/pass-culture-app-native/node_modules/@chanzuckerberg/axe-storybook-testing/build/browser/TestBrowser.js:40:69)

I tried to add a step before with yarn playwright install :

yarn run v1.22.19
$ /home/runner/work/pass-culture-app-native/pass-culture-app-native/node_modules/.bin/playwright install
Done in 0.42s.

I expect yarn axe-storybook --build-dir $BUILD_DIR to pass.

Environment :

• GitHub action Image: ubuntu-22.04
• node: v18.18.2
• npm: 9.8.1
• yarn: 1.22.19

Do you have any clue ?

I am getting the below error when running axe-storybook as part of CI. The same command works fine locally. Below is a screenshot showing the command we ran and the error:

Anyone seen something similar before? any advice would be appreciated.

I'm getting a color contrast failure on a test when running this package that I don't receive in the browser. The test environment is "seeing" different hex codes than what are set in code. To debug I tried changing the failing foreground color to css 'hotpink': #ff69b4.

Here are the values I get in the error message across all three test browsers:

None of them match the actual hex code. I have no idea what's happening here. Any thoughts?

I'm seeing many examples of color-contrast issues where the "background color" reported is incorrect.

In my case, I'm continually seeing links and buttons fail due to insufficient color contrast, and background color: #0073b1. For example:

html: <a class="tab-link" aria-current="true" href="#?trk=test" data-tracking-control-name="test" data-tracking-will-navigate="">
                   
                      First Tab
                  </a>
            summary: Fix any of the following:
                       Element has insufficient color contrast of 1.11 (foreground color: #057642, background color: #0073b1, font size: 12.0pt (16px), font weight: normal). Expected contrast ratio of 4.5:1

The actual background color in Storybook though is white. Here's the example above, depicted in Storybook:

The Storybook "Accessibility" tab correctly doesn't report any color-contrast issues here. Any idea what could be causing this issue?

Hi all,

An issue we experienced in work recently was the need to disable a specific low priority rule that affected a large number of stories while an upstream dependency worked to fix this issue.

To this we had to manually add the disabledRules parameter to all the stories affected.

I was wondering do you have any thoughts on making disabledRules a global config option.

There are a number of reasons I could see for this.

1. In cases like the above where an issue affects a large number of stories
2. When one low priority issue cause the tests to fail, and the only option is to increase the severity (e.g. going from minor to moderate because duplicate-id fails a bunch)
3. Or to simply never run a check that you don't care about.

If this was something you think we should do, the obvious approach is to add a new cli config option.

Do you have any thoughts, I could possible take a shot at implementing this if you wished.

Cheers,
Eoin

Hi,
my setup:

• node v18.9.1
• axe-storybook-testing: 6.2.0
• storybook 6.5.12

if i run the tests with default config, everything works as it should npx axe-storybook

But with reporter options i get this: npx axe-storybook --reporter markdown

Thanks for the great tool!

I have a trouble with a pseudoelement misfiring a color-contrast rule in a manner described here. Apparently, a way to ignore a pseudoelement when testing color contrast is to modify an option to include ignorePseudo: true, as in:

axe.configure({
  rules: [{
    id: 'color-contrast-minor',
    matches: 'color-contrast-matches',
    any: ['color-contrast-minor']
  }],
  checks: [{
    id: 'color-contrast-minor',
    evaluate: 'color-contrast-evaluate',
    metadata: {
      impact: 'minor'
    },
    options: {
      contrastRatio: {
        normal: {
          expected: 4.5,
          minThreshold: 4.0
        },
        large: {
          expected: 3,
          minThreshold: 2.5
        }
      },
      ignoreUnicode: true,
      ignoreLength: false,
      ignorePseudo: false,
      boldValue: 700,
      boldTextPt: 14,
      largeTextPt: 18,
      shadowOutlineEmMax: 0.1
    }
  }]
});

I however have no idea how to add this option – runOptions of axe-storybook-testing seems to correspond to axe.run() rather than axe.configure(). Is there any way to set this ignorePseudo option before running it?

I got a Storybook HTML 6.4.0-beta.13 error for each story:

 Error: page.evaluate: Evaluation failed: TypeError: storybookClientApi.store is not a function
           at emitSetCurrentStory (eval at evaluate (:3:1339), <anonymous>:10:36)
           at t.default.evaluate (<anonymous>:3:1362)
           at t.default.<anonymous> (<anonymous>:1:44)

Anything I’m missing in my setup?

We've been getting some flaky accessibility test build failures. It passes when I re-run the build. Could someone pair with me on this?

https://github.com/chanzuckerberg/edu-ps-intervention-tracking/actions/runs/8085775219/job/22094050662?pr=242

Here is the full log.
7_🤟 Accessibility tests.txt

Currently, when disabling a rule, it can only be disabled for the specific story or globally. Supporting selectors would be a safer way of ignoring a violation across stories as Axe will then only ignore very specific elements and keep reporting violations on other elements of this rule.

I'm still investigating this issue, so apologies if my description is vague.

What's currently happening is I'm running the same axe-storybook command on an unchanged Storybook directory and getting inconsistent results.

My command:

yarn axe-storybook --build-dir ./axe-storybook --failing-impact moderate --pattern TabButton

(the --pattern is likely insignificant; I've been noticing these inconsistent results for a while and applied this pattern just to try and narrow this down)

Sometimes, this passes:

Other times, it reports issues:

The story file I'm testing:

import { getRenderOptions } from '@linkedin/storybook-glimmer';
import {TabButton} from '../test/glimmer/index';
import defaultData from '/Users/tomstewa/Multiproducts/3/guest-ui-lib_trunk/test/data/components/common/tabs/tab-button/default.json';


export default {
  title: 'common/tabs/TabButton',
  parameters: {
    SetupPreview: { storyComponent: TabButton },
    axe: {
      disabledRules: ['heading-order', 'valid-lang'],
    }
  }
};


export const TabButton_with_dataFile_default = (args) => {
  return {
    componentClass: TabButton,
    renderOptions: getRenderOptions(args),
    assets: {
      css: 'css/components/common/tabs/tab-button-test.css',
      js: ''
    }
  };
};
TabButton_with_dataFile_default.args = defaultData;

I've tried to test this by setting --headless false and using different browsers; the inconsistent results still happen:

yarn axe-storybook --build-dir ./axe-storybook --failing-impact moderate --pattern TabButton --headless false --browser firefox

Not sure if anyone will be able to replicate this issue on their end, so any advice on how I can debug this would be appreciated. I'm currently trying to figure out if/how I can set a debugger breakpoint in the playwright-generated browser.

Again, early adopter of the new Storybook 6.5.0-alpha.64 version.

I’m getting the following error:

Error! The suite failed to run
An error was encountered before we started testing stories. Likely this means the browser failed to open.

Error: page.evaluate: Error: Cannot call extract() unless you call cacheAllCSFFiles() first.
    at StoryStore.extract (file:///Users/svenwolfermann/Sites/aperto/drv-design-system/storybook-static/4020.885ff43e.iframe.bundle.js:2:187701)
    at StoryStore.raw (file:///Users/svenwolfermann/Sites/aperto/drv-design-system/storybook-static/4020.885ff43e.iframe.bundle.js:2:188867)
    at checkStories (eval at evaluate (:3:2389), <anonymous>:6:65)
    at eval (eval at evaluate (:3:2389), <anonymous>:26:5)
    at new Promise (<anonymous>)
    at fetchStoriesFromWindow (eval at evaluate (:3:2389), <anonymous>:2:10)
    at t.default.evaluate (<anonymous>:3:2412)
    at t.default.<anonymous> (<anonymous>:1:44)

Can wait for the stable release, just want to let you know.

Setup

"node": ">=16.13"
....
"@chanzuckerberg/axe-storybook-testing": "^6.2.0"
....
"@storybook/addon-a11y": "^6.5.14"
"@storybook/addon-actions": "^6.5.14"
"@storybook/addon-essentials": "^6.5.14"
"@storybook/addon-interactions": "^6.5.14"
"@storybook/addon-links": "^6.5.14"
"@storybook/addon-react-native-web": "^0.0.19"
"@storybook/jest": "^0.0.10"
"@storybook/react": "^6.5.14"
"@storybook/testing-library": "^0.0.13"

Error

Hi ! 👋
We have an error when we run axe-storybook with chromatic.
Our storybook version is compatible and storybook doesn't have any errors.

Run yarn axe-storybook --storybook-address https://61fd537ecf081f003a135235-njwzppbnpc.chromatic.com/
yarn run v1.22.19
$ /home/runner/work/pass-culture-app-native/pass-culture-app-native/node_modules/.bin/axe-storybook --storybook-address https://61fd537ecf081f003a135235-njwzppbnpc.chromatic.com/
Error: Stories could not be retrieved from storybook!

Please check that...
- You're using a compatible version of Storybook
- Storybook doesn't have any errors

Otherwise this is likely a bug with axe-storybook-testing.

Original error message: Error: page.evaluate: TypeError: Cannot read properties of undefined (reading 'storyStore')
    at fetchStoriesFromWindow (eval at evaluate (:191:30), <anonymous>:3:[4](https://github.com/pass-culture/pass-culture-app-native/actions/runs/3741131221/jobs/6350454544#step:4:5)1)
    at UtilityScript.evaluate (<anonymous>:193:1[7](https://github.com/pass-culture/pass-culture-app-native/actions/runs/3741131221/jobs/6350454544#step:4:8))
    at UtilityScript.<anonymous> (<anonymous>:1:44)
    at /home/runner/work/pass-culture-app-native/pass-culture-app-native/node_modules/@chanzuckerberg/axe-storybook-testing/build/browser/StorybookPage.js:24:1[9](https://github.com/pass-culture/pass-culture-app-native/actions/runs/3741131221/jobs/6350454544#step:4:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 1.

Code

We don't want to build storybook twice, so we used the storybook url generated by chromatic with --storybook-address to launch the axe tests like this:

name: 'Chromatic'
on: push
jobs:
  chromatic-deployment:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: '100'
      - name: Install dependencies
        run: yarn
      - name: Publish to Chromatic
        id: publish
        uses: chromaui/action@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
          onlyChanged: true
    outputs:
      storybookUrl: ${{steps.publish.outputs.storybookUrl}}

  test-accessibility:
    runs-on: ubuntu-latest
    needs: chromatic-deployment
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: '1'
      - name: Install dependencies
        run: yarn
      - name: Run axe storybook accessibility tests
        run: yarn axe-storybook --storybook-address ${{needs.chromatic-deployment.outputs.storybookUrl}}

Do you have any idea why we have this error ? 🤷‍♂️
Thanks in advance for your assistance

For code bases that just start adding in a11y testing, there might be a mix of stories that are ready to be actively tested to prevent regressions and ones that aren't. Though stories could be turned off/skipped, it'd be nice to still be able to get metrics on what would fail with it enabled without failing the entire run.

It'd be great if there were a config option where you could mark a story with a level, with options for warn or error (where error is the default). That way you could still run axe on everything that it could be run on, but you only fail if the ones that shouldn't have errors started failing.