storybookjs / test-runner Goto Github PK

Describe the bug

I'm not sure to call this a bug or a feature, but I find myself wishing that the story name displayed in the test output matched storyName rather than the name of the exported CSF constant. This would make it easier to find the test in the storybook sidebar, in cases where there's a custom storyName that's being used.

Steps to reproduce the behavior

1. Create a CSF story with a custom storyName that does not match the story's exported name.
2. Run test-runner
3. Notice that the test name displayed does not match the storyName

Additional context

This is a pretty minor request, but wanted to document it since I did find it tripping me up once or twice.

Describe the bug

When using SSL with a self-signed certificate to run the development server, the test runner (with at least the Chromium browser) will reject the SSL connection, probably because it fails to validate the CA signing the certificate (that's my educated guess).

The test runner needs an option to either inject the root CA into the running instance of the browser, or disable verifying SSL certificates when connecting through HTTPS.

Steps to reproduce the behavior

To test this, generate a CA cert and a corresponding domain certificate. For convenience, I've added two bash scripts to generate the CA and domain wildcard self-signed certs to enable SSL. Not that you will need to add the root.pem certificate to your trusted root CA store in order to be able to use certificates generated by this CA.

generate-ca

#!/bin/bash

set -eu

ROOT_DIR=$(dirname $0)
BUILD_DIR="$ROOT_DIR/build"

mkdir -p "$BUILD_DIR"

openssl genrsa -out "$BUILD_DIR/root.key" 2048
openssl req -x509 -new -nodes -key "$BUILD_DIR/root.key" -sha256 -days 365 -out "$BUILD_DIR/root.pem"
openssl x509 -text -noout -in "$BUILD_DIR/root.pem"

generate-domain-cert

#!/bin/bash

set -e

ROOT_DIR=$(dirname $0)
BUILD_DIR="$ROOT_DIR/build"

ROOT_CERT="$BUILD_DIR/root.pem"
ROOT_KEY="$BUILD_DIR/root.key"

if [ ! -f "$ROOT_CERT" ]
then
  echo "Root certificate not found! Use generate-ca to generate a root CA certificate"
  exit 1
elif [ ! -f "$ROOT_KEY" ]
then
  echo "Root certificate key not found! Use generate-ca to generate a root CA certificate"
  exit 1
fi

if [ "$1" = "" ]
then
  echo "You must provide a FQDN name, e.g. ./generate-domain-name domain.name"
  exit 2
fi

DOMAIN_NAME="$1"
DOMAIN_DIR="$BUILD_DIR/$DOMAIN_NAME"

mkdir -p "$DOMAIN_DIR"

DOMAIN_KEY="$DOMAIN_DIR/wildcard.key"
DOMAIN_CERT="$DOMAIN_DIR/wildcard.crt"
DOMAIN_CSR="$DOMAIN_DIR/wildcard.csr"
DOMAIN_CONFIG_NAME="$DOMAIN_DIR/config.cnf"

openssl genrsa -out "$DOMAIN_KEY" 2048

cat <<EOF > "$DOMAIN_CONFIG_NAME"
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = $DOMAIN_NAME
[v3_req]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = 127.0.0.1
DNS.1 = localhost
DNS.2 = *.localhost
DNS.3 = $DOMAIN_NAME
DNS.4 = *.$DOMAIN_NAME
EOF

openssl req -new -out "$DOMAIN_CSR" -key "$DOMAIN_KEY" -config "$DOMAIN_CONFIG_NAME"

openssl x509 -req -in "$DOMAIN_CSR" \
-CA "$ROOT_CERT" \
-CAkey "$ROOT_KEY" \
-CAcreateserial \
-out "$DOMAIN_CERT" \
-days 365 \
-sha256 \
-extensions v3_req \
-extfile "$DOMAIN_CONFIG_NAME"

openssl verify -CAfile "$ROOT_CERT" "$DOMAIN_CERT"

openssl x509 -text -noout -in "$DOMAIN_CERT"

Run the scripts:

./generate-ca
./generate-domain-cert mydomain.com

Make sure that mydomain.com resolves to the IP of the machine running the Storybook server.

Then run the Storybook dev server together with the test runner:

concurrently -k -s first \
  "start-storybook -p 6006 -h mydomain.com --https --ssl-ca ./build/root.pem --ssl-cert ./build/mydomain.com/wildcard.crt --ssl-key ./build/mydomain.com/wildcard.key" \
  "wait-on tcp:6006 && STORYBOOK_URL=https://mydomain.com:6006 yarn test-storybook"

Verify that the Storybook is reachable by issuing CURL or by opening the URL in a browser (this verifies your system trusts the root CA cert you've generated):

curl https://mydomain:6006/

The test runner fails when trying to run tests:

$ yarn run test-storybook
yarn run v1.22.17
$ test-storybook --url "${STORYBOOK_URL:-https://mydomain.com:6006}"
[test-storybook] It seems that your Storybook instance is not running at: https://mydomain.com:6006/. Are you sure it's running?

If you're not running Storybook on the default 6006 port or want to run the tests against any custom URL, you can pass the --url flag like so:

yarn test-storybook --url http://localhost:9009

More info at https://github.com/storybookjs/test-runner#getting-started
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Expected behavior

The CURL should respond with HTML (as a sanity check) and the test runner should be able to run the tests.

Environment

• OS: Debian
• Node.js version: v16.13.0
• NPM version: 8.1.3
• Browser (if applicable): Chromium

👋 I am trying to add test-runner to an existing project that already has Storybook and Jest running, when following the Getting started instructions, I end up with the following error for each of my existing stories:

 FAIL   browser: chromium  spec/storybook/pages/request/index.stories.tsx
  ● Test suite failed to run

    TypeError: Jest: a transform must export a `process` function.

      at ScriptTransformer._getTransformer (node_modules/@jest/transform/build/ScriptTransformer.js:360:13)
      at ScriptTransformer.transformSource (node_modules/@jest/transform/build/ScriptTransformer.js:427:28)
      at ScriptTransformer._transformAndBuildScript (node_modules/@jest/transform/build/ScriptTransformer.js:569:40)
      at ScriptTransformer.transform (node_modules/@jest/transform/build/ScriptTransformer.js:607:25)
      at interopRequireDefault (node_modules/@storybook/core-common/dist/cjs/utils/interpret-require.js:64:16)

Steps to reproduce

1. run yarn storybook in terminal, wait for it to finish
2. run yarn test-storybook in another terminal

Current packages

• My major versions of ts-jest and jest match.
• I am using Webpack 4

Any thoughts on this?

Describe the bug

This seems not to be an actual bug in the test-runner, but rather in Storybook. Here's an example of a diagram that shows the sequence of steps in the test-runner:

Steps to reproduce the behavior

1. Change the Demo story in Button.stories.js from:

export const Demo = (args) => (
  <button type="button" onClick={() => args.onSubmit('clicked')}>
    Click
  </button>
);

to:

export const Demo = (args) => (
  <button type="button" onClick={() => { throw new Error('boom') }}>
    Click
  </button>
);

2. Run yarn test-storybook

The boom error should not leak into the FindBy story, which actually does not even use the same component as Demo:

Describe the bug

Right after I run the command yarn test-storybook --url http://localhost:4400 this error appears. I have the storybook up and running.

I was trying also with the stories.json file, so with the -s flag, but this returned another error:

Environment

• OS: MacOs Monterey
• Node.js version: 14.17.5
• NPM version: 6.14.4

Additional context

We are using the nrwl plugin for storybook - @nrwl/storybook in version 12.9.0 (unfortunately I can't update it at the moment).

I'm curious if it's a version issue like I need to update to the newest one? Because it's no matter if I provide a config path or not, the output is the same.

Question

How to use chromatic stories.json in GitHub Actions with test-runner?

Background

Our storybooks are hosted on chromatic, then we're aiming to execute interaction test in GitHub Actions by using Chromatic hosting stories.json. However, we cannot pass through authentication by this test-runner. I've tried like this

$ REFERENCE_URL=<chromatic-permalink> TARGET_URL=<chromatic-permalink> yarn test-storybook 

Cited from https://github.com/storybookjs/test-runner#2-running-against-locally-built-storybooks-in-ci

NOTE: Building Storybook locally makes it simple to test Storybooks that could be available remotely, but are under authentication layers. If you also deploy your Storybooks somewhere (e.g. Chromatic, Vercel, etc.), the Storybook URL can still be useful with the test-runner. You can pass it to the REFERENCE_URL environment variable when running the test-storybook command, and if a story fails, the test-runner will provide a helpful message with the link to the story in your published Storybook instead.

Describe the bug

Currently the test-runner selects *.stories.[tj]sx? but this information is already more precisely specified in .storybook/main.js.

We should use main.js to select the stories:

• Regular globs
• Strip out MDX
• Handle CSF3-style specifiers

Describe the bug

After installing @storybook/test-runner and following the Getting Started documentation, starting storybook fails to start and throws the following error:

ModuleNotFoundError: Module not found: Error: Can't resolve './.storybook/@storybook' in </home/....>

Steps to reproduce the behavior

1. Install storybook/react 6.4.19 with addon-essentuals, addon-actions, addon-interaction, add-links
2. Install @storybook/jest 0.0.9 and @storybook/testing-library 0.0.9
3. install @storybook/test-runner 0.0.4
4. Follow the getting started and add this to the ./storybook/main.js:

  stories: ['@storybook/addon-interactions'],
  features: {
    interactionsDebugger: true,
  },
};

5. run storybook yarn storybook (in my case npm run storybook)
6. Storybook fail to start with error:

ModuleNotFoundError: Module not found: Error: Can't resolve './.storybook/@storybook' in </home/....>

Expected behavior

Storybook should be able to start without error.

Screenshots and/or logs

npm run storybook logs:

ModuleNotFoundError: Module not found: Error: Can't resolve './.storybook/@storybook' in '~/web-app'
    at ~/web-app/node_modules/webpack/lib/Compilation.js:2014:28
    at ~/web-app/node_modules/webpack/lib/ContextModuleFactory.js:210:15
    at ~/web-app/node_modules/neo-async/async.js:2830:7
    at ~/web-app/node_modules/neo-async/async.js:6877:13
    at ~/web-app/node_modules/webpack/lib/ContextModuleFactory.js:180:26
    at finishWithoutResolve (~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:307:11)
    at ~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:381:15
    at ~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:430:5
    at eval (eval at create (~/web-app/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at ~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:430:5
    at eval (eval at create (~/web-app/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:27:1)
    at ~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:87:43
    at ~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/Resolver.js:430:5
    at eval (eval at create (~/web-app/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:16:1)
    at ~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/forEachBail.js:16:12
    at ~/web-app/node_modules/webpack/node_modules/enhanced-resolve/lib/AliasPlugin.js:103:14
resolve './.storybook/@storybook' in '~/web-app'
  using description file: ~/web-app/package.json (relative path: .)
    Field 'browser' doesn't contain a valid alias configuration
    using description file: ~/web-app/package.json (relative path: ./.storybook/@storybook)
      ~/web-app/.storybook/@storybook doesn't exist

Dev Dependencies:

"@babel/core": "^7.17.5",
"@storybook/addon-actions": "^6.4.19",
"@storybook/addon-essentials": "^6.4.19",
"@storybook/addon-interactions": "^6.4.19",
"@storybook/addon-links": "^6.4.19",
"@storybook/builder-webpack5": "^6.4.19",
"@storybook/jest": "^0.0.9",
"@storybook/manager-webpack5": "^6.4.19",
"@storybook/react": "^6.4.19",
"@storybook/test-runner": "^0.0.4",
"@storybook/testing-library": "^0.0.9",
"webpack": "^5.69.1",
"webpack-merge": "^5.8.0"

Environment

• OS: Arch/Manjaro
• Node.js version: v16.8.0
• NPM version: 7.21.0
• Browser (if applicable): Firefox
• Browser version (if applicable): 98
• Device (if applicable):

Additional context

Add any other context about the problem here.

Describe the bug

If I eject the config in an esm-project, the filename test-runner-jest.config.js is interpreted as esm-module and jest will fail to run. As the filename is hardcoded, I'm unable to rename the file to test-runner-jest.config.cjs.

Two possible solutions:

1. make the filename of the config adjustable via parameter
2. test for the existance of test-runner-jest.config.cjs in the cli

Describe the bug

When running the test-runner against .js files, I get this error:

It should not happen because @storybook/testing-react has nothing to do with playwright.
The error goes away when the file is changed to .tsx

Is it possible to show a trimmed version of the error when displayed in the error summary? (e.g. using something like https://www.npmjs.com/package/jest-compact-reporter)

I seem to get this error seemingly at random when running the tests.
Following the stack trace just takes me to the very bottom of my story file.

 FAIL   browser: chromium  src/Tag.stories.tsx (26.779 s)
  ● Base Components/Tag › Variants › smoke-test

    page.evaluate: ReferenceError: __test is not defined

      at eval (eval at evaluate (:3:2389), <anonymous>:4:17)
      at t.default.evaluate (<anonymous>:3:2412)
      at t.default.<anonymous> (<anonymous>:1:44)
      at testFn (src/Tag.stories.tsx:84:37)
      at Object.<anonymous> (src/Tag.stories.tsx:99:17)

Describe the bug

I am configuring my storybook to run test-runner following this guide:
Test runner guide

and when I run yarn test-storybook --url http://localhost:4400

I get an error.

yarn test-storybook --url http://localhost:4400
yarn run v1.22.17
$ test-storybook --url http://localhost:4400
● Validation Error:

  Option "testMatch" must be of type:
    array
  but instead received:
    string

  Example:
  {
    "testMatch": [
      "**/__tests__/**/*.[jt]s?(x)",
      "**/?(*.)+(spec|test).[jt]s?(x)"
    ]
  }

  Configuration Documentation:
  https://jestjs.io/docs/configuration

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Environment

• OS: OSX
• Node.js version: v16.13.2
• NPM version: 8.1.2

Currently, to set up API hooks, you need to create a custom jest config.

Here's a strawman API for adding the API hooks that doesn't require ejecting:

// .storybook/test-runner.js
module.exports = {
  hooks: {
    preRender: (page, context) => { ... },
    postRender: (page, context) => { ... }
  },
}

Notes:

• This allows us to change the test runner more easily in the future
• This uses the .storybook directory, and maybe we should consider that for the jest config also

// jest-config.js
try {
  const { hooks } = require('.storybook/test-runner');
  ..
} catch(err) {}

Would be great to be able to filter tests for stories that have the play function, or the ones without play function

Describe the Feature

I have a Themed story in many of my stories files which simply renders other stories together in a "light mode" and "dark mode". If I'm testing each of those other stories, then I would like to save the time and prevent my Themed stories from being smoke-tested. As far as I know, there's no way to do this currently.

One approach might be to add a parameter, like

const Themed = {
  ...myCSF,
  parameters: {
    testing: {
      disable: true,
    },
  },
}

If this approach was taken, it might also be a way to set the it description of the test (see #71), even though it's a little clunky. In addition, maybe it would be a cool way to add setup/teardown hooks for a particular story. Even though I haven't found that I need those so far, but I think that's because each test is a fresh page load. Which is great for isolation, but definitely slows things down. And the individual stories themselves are not new page loads, so there could need to be setup/teardown between them. But that's probably a separate issue.

We currently support all Jest CLI arguments and pass them directly to Jest.

Per @ghengeveld 's suggestion, let's prune these down to the set that we want to support. That way if users want other options, they can make feature requests and we can add them as we see fit.

Being more disciplined about this helps us understand which features users want, and also allows us to switch off Jest in the future should we see fit.

Describe the bug

Like the playwright folder, it should be included

and that results in this error:

Currently customizing the jest config is a manual process. Add an eject command to:

• Simplify customization
• Make explicit that this kind of customization may be incompatible with future updates to the test runner

Todo:

• Add comments to the config file explaining the "smarts"
• Re-eject to get the most up-to-date config & diff against your previous changes

Describe the bug

Running test-storybook fails for automated stories (.stories.play files).

Steps to reproduce

Following the steps here.

Install the test runner:

yarn add --dev @storybook/test-runner

Ensure Jest is installed.

Add a package script to run the tests:

"scripts": { "test-storybook": "test-storybook", }

Run storybook (test runner runs against a running instance, default on localhost:6006):

yarn storybook

Run the tests:

'yarn test-storybook`

Expected behaviour

Tests pass

Screenshots and/or logs

UnderConstruction.stories.tsx

import { ComponentStory, ComponentMeta } from '@storybook/react';
import { UnderConstruction } from './UnderConstruction';

export default {
  title: 'GW-WEB-UI/UnderConstruction/Manual',
  component: UnderConstruction,
  argTypes: {
    text: { name: 'text' },
    buttonText: { name: 'buttonText' },
  },
  parameters: {
    backgrounds: {
      values: [
        { name: 'red', value: '#f00' },
        { name: 'green', value: '#0f0' },
        { name: 'blue', value: '#00f' },
        { name: 'hot pink', value: '#FF69B4' },
      ],
    },
    decorators: [
      (Story: any) => (
        <div className="p-8 flex items-center justify-center bg-white">
          <div className="w-full max-w-xs mx-auto">
            <Story />
          </div>
        </div>
      ),
    ],
  },
} as ComponentMeta<typeof UnderConstruction>;

const template: ComponentStory<typeof UnderConstruction> = (args) => <UnderConstruction {...args} />;

export const Primary = template.bind({});

Primary.args = {
  text: 'Under Construction',
  buttonText: 'Click me to change colour!',
};

export const Secondary = template.bind({});

Secondary.args = {
  ...Primary.args,
  buttonText: 'See what happens when you click me!',
};

UnderConstruction.stories.play.tsx

import { ComponentStory, ComponentMeta } from '@storybook/react';
import { within, userEvent } from '@storybook/testing-library';
import { UnderConstruction } from './UnderConstruction';

export default {
  title: 'GW-WEB-UI/UnderConstruction/Automated',
  component: UnderConstruction,
  argTypes: {
    text: { name: 'text' },
    buttonText: { name: 'buttonText' },
  },
} as ComponentMeta<typeof UnderConstruction>;

const template: ComponentStory<typeof UnderConstruction> = (args) => <UnderConstruction {...args} />;

export const Primary = template.bind({});

Primary.args = {
  text: 'Under Construction',
  buttonText: 'Click me to change colour!',
};

Primary.play = async ({ canvasElement }) => {
  const canvas = within(canvasElement);
  await userEvent.type(canvas.getByTestId('input'), 'Hello');
  await userEvent.type(canvas.getByTestId('input'), ' Slowpoke', {
    delay: 500,
  });
  await userEvent.click(canvas.getByTestId('setColorButton'));
};

Error output:

main.js (storybook configuration file)

module.exports = {
  stories: ['../src/**/**/*.stories.@(js|jsx|ts|tsx)', '../src/**/**/*.stories.play.@(js|jsx|ts|tsx)'],
  addons: [
    '@storybook/addon-links',
    '@storybook/addon-essentials',
    '@storybook/addon-interactions',
    '@storybook/addon-a11y',
    // required for tailwind css
    {
      name: '@storybook/addon-postcss',
      options: {
        postcssLoaderOptions: {
          implementation: require('postcss'),
        },
      },
    },
    '@storybook/addon-viewport',
  ],
  features: {
    interactionsDebugger: true,
  },
  framework: '@storybook/react',
  core: {
    builder: '@storybook/builder-vite',
  },
  //ISSUE: https://github.com/storybookjs/storybook/issues/17852
  features: { storyStoreV7: true },
};

Environment

MacOS Big Sur
Node 16.10.0
Storybook 6.4.2
Jest
Typescript

Describe the Feature

Sometimes it is nice to give some extra context around what the test is doing, and I find that the story name is not always an ideal place for this. I've taken to just adding comments at the top of my play function, but these don't show up in the test result output of course. It might be nice to have a way to customize what is shown there instead of always just smoke-test or play-test.

Additional context

I'm not really sure how'd you'd accomplish this, short of adding another property onto CSF objects, which does not feel ideal. My only other idea would be to allow a return value from play functions, which you could use to pass before/after hooks, and maybe also a description.

Describe the bug

When multiple stories fail in the test runner, the following error occurs:

Notice that the first error report is correct, but the second is not.

I suspect the page instance is being shared between tests and if one breaks the rest of the tests will not have access to that instance correctly

Steps to reproduce the behavior

1. Add a failure to a story play function, e.g.

FindBy.play = async ({ canvasElement }) => {
  const canvas = within(canvasElement);
  await canvas.findByRole('button');
-  await expect(true).toBe(true);
+  await expect(true).toBe(false);
};

2. Run the test runner
3. See error

Update README

• Remove JSDOM documentation
• Add CI recipes

I have an issue at the moment where some tests are failing, and I have decided to temporarily disable them so that the whole command doesn't fail.

Jest has a CLI option called --testPathIgnorePatterns=|[array] - but it doesn't seem to work with the test runner.

When I run it - I get:

test-storybook --testPathIgnorePatterns Button.stories.jsx

error: unknown option '--testPathIgnorePatterns'
Usage: test-storybook [options]

I'm not sure if I'm doing something wrong, or I've misunderstood how the test runner handles flags/options - I assumed that all jest options would be "carried through" to Jest.

When attempting to run the code in this repo, I am getting the error:

Error: Cannot find module '../dist/cjs/util'
Require stack:
- /Users/jeremy/Sites/storybookjs/test-runner/bin/test-storybook.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:925:15)
    at Function.Module._load (node:internal/modules/cjs/loader:769:27)
    at Module.require (node:internal/modules/cjs/loader:997:19)
    at require (node:internal/modules/cjs/helpers:92:18)
    at Object.<anonymous> (/Users/jeremy/Sites/storybookjs/test-runner/bin/test-storybook.js:11:49)
    at Module._compile (node:internal/modules/cjs/loader:1108:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1137:10)
    at Module.load (node:internal/modules/cjs/loader:973:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/jeremy/Sites/storybookjs/test-runner/bin/test-storybook.js'
  ]
}
error Command failed with exit code 1.

Steps to reproduce the behavior

1. clone repo
2. run yarn storybook
3. in a new terminal window, run yarn test-storybook

I noticed that jest-playwright package recommends using @playwright/test over playwright. Any plans to move this this in the future, since it has typescript support out of the box?

• jest-playwright https://github.com/playwright-community/jest-playwright#configuration
• @playwright/test https://playwright.dev/docs/intro#installation

Describe the bug

Published test-runner-jest.config.js should refer to absolute paths:

  globalSetup: '@storybook/test-runner/playwright/global-setup.js',
  globalTeardown: '@storybook/test-runner/playwright/global-teardown.js',
  testEnvironment: '@storybook/test-runner/playwright/custom-environment.js',

This makes it harder to develop, but easier for users to copy the config file and edit themselves. Plus a fresh install fails for me with:

MMBP15:design-system shilman$ yarn test-storybook
yarn run v1.22.17
$ /Users/shilman/projects/storybookjs/design-system/node_modules/.bin/test-storybook
● Validation Error:

  Test environment ./playwright/custom-environment.js cannot be found. Make sure the testEnvironment configuration option points to an existing node module.

  Configuration Documentation:
  https://jestjs.io/docs/configuration

error Command failed with exit code 1.

Steps to reproduce the behavior

Add @storybook/test-runner to a project that doesn't have jest already installed e.g. @storybook/design-system

Hi guys! Thanks for your work.
Can you help me with a receipt for jest-dom methods availability?

test-runner-jest.config.js:

import '@testing-library/jest-dom';

module.exports = {
  setupFilesAfterEnv: ['<rootDir>/jest-setup.js'],
};

story.js:

import { expect } from '@storybook/jest';

...

// Inside `play` method:
expect(canvas.getByTestId('my-element-1')).toBeNull();
expect(canvas.getByTestId('my-element-2')).toBeVisible();

toBeNull — is worked;
toBeVisible — Is not worked: is not a function

I want to get jest-dom methods for expect, but how to do it right, can you help me? Sorry for the interruption.

Currently, the test runner works by transforming CSF files into Jest tests. This is great because you can edit your stories & re-run your tests interactively.

However, sometimes all you have is a link to a publicly hosted storybook. If we had a --stories-json mode that could transform stories.json into a set of tests, then you could run tests against any Storybook with zero configuration.

Describe the bug

test-storybook is displaying PASS when the story has a TypeError(s).

Steps to reproduce the behavior

1. create a story that has type errors, e.g. BlogPost.stories.tsx

// BlogPost.stories.tsx
import BlogPost from './BlogPost'

export const generated = () => {
  return <BlogPost />
}

export default { title: 'Components/BlogPost' }

// BlogPost.tsx
interface Props {
  post: { id: number, title: string, body: string, createdAt: string }
}


const BlogPost = ({ post }: Props) => {
  return (

    <article>
      <header className="mt-4">
        <p className="text-sm">
          {new Intl.DateTimeFormat('en-US', {  year: 'numeric', month: 'long', day: 'numeric' }).format(new Date(post.createdAt))}
        </p>
        <h2 className="text-xl mt-2 font-semibold">
          <div className="hover:text-blue-600">{post.title}</div>
        </h2>
      </header>
      <div className="mt-2 mb-4 text-gray-900 font-light">{post.body}</div>
    </article>

  );
}

export default BlogPost

2. run test-storybook

Expected behavior

test-storybook fails for those stories which have a TypeError(s)

Screenshots and/or logs

$ yarn workspace web test-storybook
yarn workspace v1.23.0-20211220.1904
yarn run v1.23.0-20211220.1904
$ test-storybook
page loaded in 1847ms.
page loaded in 1943ms.
page loaded in 2047ms.
page loaded in 1919ms.
page loaded in 2193ms.
page loaded in 2281ms.
page loaded in 2230ms.
page loaded in 2289ms.
 PASS   browser: chromium  src/pages/ContactPage/ContactPage.stories.tsx
 PASS   browser: chromium  src/pages/AboutPage/AboutPage.stories.tsx (5.231 s)
 PASS   browser: chromium  src/pages/BlogPostPage/BlogPostPage.stories.tsx (5.235 s)
 PASS   browser: chromium  src/components/BlogPost/BlogPost.stories.tsx (5.101 s)
 PASS   browser: chromium  src/layouts/BlogLayout/BlogLayout.stories.tsx (5.221 s)
 PASS   browser: chromium  src/pages/HomePage/HomePage.stories.tsx (5.186 s)
 PASS   browser: chromium  src/components/BlogPostCell/BlogPostCell.stories.tsx (5.441 s)
 PASS   browser: chromium  src/components/BlogPostsCell/BlogPostsCell.stories.tsx (5.061 s)

Test Suites: 8 passed, 8 total
Tests:       14 passed, 14 total
Snapshots:   0 total
Time:        8.308 s
Ran all test suites.
Done in 10.74s.
Done in 11.12s.

Describe the bug

This is used by the test runner to link to the interactions addon in Storybook. However, we don't have this information in stories.json mode. Always linking to interactions will fix stories.json mode and also simplify the code significantly.

• Chromatic
• Mealdrop
• Design system

example: in CE project, some logos took time to load which resulted in flaky snapshot tests

Describe the bug

Starting npm run test-storybook fails with an TypeError

TypeError: Jest: Got error running globalSetup - /Users/ingo.fahrentholz/Developer/delme/storybook-testing/node_modules/@storybook/test-runner/playwright/global-setup.js, reason: Class extends value #<Object> is not a constructor or null
    at Object.getPlaywrightEnv (/Users/ingo.fahrentholz/Developer/delme/storybook-testing/node_modules/jest-playwright-preset/lib/PlaywrightEnvironment.js:59:48)
    at Object.<anonymous> (/Users/ingo.fahrentholz/Developer/delme/storybook-testing/node_modules/jest-playwright-preset/lib/PlaywrightEnvironment.js:242:27)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
    at Object.newLoader (/Users/ingo.fahrentholz/Developer/delme/storybook-testing/node_modules/pirates/lib/index.js:141:7)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/ingo.fahrentholz/Developer/delme/storybook-testing/node_modules/jest-playwright-preset/index.js:1:18)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
    at Object.newLoader (/Users/ingo.fahrentholz/Developer/delme/storybook-testing/node_modules/pirates/lib/index.js:141:7)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)

Steps to reproduce the behavior

start storybook
start test-storybook

Expected behavior

test-runner should run the tests defined as play functions inside the story.

Environment

• OS: macOS 12.3.1
• Node.js version: 16.14.2
• NPM version: 8.5.0

Repository

https://github.com/ifahrentholz/storybook-test-runner-bug

Describe the bug

Jest should either be a dependency or a peer dependency

EDIT: It is a peer dependency, but we should document in README

Steps to reproduce the behavior

Add @storybook/test-runner to a project that doesn't have jest already installed e.g. @storybook/design-system

Describe the bug

I've just added @storybook/test-runner to the storybook-builder-vite, and found that tests are not running correctly on our svelte example project. I get the following message:

YN0000: No tests found, exiting with code 1
Run with `--passWithNoTests` to exit with code 0
In /home/runner/work/storybook-builder-vite/storybook-builder-vite/examples/svelte
   22 files checked.
   testMatch: /home/runner/work/storybook-builder-vite/storybook-builder-vite/examples/svelte/stories/**/*.stories.mdx, /home/runner/work/storybook-builder-vite/storybook-builder-vite/examples/svelte/stories/**/*.stories.@(js|jsx|ts|tsx|svelte) - 0 matches
   testPathIgnorePatterns: /node_modules/ - 22 matches
   testRegex:  - 0 matches
Pattern:  - 0 matches
ERROR: "test" exited with 1.

This can be seen in a CI run here: https://github.com/eirslett/storybook-builder-vite/runs/5538433882?check_suite_focus=true#step:7:47

Steps to reproduce the behavior

1. Clone the storybook-builder-vite repo
2. git checkout storybook-test-ci
3. yarn install
4. cd examples/svelte
5. yarn build-storybook
6. yarn test-ci

Expected behavior

I expect that the stories are found, and tests are run correctly.

Additional context

We are using svelte component stories, maybe the test-runner does not support them? I didn't see any notes or issues about it yet, though.

@yannbf WDYT about prefixing all our env vars with SB_ to avoid clashes?

Currently there is no stylized way to retrieve story parameters in API hooks.

This would be useful for reusing:

• viewports configurations
• a11y parameters
• etc.

This could be provided automatically by the test-runner as part of the context argument.

  async preRender(page, context) {
    const { parameters } = context;
  },

Or maybe on=demand:

  async preRender(page, context) {
    const parameters = await getParameters(page, context);
    // OR
    const parameters = await context.getParameters(page);
  },

The benefit of on-demand is that the user would only "pay" the performance cost of retrieving parameters when they are actually used.

Describe the bug

The test runner runs tests full screen inside the default playwright Chrome viewport. It does not resize the viewport according to the viewport parameter of the story.

Steps to reproduce the behavior

I have a component where a dropdown is hidden on large screens using a media query. I configured the story to use a mobile2 viewport by default and wrote a play function to test the visibility of the element:

export const Mobile = Template.bind({})
Mobile.parameters = {
  viewport: {
    defaultViewport: 'mobile2',
  },
}
Mobile.play = async ({ canvasElement }) => {
  const canvas = within(canvasElement)
  const toggleButton = await canvas.findByTestId('dropdownToggle')
  await waitFor(() => expect(toggleButton).toBeVisible())
}

When I run storybook locally and use the interactions addon, the test runs fine, but when I run the test runner, it fails.

I debugged the test runner using this custom test-runner-jest.config.js, --maxWorkers 1 and by adding sleep timeouts after every line. I can see that the playwright Chrome window has the wrong size and that it loads the story iframe in fullscreen.

const { getJestConfig } = require('@storybook/test-runner');

module.exports = {
  ...getJestConfig(),
  testEnvironmentOptions: {
    'jest-playwright': {
      launchOptions: {
        headless: false,
      }
    },
  },
}

Environment

• Storybook 6.4.20
• @storybook/test-runner 0.0.4
• @storybook/testing-library 0.0.9

Additional context

As a workaround, I wrote a prerender hook in .storybook/test-runner.js that resizes the playwright page if the test has Mobile in its name. A solution that does not rely on magic strings would be better though.

module.exports = {
  preRender(page, story) {
    if (story.name.includes('Mobile')) {
      page.setViewportSize({ width: 896, height: 414 })
    }
  },
}

All my stories are in mdx format.
It would be great if I could run all the play function tests therein automatically and during the ci pipeline.

I tried using --stories-json, but the runner still did not find any tests...

• Improve error message when Storybook is not running (e.g. mention about TARGET_URL for different ports)
• If people run against a deployed Storybook (e.g. we can make a regex to check if URL is localhost or IP address), provide a prompt (that is skipped with —yes) to run stories-json mode to get things synchronised

Describe the bug

Currently stories without title resolve into this error:

• Remove support for "test" function
• Add unit tests for transformer

Describe the Feature request

Currently, it seems that a new browser window is being opened for each test. Unfortunately, that causes a long loading time in development for vite projects, which do not bundle modules in development, which can result in thousands of requests at page load (with the promised payback of faster hot module reload times). Currently, I'm seeing load times of 20-30 seconds per test. Production is much faster, because vite does bundle production builds.

Ideally, it would be great if a single browser window could remain open, and navigation to different stories could happen via history.replaceState or something, rather than full new page loads. I'm not sure how feasible that is, honestly, but it could help speed things up, maybe.

Additional context

FWIW, web-test-runner uses websockets in its interactions with playwright, which means it only needs to open the browser once, and each test is injected via ws. Not sure if that would work here, but it's an idea.

Let's set up test-runner in this repo:

• Add to CI
• Add interesting repo examples to the stories (loaders, timeouts, etc.)