Running CI tests in BuildKite only end up with the first attempt running without follo

You're welcome <a class="user-mention notranslate" data-hovercard-type="user" data-hov

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Retries with Cypress 5.x and BuildKite about knapsack-pro-cypress HOT 7 CLOSED

knapsackpro commented on May 29, 2024

Retries with Cypress 5.x and BuildKite

from knapsack-pro-cypress.

Comments (7)

ArturT commented on May 29, 2024 1

You're welcome @okeeffed I'm glad you solved the issue. :)

from knapsack-pro-cypress.

ArturT commented on May 29, 2024

Hi @okeeffed

Running tests on BuildKite in parallel using the approach shown in RSpec examples.

What do you mean? Are you talking about RSpec split by examples feature? https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it
It's only for RSpec. It's not related to Cypress.

Regarding Cypress itself. By retries do you mean a new feature in Cypress to auto retry failing tests?
Test Retries https://docs.cypress.io/guides/guides/test-retries.html#Introduction

In how it works section they mentioned "The following is a screen capture of what test retries looks like on the same failed test when run via cypress run."

@knapsack-pro/cypress uses cypress.run command. Knapsack Pro. is just a thin wrapper that calls cypress.run.
See

knapsack-pro-cypress/src/knapsack-pro-cypress.ts

Line 37 in 8942e04

const { runs: tests, totalFailed } = await cypress.run({

Here you can learn more about how it integrates with Cypress https://docs.knapsackpro.com/2020/how-to-build-native-integration-with-knapsack-pro-api-to-run-tests-in-parallel-for-any-test-runner-testing-framework

You can double-check if you have the lastest Cypress version installed and if you have properly configured Cypress test retries
https://docs.cypress.io/guides/guides/test-retries.html#Introduction

from knapsack-pro-cypress.

ArturT commented on May 29, 2024

I've just checked test retries and it works fine. I suspect there is something wrong with your Cypress configuration maybe :/

KnapsackPro/cypress-example-kitchensink@cb64530

and my log out

$ bin/knapsack_pro_cypress_test_file_pattern

2020-11-10T11:32:07.428Z [@knapsack-pro/core] info: POST https://api-staging.knapsackpro.com/v1/queues/queue

2020-11-10T11:32:07.930Z [@knapsack-pro/core] info: 200 OK

Request ID:
6084a19a-752d-4a88-b6b0-89fa4f15676c

Response body:
{
  queue_name: '23:87a63fcc43c39542ae03e3b389d77cf6',
  build_subset_id: null,
  test_files: [
    {
      path: 'cypress/integration/examples/actions.spec.js',
      time_execution: 20.116
    }
  ]
}

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    5.1.0                                                                              │
  │ Browser:    Electron 83 (headless)                                                             │
  │ Specs:      1 found (examples/actions.spec.js)                                                 │
  │ Searched:   cypress/integration/examples/actions.spec.js                                       │
  │ Params:     Tag: false, Group: false, Parallel: false                                          │
  │ Run URL:    https://dashboard.cypress.io/projects/47f44n/runs/337                              │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  examples/actions.spec.js                                                        (1 of 1)
  Estimated: 17 seconds


  Actions
    (Attempt 1 of 3) .type() - type into a DOM element
    (Attempt 2 of 3) .type() - type into a DOM element
    1) .type() - type into a DOM element
    ✓ .focus() - focus on a DOM element (341ms)
    ✓ .blur() - blur off a DOM element (766ms)
    ✓ .clear() - clears an input or textarea element (934ms)
    ✓ .submit() - submit a form (536ms)
    ✓ .click() - click on a DOM element (2837ms)
    ✓ .dblclick() - double click on a DOM element (416ms)
    ✓ .rightclick() - right click on a DOM element (482ms)
    ✓ .check() - check a checkbox or radio element (1419ms)
    ✓ .uncheck() - uncheck a checkbox element (1826ms)
    ✓ .select() - select an option in a <select> element (2213ms)
    ✓ .scrollIntoView() - scroll an element into view (761ms)
    ✓ .trigger() - trigger an event on a DOM element (629ms)
    ✓ cy.scrollTo() - scroll the window or element to a position (2357ms)


  13 passing (34s)
  1 failing

  1) Actions
       .type() - type into a DOM element:

      Timed out retrying
      + expected - actual

      -'[email protected]'
      +'[email protected]'

      at Context.eval (http://localhost:8080/__cypress/tests?p=cypress/integration/examples/actions.spec.js:107:52)




  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        14                                                                               │
  │ Passing:      13                                                                               │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  3                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     34 seconds                                                                       │
  │ Estimated:    17 seconds                                                                       │
  │ Spec Ran:     examples/actions.spec.js                                                         │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

On 3rd attempt test failed:

Actions
    (Attempt 1 of 3) .type() - type into a DOM element
    (Attempt 2 of 3) .type() - type into a DOM element
    1) .type() - type into a DOM element

from knapsack-pro-cypress.

ArturT commented on May 29, 2024

@okeeffed Is it possible that you have configured only 1 retry attempt in your cypress.json config file for the project?

This would explain why you see only

(Attempt 1 of 2).

As you can see in my log output. Cypress is printing attempts, but the last attempt is printed as a failed test so there is not shown something like (Attempt 3 of 3).
In my case on 3rd attempt test is just marked as failed. It looks like that's expected Cypress behavior (but might be confusing).

This explains why you don't see (Attempt 2 of 2)..

from knapsack-pro-cypress.

okeeffed commented on May 29, 2024

Apologies, there are a number of things I should have been more explicit about in the initial message.

Are you talking about RSpec split by examples feature?

What I meant was that the README only shows an example of setting up the initial parallelisation with BuildKite through an RSpec repo example and not anything for Cypress to compare to in case I had issues with settings. I should elaborate to say that we have had a parallelization working in our build for the last few months.

Is it possible that you have configured only 1 retry attempt in your cypress.json config file for the project?

The Cypress config is fine and running locally confirms that second attempts are made. The (Attempt 1 of 2) only shows when you have tests configured to try twice. The runMode is set to run twice (similar to your config).

As for the Cypress version, I am running 5.1.0.

Here was the full test return, initially I didn't post thinking it might be against work policy but it does not look like it will cause any issues:


2020-11-10T02:38:51.256Z [@knapsack-pro/core] info: POST https://api.knapsackpro.com/v1/queues/queue
--
  |  
  | 2020-11-10T02:38:51.872Z [@knapsack-pro/core] info: 200 OK
  |  
  | Request ID:
  | 9adb9bee-3156-4177-9a23-60e300227d2f
  |  
  | Response body:
  | {
  | queue_name: '1766:521be4abf583ac1152d1eb3def7ccc86',
  | build_subset_id: null,
  | test_files: [
  | {
  | path: 'cypress/integration/home/home.spec.ts',
  | time_execution: 16.987
  | },
  | {
  | path: 'cypress/integration/team_based_feedback/teamBasedFeedback.spec.ts',
  | time_execution: 13.327
  | }
  | ]
  | }
  | Missing baseUrl in compilerOptions. tsconfig-paths will be skipped
  |  
  | ====================================================================================================
  |  
  | (Run Starting)
  |  
  | ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  | │ Cypress:    5.5.0                                                                              │
  | │ Browser:    Chrome 85 (headless)                                                               │
  | │ Specs:      2 found (home/home.spec.ts, team_based_feedback/teamBasedFeedback.spec.ts)         │
  | │ Searched:   cypress/integration/home/home.spec.ts, cypress/integration/team_based_feedback/tea │
  | │             mBasedFeedback.spec.ts                                                             │
  | └────────────────────────────────────────────────────────────────────────────────────────────────┘
  |  
  |  
  | ────────────────────────────────────────────────────────────────────────────────────────────────────
  |  
  | Running:  home/home.spec.ts                                                               (1 of 2)
  |  
  |  
  | /
  | visiting the page
  | ✓ does not error or redirect to a 404 (2357ms)
  | task list
  | Complete reviews for user's direct reports
  | ✓ displays direct report users and links user through to complete their review (2227ms)
  | Request feedback about user's direct reports
  | (Attempt 1 of 2) displays direct report users and links user through to request feedback for them
  | 1) displays direct report users and links user through to request feedback for them
  | Review shared calibration
  | ✓ links user through to a review shared calibration (1073ms)
  | Give peer feedback
  | ✓ displays options for the user to give feedback to or decline the request (1317ms)
  | notifications list
  | filters
  | Type Filter
  | ✓ only displays notifications for selected category (1436ms)
  | Complete/Incomplete Filter
  | ✓ only displays notifications for incomplete of selected category (498ms)
  | Search name filter
  | ✓ displays only notifications that match search term (884ms)
  | notifications
  | completed requested feedback
  | ✓ displays a modal with your feedback to Philippe (1189ms)
  | completed given feedback
  | ✓ displays a modal with feedback given (1136ms)
  |  
  |  
  | 9 passing (24s)
  | 1 failing
  |  
  | 1) /
  | task list
  | Request feedback about user's direct reports
  | displays direct report users and links user through to request feedback for them:
  | AssertionError: Timed out retrying: Expected <div.ErrorPage--notification-container.flex.layout.vertical.center-center> not to exist in the DOM, but it was continuously found.
  | at Context.eval (http://performance-ui/__cypress/tests?p=cypress/integration/home/home.spec.ts:6310:22)
  |  
  |  
  |  
  |  
  | (Results)
  |  
  | ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  | │ Tests:        10                                                                               │
  | │ Passing:      9                                                                                │
  | │ Failing:      1                                                                                │
  | │ Pending:      0                                                                                │
  | │ Skipped:      0                                                                                │
  | │ Screenshots:  2                                                                                │
  | │ Video:        true                                                                             │
  | │ Duration:     24 seconds                                                                       │
  | │ Spec Ran:     home/home.spec.ts                                                                │
  | └────────────────────────────────────────────────────────────────────────────────────────────────┘

@knapsack-pro/cypress uses cypress.run command. Knapsack Pro. is just a thin wrapper that calls cypress.run.

Did that test you run above run on BuildKite? If so, this does make me think I something odd must be configured as I am unsure as to why local tests would retry. I'll bump retries up to repeat twice as you have and see if anything different occurs.

Documentation I had been referencing through this has been:

I just woke up here in Australia so I am going to take another look. I'll close this issue and consider chatting to Cypress about it.

from knapsack-pro-cypress.

okeeffed commented on May 29, 2024

Also, would be remiss of me not to say thank you - those replies went to a lot of effort and I will feel bad if my own configuration in our setup cost you time!

from knapsack-pro-cypress.

okeeffed commented on May 29, 2024

Yep, so in summary (now that I've gone back with fresh eyes): things were working as expected but Cypress - while supporting test retries from a describe block - still does not re-run the suite (only eligible for re-running single it blocks).

I did end up wasting your time, apologies on that one. If you have a link for support/donations I will send you money for a coffee if you would like!

from knapsack-pro-cypress.

Retries with Cypress 5.x and BuildKite about knapsack-pro-cypress HOT 7 CLOSED

Comments (7)

Related Issues (11)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent