<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clip

Some examples: <div class="snippet-clipboard-content notranslate position-relative

and a third: <div class="snippet-clipboard-content notranslate position-relative o

and a forth: <div class="snippet-clipboard-content notranslate position-relative o

yes, we need to get to the bottom of these - <a class="user-mention notranslate" data-

This intermittently failed for me:- Failures: <l

got another random failure <div class="snippet-clipboard-content notranslate posit

intermittent js failure?,about agileventures/websiteone

Comments (58)

freeranger commented on May 29, 2024 1

Some examples:

Scenario: See my avatar on My account page                                                                      # features/users/avatar.feature:10
    When I click pulldown link "My account"                                                                       # features/step_definitions/user_steps.rb:259
      Ambiguous match, found 2 elements matching link or button "My account" (Capybara::Ambiguous)
      ./features/step_definitions/user_steps.rb:261:in `/^I click pulldown link "([^"]*)"$/'
      features/users/avatar.feature:11:in `When I click pulldown link "My account"'
    Then I should see my avatar image                                                                             # features/step_definitions/avatar_steps.rb:1
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-15-21-25-55.123.html

@javascript
  Scenario: Insert media model accepts full url youtube links                                                            # features/documents.feature:157
    Given I am logged in                                                                                                 # features/step_definitions/user_steps.rb:19
    And I am using the Mercury Editor to edit document "Guides"                                                          # features/step_definitions/mercury_steps.rb:27
    And I am focused on the "document body" within the Mercury Editor                                                    # features/step_definitions/contained_search_steps.rb:33
    And I click on the "Insert Media" button within the Mercury Toolbar                                                  # features/step_definitions/contained_search_steps.rb:33
    And I fill in "YouTube URL" with "https://www.youtube.com/watch?v=foo" within the Mercury Editor Modal               # features/step_definitions/contained_search_steps.rb:33
    And I click "Insert Media" within the Mercury Editor Modal                                                           # features/step_definitions/contained_search_steps.rb:33
    Then the Mercury Editor modal window should not be visible                                                           # features/step_definitions/mercury_steps.rb:61
    And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:356:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:165:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-16-09-23-59.288.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-16-09-23-59.288.png

@javascript
  Scenario: Closing an existing video and opening a new one should update the player # features/scrums.feature:19
    Given I visit "/scrums"                                                          # features/step_definitions/basic_steps.rb:60
    And I click the first scrum in the timeline                                      # features/step_definitions/scrums_steps.rb:24
    Then I should see a modal window with the first scrum                            # features/step_definitions/scrums_steps.rb:13
    When I close the modal                                                           # features/step_definitions/scrums_steps.rb:36
    And I click the second scrum in the timeline                                     # features/step_definitions/scrums_steps.rb:30
    Then I should see a modal window with the second scrum                           # features/step_definitions/scrums_steps.rb:13
      expected to find css "#playerTitle" but there were no matches. Also found "", which matched the selector but not all filters. (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/scrums_steps.rb:17:in `/^I should see a modal window with the (first|second) scrum$/'
      features/scrums.feature:25:in `Then I should see a modal window with the second scrum'
capybara-screenshot could not detect a screenshot driver for 'poltergeist_debug'. Saving with default with unknown results.
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-01-04.211.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-01-04.211.png

from websiteone.

tansaku commented on May 29, 2024

got this for a second time today on a trivial documentation update

from websiteone.

tansaku commented on May 29, 2024

and a third:

Failing Scenarios:
cucumber features/documents.feature:159 # Scenario: Insert media model accepts full url youtube links
cucumber features/documents.feature:170 # Scenario: Insert media model rejects badly formatted youtube links

from websiteone.

tansaku commented on May 29, 2024

and a forth:

cucumber features/documents.feature:159

from websiteone.

freeranger commented on May 29, 2024

I get this quite frequently on my ci build of WSO which is basically a clone of the main one - it can happen two or three times in a row and then just work.....possibly a timing issue somewhere?

from websiteone.

tansaku commented on May 29, 2024

most likely

from websiteone.

freeranger commented on May 29, 2024

Could these two scenarios be disabled until someone looks at the issue?
I get one or both failing pretty constantly on my semaphore build - it's only pot luck if it works or not.
The downside ofc to disabling is that nobody looks at them because the issue has "gone away"...

from websiteone.

tansaku commented on May 29, 2024

yes, we need to get to the bottom of these - @decareano posted a great link on the subject in localsupport - if I can stabilize the grader I might try to make this my priority tomorrow:

https://spin.atomicobject.com/2012/04/27/intermittent-test-failures/

from websiteone.

mtc2013 commented on May 29, 2024

This intermittently failed for me:-

Failures:

EventsController GET show assigns a event instance
Failure/Error: expect(assigns(:recent_hangout)).to eq(event_instance)

   expected: #<EventInstance id: 43, event_id: 23, title: "Hangout_91", hangout_url: "http://hangout.test", created_at: "2016-03-02 14:16:09", updated_at: "2016-03-02 14:16:09", uid: "uid_102", category: "Category_89", project_id: 41, user_id: 194, yt_video_id: "yt_video_id_84", participants: [["91", {:person=>{:displayName=>"Broadcaster91", :id=>"youtube_id_91", :isBroadcaster=>"true"}}], ["91", {:person=>{:displayName=>"Participant_91", :id=>"youtube_id_91", :isBroadcaster=>"false"}}]], hoa_status: nil>
        got: nil

   (compared using ==)
 # ./spec/controllers/events_controller_spec.rb:32:in `block (3 levels) in <top (required)>'

from websiteone.

tansaku commented on May 29, 2024

got another random failure

(::) failed steps (::)

expected to find css "img[src*=\"/assets/mercury/missing-image.png\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/basic_steps.rb:348:in `/^I should see an image with source "([^"]*)"$/'
./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
features/documents.feature:155:in `Then I should see an image with source "/assets/mercury/missing-image.png" within the Mercury Editor'

Failing Scenarios:
cucumber features/documents.feature:146 # Scenario: Missing Image gets added when a user can inserts an invalid image

from websiteone.

tansaku commented on May 29, 2024

I just got the same rspec failure as @mtc2013 had locally

  1) EventsController GET show assigns a event instance
     Failure/Error: expect(assigns(:recent_hangout)).to eq(event_instance)

       expected: #<EventInstance id: 100, event_id: 92, title: "Hangout_210", hangout_url: "http://hangout.test", created_at: "2016-03-03 08:05:58", updated_at: "2016-03-03 08:05:58", uid: "uid_224", category: "Category_186", project_id: 211, user_id: 587, yt_video_id: "yt_video_id_209", participants: [["216", {:person=>{:displayName=>"Broadcaster216", :id=>"youtube_id_216", :isBroadcaster=>"true"}}], ["216", {:person=>{:displayName=>"Participant_216", :id=>"youtube_id_216", :isBroadcaster=>"false"}}]], hoa_status: nil>
            got: nil

       (compared using ==)
     # ./spec/controllers/events_controller_spec.rb:32:in `block (3 levels) in <top (required)>'

went away on a second run ...

from websiteone.

tansaku commented on May 29, 2024

just got three docs failures in CI again:

cucumber features/documents.feature:146 # Scenario: Missing Image gets added when a user can inserts an invalid image
cucumber features/documents.feature:159 # Scenario: Insert media model accepts full url youtube links
cucumber features/documents.feature:170 # Scenario: Insert media model rejects badly formatted youtube links

from websiteone.

tansaku commented on May 29, 2024

this in @freerangers cucumber pull request:

  Scenario: Show a document                                      # features/documents.feature:47
    Given I am on the "Show" page for project "hello mars"       # features/step_definitions/basic_steps.rb:256
    When I click the sidebar link "Guides"                       # features/step_definitions/documents_steps.rb:44
    Then I should be on the "Show" page for document "Guides"    # features/step_definitions/basic_steps.rb:252
    And I should see "Guides"                                    # features/step_definitions/basic_steps.rb:185
    And I should see "New content 0"                             # features/step_definitions/basic_steps.rb:185
    And I should see a link to "Edit" page for document "Guides" # features/step_definitions/basic_steps.rb:260
      expected to find link or button nil with text "Guides" but there were no matches. Also found "", "Log in", "Sign up", "", "", "About", "Projects", "Members", "Articles", "Events", "Upcoming events", "Past scrums", "Past events", "Getting started", "", "", "hello world", "Documentation", "", "hello mars", "", "", "", "Become a supporter", "×", "List of Projects", "hello mars", "Revisions", "comments powered by Disqus.", "comments powered by Disqus", "About Us", "Getting Started", "Dashboard", "Facebook", "Twitter", "Makers Academy", "AirPair", "Becoming a sponsor", "[email protected]", which matched the selector but not all filters. (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:263:in `/^I should( not be able to)? see a link to "([^"]*)" page for ([^"]*) "([^"]*)"$/'
      features/documents.feature:54:in `And I should see a link to "Edit" page for document "Guides"'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-03-12-33-13.782.html

  @javascript
  Scenario: Insert media model accepts full url youtube links                                                            # features/documents.feature:159
    Given I am logged in                                                                                                 # features/step_definitions/user_steps.rb:19
    And I am using the Mercury Editor to edit document "Guides"                                                          # features/step_definitions/mercury_steps.rb:27
    And I am focused on the "document body" within the Mercury Editor                                                    # features/step_definitions/contained_search_steps.rb:33
    And I click on the "Insert Media" button within the Mercury Toolbar                                                  # features/step_definitions/contained_search_steps.rb:33
    And I fill in "YouTube URL" with "https://www.youtube.com/watch?v=foo" within the Mercury Editor Modal               # features/step_definitions/contained_search_steps.rb:33
    And I click "Insert Media" within the Mercury Editor Modal                                                           # features/step_definitions/contained_search_steps.rb:33
    Then the Mercury Editor modal window should not be visible                                                           # features/step_definitions/mercury_steps.rb:61
    And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:352:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:167:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-03-12-33-37.869.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-03-03-12-33-37.869.png

Failing Scenarios:
cucumber features/documents.feature:47 # Scenario: Show a document
cucumber features/documents.feature:159 # Scenario: Insert media model accepts full url youtube links

from websiteone.

tansaku commented on May 29, 2024

so cucumber features/documents.feature:47 was consistently failing for @freeranger while cucumber features/documents.feature:159 is one of our intermittent failures ... highlighting how confusing intermittent failures can be

from websiteone.

mtc2013 commented on May 29, 2024

@mercury_step @javascript
  Scenario: Editing Pitch content with Mercury Editor                                                    # features/projects.feature:184
    Given I am logged in                                                                                 # features/step_definitions/user_steps.rb:19
    And I am on the "Show" page for project "hello mars"                                                 # features/step_definitions/basic_steps.rb:256
    And I click the "Join Project" button                                                                # features/step_definitions/basic_steps.rb:77
    And I am using the Mercury Editor to edit project "hello mars"                                       # features/step_definitions/mercury_steps.rb:27
    When I fill in the editable field "Pitch" for "project" with "This is my exciting marketing content" # features/step_definitions/mercury_steps.rb:9
    And I click "Save" within Mercury Editor toolbar                                                     # features/step_definitions/mercury_steps.rb:1
    Then I should see a flash "The project has been successfully updated."                               # features/step_definitions/basic_steps.rb:193
      expected to find css ".alert" with text "The project has been successfully updated." but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:195:in `/^I should( not)? see a flash "([^"]*)"$/'
      features/projects.feature:191:in `Then I should see a flash "The project has been successfully updated."'
    Then I should be on the "Show" page for project "hello mars"                                         # features/step_definitions/basic_steps.rb:252
    And I should see "This is my exciting marketing content"                                             # features/step_definitions/basic_steps.rb:185
    HTML screenshot: /Users/michael/Desktop/WebsiteOne/tmp/capybara/screenshot_2016-03-08-14-28-34.642.html
    Image screenshot: /Users/michael/Desktop/WebsiteOne/tmp/capybara/screenshot_2016-03-08-14-28-34.642.png

from websiteone.

freeranger commented on May 29, 2024

Is it me or is it getting worse? I pretty much can't get any semaphore build green now....

from websiteone.

tansaku commented on May 29, 2024

it feels like something is going on ...

from websiteone.

tansaku commented on May 29, 2024

please post specifics of fails that you get here @freeranger - it helps us debug

from websiteone.

tansaku commented on May 29, 2024

thanks @freeranger - I've been trying to mark the intermittent failure points in this branch: https://github.com/AgileVentures/WebsiteOne/tree/806_failure_points which also uses some cucumber hooks to try and give those scenarios longer to run.

I suspect that we won't be able to achieve stability without losing some functionality or adding proper sandboxing via puffing_billy or vcr or possibly both

from websiteone.

freeranger commented on May 29, 2024

hmm...may be beyond my current skill level so no promises but I'll see what I can do....

from websiteone.

freeranger commented on May 29, 2024

The Mercury Editor tests seem failing before it even gets to post to youtube so puffing_billy won't help here.
This is the sort of thing I see for example when the "Insert media model accepts full url youtube links" scenario fails:

Where it failed to fill in the YouTube URL field so there is nothing on screen when submit is pressed so validation is failing.

from websiteone.

freeranger commented on May 29, 2024

I increased the timeout and did a load of builds (21 at last count) here:

https://semaphoreci.com/freeranger/websiteone/branches/806_intermittent_failures

And a lot of them worked, more than some of the other builds of different branches that I kicked off around the same time - though I did get one new failure:

As I said yesterday, I think it is largely, if not entirely, down to not entering the YouTuble URL video so validation fails and the page is never submitted.
I tried various things to select the radio button next to it first (rather than start typing in the text field, which results in the radio being selected) but I couldn't find the radio on the page - I don't know if it is the complexity of having this as an iframe or just my lack of capybara skills, but trying various suggestions from mr google - xpath selections, css finds etc, didn't work for me....and perhaps wouldn't have made any difference anyway...

It's not a timeout issue and it's not a network (as in talking to youtube etc) issue - just seems to be intermittently failing to fill in forms.
Also, IME, seems to happen much more on semaphore than it does running locally...

from websiteone.

tansaku commented on May 29, 2024

Thanks for this review.

I think puffing billy might still help because we are not "posting to youtube", we are posting to our own server. What puffing billy sandboxes is not a post to our own server (which we complete in these acceptance tests), but all the network connections going out from the browser to load remote javascripts and images etc. All things that can block and slow things down and create intermittent failures when javascript is trying to get the right forms on the screen.

You might be right that it won't help in this case, but we do need to add it, and we should try it.

from websiteone.

freeranger commented on May 29, 2024

@tansaku If you look at the screenshot from the weekend though, the form is indeed loaded
The problem is there is nothing filled in on the form because this
And I fill in "YouTube URL" with "https://www.youtube.com/watch?v=foo" within the Mercury Editor Modal
isn't actually filling in the form.
Even if some remote javascript hadn't loaded, would that affect capybara finding the control and populating it? capybara itself is already loaded and has been able to bring up the form by clicking on a link or button....

from websiteone.

tansaku commented on May 29, 2024

It's a wild card so its something we should lock down. I find that part of the trick to making progress with this kind of error is to not be a stickler for the logic of it, but make sure the hatches are battened down.

Also you don't know that the screenshot is completely accurate. Anyway, there comes a point that if you don't have a specific alternative to try, then why not "just try" the thing that I am suggesting, even if at the moment you can't see the logic of it ...? :-)

We can argue back and forth the logic which takes up a lot of time - more time than actually just trying it perhaps ...

from websiteone.

freeranger commented on May 29, 2024

Fair enough. I would like to have tried selecting the radio button rather than relying on the focus of the text box magically selecting it, but I couldn't get that working.
setting up puffing billy is beyond me at the moment - I've only barely scratched the surface of WSO to understand it, nevermind this added complexity - back to you or some other volunteer I'm afraid....

from websiteone.

tansaku commented on May 29, 2024

fair enough - see #932

from websiteone.

tansaku commented on May 29, 2024

More failures:

cucumber -p semaphoreci features/documents.feature:157 # Scenario: Insert media model accepts full url youtube links
cucumber -p semaphoreci features/documents.feature:168 # Scenario: Insert media model rejects badly formatted youtube links
cucumber -p semaphoreci features/scrums.feature:19 # Scenario: Closing an existing video and opening a new one should update the player

@yggie I'm trying to keep a record of all the places we see intermittent failures in this issue

    Then I should see a modal window with the second scrum                           # features/step_definitions/scrums_steps.rb:13
      expected "display: none; " to include "display: block;" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/scrums_steps.rb:15:in `/^I should see a modal window with the (first|second) scrum$/'
      features/scrums.feature:25:in `Then I should see a modal window with the second scrum'

    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)�� Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:175:in `Then I should see "is invalid" within the Mercury Modal'

    And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:360:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:165:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'

from websiteone.

tansaku commented on May 29, 2024

and annoyingly our new stripe test has an intermittent failure too!

cucumber -p semaphoreci features/charge_activity.feature:6

Your userAgent seems to be not supported by inline-style-prefixer. Feel free to open an issue.
    And I fill in appropriate card details     # features/step_definitions/charge_steps.rb:1
      Unable to find field "cc-exp" (Capybara::ElementNotFound)
      ./features/step_definitions/charge_steps.rb:7:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/charge_steps.rb:3:in `/^I fill in appropriate card details$/'
      features/charge_activity.feature:9:in `And I fill in appropriate card details'

from websiteone.

tansaku commented on May 29, 2024

and this one again:

    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)�� Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:175:in `Then I should see "is invalid" within the Mercury Modal'

cucumber -p semaphoreci features/documents.feature:168 # Scenario: Insert media model rejects badly formatted youtube links

from websiteone.

tansaku commented on May 29, 2024

and another:

    Then I should see a modal window with the second scrum                           # features/step_definitions/scrums_steps.rb:13
      expected "display: none; " to include "display: block;" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/scrums_steps.rb:15:in `/^I should see a modal window with the (first|second) scrum$/'
      features/scrums.feature:25:in `Then I should see a modal window with the second scrum'

cucumber -p semaphoreci features/scrums.feature:19

from websiteone.

tansaku commented on May 29, 2024

this looks interesting: https://bibwild.wordpress.com/2016/02/18/struggling-towards-reliable-capybara-javascript-testing/

from websiteone.

tansaku commented on May 29, 2024

So I'm focusing on this last individual failure features/scrums.feature:19 and I note that if I reduce the timeout settings to 0 or 0.1 I get these kinds of errors:

Scenario: Closing an existing video and opening a new one should update the player # features/scrums.feature:19
    Given I visit "/scrums"                                                          # features/step_definitions/basic_steps.rb:64
    And I click the first scrum in the timeline                                      # features/step_definitions/scrums_steps.rb:24
      Timed out waiting for response to {"id":"d9ae4d24-8368-4982-9eaa-a8cee66f8880","name":"body","args":[]}. It's possible that this happened because something took a very long time (for example a page load was slow). If so, setting the Poltergeist :timeout option to a higher value will help (see the docs for details). If increasing the timeout does not help, this is probably a bug in Poltergeist - please report it to the issue tracker. (Capybara::Poltergeist::TimeoutError)
      ./features/step_definitions/scrums_steps.rb:25:in `/^I click the first scrum in the timeline$/'
      features/scrums.feature:21:in `And I click the first scrum in the timeline'

which are different in kind from the errors we get in batch. With a poltergeist timeout of 1 and a Capybara.default_max_wait_time of 1 the tests passes for me reliably on my laptop, when I have an internet connection.

I note however that despite our attempt to plug the billy cache leaks that just from running this single test I have a set of new cache entries:

    features/req_cache/get_s.ytimg.com_915edb4b5f224648b2607dc57119661559439dc1.yml
    features/req_cache/get_s.ytimg.com_97ea91ae8364c347e1bafbcf3a60d3830420c3e5.yml
    features/req_cache/get_www.google.com_dda3078e234d52b37f250121ebb31829d67f6628.yml
    features/req_cache/get_www.youtube.com_754eb41a2f135e6eafbc40d9a4acf987250d14d4.yml
    features/req_cache/get_www.youtube.com_771e045e849dcf0e8325651601d2397edca211c9.yml
    features/req_cache/get_www.youtube.com_7ce01254dc22fb0ec070a5d431931551248b7971.yml

indicating that this test is still hitting some remote servers ... none of which have signficant variation in their parameters ... but I note that several of them have (particularly the www.youtube.com ones) are getting 301 (moved permanently) responses and as a consequence may not be being cached effectively - I thought that puffing billy cached these by default, but there is an extra setting for that that I can add:

    non_successful_cache_disabled = false

I also note that if I wrap this test in a @vcr tag we also get server network connections being cached.

from websiteone.

tansaku commented on May 29, 2024

hmm, interesting to note that if I put @vcr tags on the documents.feature (one of the prime culprits) I get a fresh load of billy leaks ...

    features/req_cache/get_a.disquscdn.com_005fa385265710de9b800462d9db61ef21ec4c99.yml
    features/req_cache/get_a.disquscdn.com_2aa0b18e4bd9e580dc09cf4f417b21146ddf05b6.yml
    features/req_cache/get_a.disquscdn.com_2bc23452ce0ff2a22e561bb4519a23945f0ba922.yml
    features/req_cache/get_a.disquscdn.com_bca73df499bedab281bd05c54b027f262189d85d.yml
    features/req_cache/get_a.disquscdn.com_d008b8a1c5d90ecb9fc1701353d06b9481fa230b.yml
    features/req_cache/get_disqus.com_442dc930b209a7b62857b10a08ab762f55c39f56.yml
    features/req_cache/get_google.com_8a28b678b788b1c051c987cd0d6f8c697a48b352.yml
    features/req_cache/get_s.ytimg.com_915edb4b5f224648b2607dc57119661559439dc1.yml
    features/req_cache/get_s.ytimg.com_97ea91ae8364c347e1bafbcf3a60d3830420c3e5.yml
    features/req_cache/get_ssl.google-analytics.com_7b9a51d3b1e9af83bfc375f4813db28046b99513.yml
    features/req_cache/get_www.google.com_dda3078e234d52b37f250121ebb31829d67f6628.yml
    features/req_cache/get_www.youtube.com_36a2809495d3decd3d207abd85e5162a07e86986.yml
    features/req_cache/get_www.youtube.com_e6558ad62c301081754b9ab9f6a15aeb1284f981.yml

from websiteone.

tansaku commented on May 29, 2024

what's really odd is that these billy leaks don't appear when running in batch (without vcr) - need to check variations ...

from websiteone.

tansaku commented on May 29, 2024

interesting - just running the document.feature by itself (without any vcr mods) generates new billy cache files, implying that somehow there is some different behaviour in terms of what sites the browser is hitting when we run the tests in batch and we run them at the feature file level.

For scrums I also saw this behaviour at the scenario level ...

from websiteone.

tansaku commented on May 29, 2024

this implies some fundamental inconsistency in the browser based behaviour depending on whether we are in batch mode or individual feature or scenario mode, which is not a good thing ... locking down everything with vcr and billy will help reduce variation, but if there is something else causing this change between the different running modes then we won't get stability until we can identify and then either patch or remove that underlying problem ...

from websiteone.

tansaku commented on May 29, 2024

Options:

comprehensive VCR sandboxing is required
upgrading all the relevant gems, e.g. capybara, phantomjs, poltergeist etc.
review features (and cucumber env settings one by one) --> start with a branch that had a single feature test - and check that passes on CI and developer machines, and then we would add in each acceptance in turn, checking what the tags mean - ensuring we have billy and vcr sandboxing, and working through everything to ensure maximum consistency ... could start with usual suspects
making tests run in fixed order (i.e. not randomised by cucumber)
play with rails concurrency settings
prune back on javascript tests - can some be thrown out?
comment out the intermittent tests - or just have them not run on CI using a tag like @intermittent_fail

from websiteone.

freeranger commented on May 29, 2024

At least some of the common failures are when trying to select form elements and no network traffic should be involved at that point, as we have discussed earlier in the thread.
It may be that the focus is in the wrong area since adding billy and/or vcr in those areas didn't fix the problems...so maybe it's more to do with field selection/validation than network reliability?
Sandboxing may be a good idea generally, but not necessarily as a solution to these failures...

from websiteone.

tansaku commented on May 29, 2024

thanks @freeranger - it may well be that network sandboxing will not fix this, however note that previously I only added billy, not VCR. What we have established so far is that billy by itself has not conclusively fixed these issues. Network variability causing other tests to hang around and influence each other may still be a strong component of this problem, since VCR has not yet been added throughout for a complete sandbox.

That's what we're doing in the cleanroom branch, as well as being on the lookout for other issues, but being able to do with a more manageable subset of the tests ... There seem to be several factors in play, and to be able to manage that I think we have to work hard to reduce all sources of external variation from our tests.

from websiteone.

tansaku commented on May 29, 2024

so got this with what we think is fully sandboxed code:

    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)× Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:174:in `Then I should see "is invalid" within the Mercury Modal'

cucumber -p semaphoreci features/documents.feature:167 # Scenario: Insert media model rejects badly formatted youtube links

from websiteone.

freeranger commented on May 29, 2024

Ah that's annoying, perhaps leading to where I was going with it (ie not the lack of sandboxing being the issue) but still, lot of good work there so even if it doesn't help at all for this case, the tests should be more reliable overall I should think - so don't be disheartened!
Does this happen for you locally or was it a semaphore build that failed?
Do you want people to blast it again a load of times to see if we get it?
Have you tried it in an environment that wasn't a mac, maybe c9.io or an ubunto vm to see if you can get it to fail on a more regular basis and hopefully be easier to track down then?

from websiteone.

tansaku commented on May 29, 2024

Another failure in a new location (this time in the old code):

 And I create a document named "A New Guide to the Galaxy" # features/step_definitions/activity_feed_steps.rb:15
    And I create a project named "Build NCC-1701 Enterprise"  # features/step_definitions/activity_feed_steps.rb:25
      Timed out waiting for response to {"id":"2866f7e2-cdae-49c3-b35e-05b0eaad0654","name":"click","args":[11,6]}. It's possible that this happened because something took a very long time (for example a page load was slow). If so, setting the Poltergeist :timeout option to a higher value will help (see the docs for details). If increasing the timeout does not help, this is probably a bug in Poltergeist - please report it to the issue tracker. (Capybara::Poltergeist::TimeoutError)
      ./features/step_definitions/basic_steps.rb:82:in `/^I click the "([^"]*)" button$/'
      ./features/step_definitions/activity_feed_steps.rb:26:in `/^I create a project named "([^"]*)"$/'
      features/public_activity.feature:23:in `And I create a project named "Build NCC-1701 Enterprise"'
capybara-screenshot could not detect a screenshot driver for 'poltergeist_billy_debug'. Saving with default with unknown results.
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-00-50.831.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-00-50.831.png

Failing Scenarios:
cucumber -p semaphoreci features/public_activity.feature:25

@freeranger to answer some of your questions:

that was a failure on CI
the latest (and now complete) cleanroom branch could be blasted a few times to see if we get local failures - @mtc2013 at least had none locally while we were developing the whole thing which involved about 40+ runs, and I only had one failure, but that was with some extra cache files that I then added ...
we haven't tried on other platforms apart from it being run on @diraulo, my and @mtc2013's systems. A repeat run on C9 is a good idea - particularly in that all our projects should have C9 setup guides - I tried to reduce timeouts to make more reliable failures, but made no progress there.

My main plan of attack next is once we have the cleanroom in develop is wait for the next failure and then pounce on it for a detailed analysis with all networking variation removed ... there's also making the cleanroom work with ruby 2.3 ...

from websiteone.

freeranger commented on May 29, 2024

@tansaku Ok I'll try to get a c9 build up and running at the weekend and give it a good few runs.

from websiteone.

tansaku commented on May 29, 2024

just got this locally:

expected to find css ".alert" with text "The project has been successfully updated." but there were no matches (RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/basic_steps.rb:199:in `/^I should( not)? see a flash "([^"]*)"$/'
features/projects.feature:192:in `Then I should see a flash "The project has been successfully updated."'

Failing Scenarios:
cucumber features/projects.feature:185 # Scenario: Editing Pitch content with Mercury Editor

but also had some stray network connections:

:scope: 
:url: http://csi.gstatic.com/csi?v=3&s=gwidget&action=hangout&it=wdi.116,wri.116&srt=21&e=abc_e81,abc_e97,abc_e100,abc_e122,abc_e123,abc_e30,abc_e79,abc_e127,abc_l0,abc_m0,abc_n0,abc_m0n0,abc_u0&rt=
:body: ''
:status: 204
:method: get
:headers:
  Access-Control-Allow-Origin: "*"
  Date: Thu, 14 Apr 2016 14:45:53 GMT
  Pragma: no-cache
  Expires: Fri, 01 Jan 1990 00:00:00 GMT
  Cache-Control: no-cache, no-store, must-revalidate
  Last-Modified: Wed, 21 Jan 2004 19:51:30 GMT
  Content-Type: image/gif
  Server: Golfe2
  Content-Length: '0'
  Connection: close
:content: ''

from websiteone.

tansaku commented on May 29, 2024

another CI failure:

 And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor # features/step_definitions/contained_search_steps.rb:33
      expected to find css "iframe[src*=\"http://www.youtube.com/embed/foo?wmode=transparent\"]" but there were no matches (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:360:in `/^I should see an video with source "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:38:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:38:in `/^(.*) within the (.*)$/'
      features/documents.feature:164:in `And I should see an video with source "http://www.youtube.com/embed/foo?wmode=transparent" within the Mercury Editor'
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-04-14-15-14-41.579.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2016-04-14-15-14-41.579.png

Failing Scenarios:
cucumber -p semaphoreci features/documents.feature:156 # Scenario: Insert media model accepts full url youtube links

from websiteone.

tansaku commented on May 29, 2024

So we've got several more intermittent fails on CI with the new setup:

features/documents.feature:156
features/documents.feature:167
features/scrums.feature:22

The new sandboxed setup seems (?) more stable on individual developer machines, but if anything these fails on CI seem even more common? At least if something is reliably failing we have a better chance of fixing it ...

from websiteone.

tansaku commented on May 29, 2024

I was taking a little look at features/documents.feature:167 in more detail locally (it passes single and in batch on my local machine).

It occurred to me that perhaps there is a way to reset the JS browser inbetween test runs - which led me to find this post:

teampoltergeist/poltergeist#375

which gives us another thing to experiment with. What I tried first was printing out images around the elements that fails on CI like so:

  @javascript
  Scenario: Insert media model rejects badly formatted youtube links
    Given I am logged in
    And I am using the Mercury Editor to edit document "Guides"
    And I am focused on the "document body" within the Mercury Editor
    And I click on the "Insert Media" button within the Mercury Toolbar
    And I fill in "YouTube URL" with "https://www.youtube.io/watch?v=foo" within the Mercury Editor Modal
    And save a screenshot of the page at "/tmp/before_insert.png"
    And I click "Insert Media" within the Mercury Editor Modal
    And save a screenshot of the page at "/tmp/after_insert.png"
    Then I should see "is invalid" within the Mercury Modal
    And the Mercury Editor modal window should be visible

now interestingly that fails for me locally the way it fails on CI and the screen shots look like this:

from websiteone.

tansaku commented on May 29, 2024

and the error I get is this:

  And I click "Insert Media" within the Mercury Editor Modal                                            # features/step_definitions/contained_search_steps.rb:33
    And save a screenshot of the page at "/tmp/after_insert.png"                                          # features/step_definitions/basic_steps.rb:275
    Then I should see "is invalid" within the Mercury Modal                                               # features/step_definitions/contained_search_steps.rb:33
      expected to find text "is invalid" in "Insert Media (images and videos)× Images URL can't be blank Videos YouTube URL Vimeo URL Options Alignment Float Width Height" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/basic_steps.rb:191:in `/^I should( not)? see "([^"]*)"$/'
      ./features/step_definitions/contained_search_steps.rb:41:in `block (2 levels) in <top (required)>'
      ./features/step_definitions/contained_search_steps.rb:41:in `/^(.*) within the (.*)$/'
      features/documents.feature:176:in `Then I should see "is invalid" within the Mercury Modal'
    And the Mercury Editor modal window should be visible                                                 # features/step_definitions/mercury_steps.rb:61
    HTML screenshot: /Users/tansaku/Documents/GitHub/AgileVentures/WebsiteOne/tmp/capybara/screenshot_2016-04-15-12-08-29.698.html
    Image screenshot: /Users/tansaku/Documents/GitHub/AgileVentures/WebsiteOne/tmp/capybara/screenshot_2016-04-15-12-08-29.698.png

and the dumped screen looks like this:

the same stuff that @freeranger was looking at previously ...

from websiteone.

tansaku commented on May 29, 2024

and this is kind of strange, since it makes it look like the youtube URL has been inserted correctly with the correct radiobutton clicked, but then after insert media is clicked the modal reloads with a different kind of issue? Has the invalid issue appeared and then disappeared? Are we running too fast or too slow here ... weird ...

from websiteone.

tansaku commented on May 29, 2024

but the interesting thing is that if I run this again it fails in precisely the same way ...

from websiteone.

tansaku commented on May 29, 2024

the presence of saving the first screenshot it what appears to force the failure ...

from websiteone.

tansaku commented on May 29, 2024

here's a save of the correct output from a passing run:

from websiteone.

tansaku commented on May 29, 2024

hmmm https://robots.thoughtbot.com/write-reliable-asynchronous-integration-tests-with-capybara

from websiteone.

tansaku commented on May 29, 2024

hmm, so I added a step to check the YouTube URL checkbox in the case of failure, and that works and I can see it is checked:

but we are still stuck with the other error message ...

from websiteone.

tansaku commented on May 29, 2024

so I've added extra selectors of that YouTube checkbox in multiple places that might help with these document.feature issues.

However we also have:


  @javascript
  Scenario: Closing an existing video and opening a new one should update the player # features/scrums.feature:22
    Given I visit "/scrums"                                                          # features/step_definitions/basic_steps.rb:64
    And I click the first scrum in the timeline                                      # features/step_definitions/scrums_steps.rb:24
    Then I should see a modal window with the first scrum                            # features/step_definitions/scrums_steps.rb:13
    When I close the modal                                                           # features/step_definitions/scrums_steps.rb:28
    And I click the second scrum in the timeline                                     # features/step_definitions/scrums_steps.rb:24
      Firing a click at co-ordinates [583.5, 403.5] failed. Poltergeist detected another element with CSS selector 'html body.modal-open div#wrap div#main div.container div.col-lg-9 div#scrumVideo.modal.fade.in div.modal-dialog div.modal-content div.modal-body div.video-container iframe' at this position. It may be overlapping the element you are trying to interact with. If you don't care about overlapping elements, try using node.trigger('click'). (Capybara::Poltergeist::MouseEventFailed)
      ./features/step_definitions/scrums_steps.rb:25:in `/^I click the (first|second) scrum in the timeline$/'
      features/scrums.feature:27:in `And I click the second scrum in the timeline'
    Then I should see a modal window with the second scrum                           # features/step_definitions/scrums_steps.rb:13
    HTML screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-00-45.158.html
    Image screenshot: /home/runner/WebsiteOne/tmp/capybara/screenshot_2014-07-15-12-00-45.158.png

which is different again entirely ...

from websiteone.

tansaku commented on May 29, 2024

so we could try updating to:

When(/^I click the (first|second) scrum in the timeline$/) do |ordinal|
  page.all(:css, "a.scrum_yt_link").send(ordinal.to_sym).trigger('click')
end

from websiteone.

tansaku commented on May 29, 2024

or perhaps better to make the closing of the modals more reliable we should have:

When(/^I close the modal$/) do
  page.find(:css,'.close').click
  expect(page).not_to have_css('.close')
end

from websiteone.

intermittent js failure? about websiteone HOT 58 CLOSED

Comments (58)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent