npilon / planterbox Goto Github PK
View Code? Open in Web Editor NEWA plugin for nose2 for running behavior-driven tests.
License: MIT License
A plugin for nose2 for running behavior-driven tests.
License: MIT License
Running (deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c ../qa_tool/webtests/shawn_sauce_windows_7_c44_law-stage.cfg winslow.tests.features --plugin=nose2.plugins.mp -N=2 --log-level debug -v
Nose2 / planterbox would hang.
running with --log-level debug
DEBUG:nose2.main:Internal Error
Traceback (most recent call last):
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/nose2/main.py", line 265, in runTests
self.result = runner.run(self.test)
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/nose2/runner.py", line 53, in run
executor(test, result)
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/nose2/runner.py", line 41, in <lambda>
executor = lambda suite, result: suite(result)
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/unittest2/suite.py", line 87, in __call__
return self.run(*args, **kwds)
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/unittest2/suite.py", line 126, in run
test(result)
File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
test(result)
File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
return self.run(*args, **kwds)
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 168, in run
result=result,
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 248, in run_outline
example_scenario = substitute_steps(scenario, example)
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 432, in substitute_steps
for step in scenario
KeyError: u'see_all_link'
Internal Error: runTests aborted: u'see_all_link'
Only seems to hang when running in parallel.
Resolving error (typo in variable) in scenario outline solved issue.
Possibly need to better catch exception in scenario outlines.
Given the following test;
Feature: Test for Testing
Tests for testing
Scenario Outline: Test if scenario outlines work properly
And I point browser to "<url>"
Then there should be an element matching $("<wait_condition>") within 5 seconds
And I capture a screenshot
Examples:
| page | url | wait_condition |
| example | https://www.example.com | .foo |
| another example | https://www.example.com | #bar |
You'll get the following output.
(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c ../qa_tool/webtests/planterbox.cfg -c ../qa_tool/webtests/sauce-shawn.cfg -c ../qa_tool/webtests/law-prod.cfg winslow.tests.features:test.feature:0 -v
Test for Testing (winslow.tests.features:test.feature:0)
Scenario Outline: Test if scenario outlines work properly ... FAIL
Test for Testing (winslow.tests.features:test.feature:0)
Scenario Outline: Test if scenario outlines work properly <- {u'': u'', u'url': u'https://www.example.com', u'wait_condition': u'#bar', u'page': u'another example'} ... FAIL
======================================================================
FAIL: Test for Testing (winslow.tests.features:test.feature:0)
Tests for testing
----------------------------------------------------------------------
Scenario Outline: Test if scenario outlines work properly
And I point browser to "https://www.example.com"
Then there should be an element matching $(".foo") within 5 seconds
Traceback (most recent call last):
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 202, in run_scenario
step_fn(self, *step_arguments)
File "/home/shawn/forks/planterbox-webdriver/planterbox_webdriver/css_selector_steps.py", line 89, in wait_for_element_by_selector
test.assertTrue(elems)
AssertionError: [] is not true
======================================================================
FAIL: Test for Testing (winslow.tests.features:test.feature:0)
Tests for testing
----------------------------------------------------------------------
Scenario Outline: Test if scenario outlines work properly <- {u'': u'', u'url': u'https://www.example.com', u'wait_condition': u'#bar', u'page': u'another example'}
And I point browser to "https://www.example.com"
Then there should be an element matching $("#bar") within 5 seconds
Traceback (most recent call last):
File "/home/shawn/.virtualenvs/deus_lex/local/lib/python2.7/site-packages/planterbox/__init__.py", line 202, in run_scenario
step_fn(self, *step_arguments)
File "/home/shawn/forks/planterbox-webdriver/planterbox_webdriver/css_selector_steps.py", line 89, in wait_for_element_by_selector
test.assertTrue(elems)
AssertionError: [] is not true
----------------------------------------------------------------------
Ran 2 tests in 50.869s
FAILED (failures=2)
The second test includes the deatils of that particular scenario outline step <- {u'': u'', u'url': u'https://www.example.com', u'wait_condition': u'#bar', u'page': u'another example'}
but the first does not.
https://qa.tool.lexmachina.com/report/364270
The scenario outline looks malformed.
Feature
Walk And Shoot
Scenario
Scenario Outline: Winslow Help Center pages - {u'': u'', u'url': u'help/quick-tips', u'wait_condition': uspan:contains('Tip 1: Analyze Judge Behavior'), u'page': u'Help Quick Tips page'}
Step
Scenario Outline: Winslow Help Center pages <- {u'': u'', u'url': u'help/quick-tips', u'wait_condition': u"span:contains('Tip 1: Analyze Judge Behavior')", u'page': u'Help Quick Tips page'}
The uspan:contains('Tip 1: Analyze Judge Behavior')
should look like u'span:contains('Tip 1: Analyze Judge Behavior')
Full traceback
Scenario Outline: Winslow Help Center pages <- {u'': u'', u'url': u'help/quick-tips', u'wait_condition': u"span:contains('Tip 1: Analyze Judge Behavior')", u'page': u'Help Quick Tips page'}
Given I point browser to "help/quick-tips"
And I dismiss notifications
Then I take a screenshot
Traceback (most recent call last):
File "/home/lex/local/lib/python2.7/site-packages/planterbox/feature.py", line 215, in run_scenario
step_fn(self, *step_arguments)
File "/home/lex/src/deus_lex/qatool/qatool/tests/features/__init__.py", line 175, in record_visdiff_details
common_planterbox.record_visdiff_screenshot(test, _scenario_id)
File "/home/lex/src/deus_lex/deus_lib/deus_lib/common_planterbox.py", line 149, in record_visdiff_screenshot
screenshot = take_screenshot(test, time_of_shot)
File "/home/lex/src/deus_lex/deus_lib/deus_lib/common_planterbox.py", line 214, in take_screenshot
cloud_storage_screenshot,)
File "/home/lex/src/deus_lex/bias/bias/webscraping.py", line 75, in upload_to_gs_bucket
gs_key.set_contents_from_file(upload_file, rewind=True)
File "/home/lex/local/lib/python2.7/site-packages/boto/gs/key.py", line 580, in set_contents_from_file
self.send_file(fp, headers, cb, num_cb, size=size)
File "/home/lex/local/lib/python2.7/site-packages/boto/gs/key.py", line 313, in send_file
hash_algs=hash_algs)
File "/home/lex/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
query_args=query_args
File "/home/lex/local/lib/python2.7/site-packages/boto/s3/connection.py", line 665, in make_request
retry_handler=retry_handler
File "/home/lex/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/home/lex/local/lib/python2.7/site-packages/boto/connection.py", line 940, in _mexe
request.body, request.headers)
File "/home/lex/local/lib/python2.7/site-packages/boto/s3/key.py", line 884, in sender
response.status, response.reason, body)
File "/home/lex/local/lib/python2.7/site-packages/boto/exception.py", line 296, in __init__
super(StorageResponseError, self).__init__(status, reason, body)
File "/home/lex/local/lib/python2.7/site-packages/boto/exception.py", line 115, in __init__
h.parseString(self.body)
File "/home/lex/local/lib/python2.7/site-packages/boto/handler.py", line 60, in parseString
return self.parser.parse(StringIO(content))
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 210, in feed
self._parser.Parse(data, isFinal)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 1441: ordinal not in range(128)
And the original Scenario
https://github.com/LexMachinaInc/deus_lex/blob/production/qatool/qatool/tests/features/winslow_walk_and_shoot.feature#L411
Scenario Outline: Winslow Help Center pages
Given I point browser to "<url>"
And I dismiss notifications
Then I take a screenshot
Examples:
| page | url | wait_condition |
| Help Documentation page | help/documentation | span:contains('Data Coverage') |
| Help Quick Tips page | help/quick-tips | span:contains('Tip 1: Analyze Judge Behavior') |
| Help Video Tutorials page | help/video-tutorials | .caption:contains('Size Up Counsel') |
| Help Published Reports page | help/published-reports | .text-region:contains('PTAB Report 2015') |
| Help Quick Tip Analyze Judge Behavior page | help/quick-tips/analyze-judge-behavior | span:contains('Case Timing by Judge') |
| Help Video Tutorial Size Up Counsel page | help/video-tutorials/size-up-counsel | .scrollable-content-layout:contains('Accelerate with Legal Analytics: Size Up Counsel') |
Example output
(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c unittest-sauce-law-stage.cfg winslow.tests.features:auth.feature -v
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... FAIL
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Winslow Authentication Testing (winslow.tests.features:auth.feature)
Sign in as a user with an incorrect email ... ok
Example Feature summery;
Feature: Winslow Authentication Testing
Sign in as a user with an incorrect email
Sign in as a user with an incorrect password
Sign in as a registered user
Sign in as a user with an expired token
Sign in as a user who's account has been disabled
Sign in as a user who's trail has expired
Register as a user who is already singed in
Attempt to reset a password when non is set
Instead of showing the same first line of the Feature summery show the relevant Scenario name?
Running nose2 with planterbox-webdriver tests that include comments produces the following output:
(planterbox-webdriver)shawns-MacBook-Air:planterbox-webdriver shawn$ nose2
..............FE....F..F.
======================================================================
ERROR: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 242, in run_scenario
step_fn, step_arguments = self.match_step(step)
File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 192, in match_step
raise UnmatchedStepException()
UnmatchedStepException
======================================================================
FAIL: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Scenario: Combo boxes fail
Given I go to "basic_page"
Then I should not see option "Mercedes" in selector "car_choice"
Traceback (most recent call last):
File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 246, in run_scenario
step_fn(self, *step_arguments)
File "/Users/shawn/source/planterbox-webdriver/planterbox_webdriver/webdriver.py", line 414, in select_does_not_contain
test.assertIsNone(option_in_select(test.world.browser, id_, option))
AssertionError: <selenium.webdriver.remote.webelement.WebElement object at 0x1098d8850> is not None
======================================================================
FAIL: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Scenario: test labels
When I go to "basic_page"
And I click on label "Favorite Colors:"
Then element with id "fav_colors" should be focused
And element with id "bio_field" should not be focused
Traceback (most recent call last):
File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 246, in run_scenario
step_fn(self, *step_arguments)
File "/Users/shawn/source/planterbox-webdriver/planterbox_webdriver/webdriver.py", line 286, in element_not_focused
test.assertEqual(elem, focused)
AssertionError: <selenium.webdriver.remote.webelement.WebElement object at 0x1098d86d0> != <selenium.webdriver.remote.webelement.WebElement object at 0x1098f7ed0>
======================================================================
FAIL: I should see, I should not see (planterbox_webdriver.tests.test_webdriver:basics.feature)
# Forms stuff
# Requires known failure support
# Requires Multi-line step support
# alerts
#possible bug here?
# Requires known failure support
----------------------------------------------------------------------
Scenario: test date input
When I go to "basic_page"
And I fill in "dob" with "1900/01/01"
Traceback (most recent call last):
File "/Users/shawn/source/planterbox-webdriver/lib/python2.7/site-packages/planterbox/__init__.py", line 246, in run_scenario
step_fn(self, *step_arguments)
File "/Users/shawn/source/planterbox-webdriver/planterbox_webdriver/webdriver.py", line 235, in fill_in_textfield
test.assertTrue(field, 'Can not find a field named "%s"' % field_name)
AssertionError: Can not find a field named "dob"
----------------------------------------------------------------------
Ran 25 tests in 29.038s
FAILED (failures=3, errors=1)
Comments can be useful and I've used them sparingly in lettuce-webdriver tests. One useful example;
Scenario: 65. A Shared Page
# This will fail every two weeks and need to be regenerated in steps.py
...
A comment like this can be useful if someone besides myself takes over test maintenance and is looking for information to why that particular test has started failing.
In refactoring some cfg's I happened to move the position of
[planterbox-summary]
always-on = True
From the bottom of a config file closer to top.
All config values below it are not loaded.
Able to verify by running nose2 with -D
and self["some_expected_value"] not being defined.
Is this expected behavior?
Sometimes feature files get big.
When one scenario in a feature gets updated, it's a pain to manually count each test in the file to find it's ordinal number so you can run just that test.
@npilon suggests
"The scenario name must be unique in the file and allow it to be specified on the command line"
So I'm guessing this makes running a test like;
nose2 winslow.tests.features:walk_and_shoot.feature:0
look somthing like;
nose2 winslow.tests.features:walk_and_shoot.feature:"Winslow pages requiring user to be signed out"
or;
nose2 winslow.tests.features:walk_and_shoot.feature --scenario "Winslow pages requiring user to be signed out"
This would make it a bit easier to run a specific scenario in a feature.
Currently when you iterate through a scenario outline like:
Scenario Outline: Winslow pages requiring user to be signed out
Given I go to "<foo>"
Then there should be "<bar>"
Examples:
| foo | bar |
| example.com | Hi |
| google.com | how are you |
The hooks ie @hook('before', 'scenario')
only fire at the beginning/end of the entire Scenario Outline.
In my mind it seems that each 'run' through a scenario outline should be treated as a distinct scenario. Behavior that should be triggered before or after a scenario (as defined by the hooks) should be triggered for each outline example row.
The gherkin syntax isn't 100% clear on this https://github.com/cucumber/cucumber/wiki/Scenario-Outlines but the example at the top of the page that starts with "Copying and pasting scenarios to use different values quickly becomes tedious and repetitive:" leads some credibility to my assumptions.
Let me know if you need more details.
In addition to words like, 'And', 'Given', 'Then', 'But' is typically supported.
https://github.com/cucumber/cucumber/wiki/Given-When-Then#and-but
Example use in a Scenario
Scenario: A Landing Page without Client Matter renders
Given I sign in
Then I should see "Welcome Enterprise Test" within 5 seconds
But there should not be an element matching $(".fa-clock-o")
Fitst test uses, 'And', second, 'But';
(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c unittest-sauce-law-stage.cfg winslow.tests.features:landing_page.feature:0 -v
Landing Page Permutations (winslow.tests.features:landing_page.feature:0)
Test each permutation of the Landing Page; ... ok
----------------------------------------------------------------------
Ran 1 test in 21.207s
OK
(deus_lex) shawn@debian8 ~/source/deus_lex/winslow> nose2 -c unittest-sauce-law-stage.cfg winslow.tests.features:landing_page.feature:0 -v
Landing Page Permutations (winslow.tests.features:landing_page.feature:0)
Test each permutation of the Landing Page; ... ERROR
======================================================================
ERROR: Landing Page Permutations (winslow.tests.features:landing_page.feature:0)
Test each permutation of the Landing Page;
----------------------------------------------------------------------
Scenario: A Landing Page without Client Matter renders
Given I sign in
Then I should see "Welcome Enterprise Test" within 5 seconds
But there should not be an element matching $(".fa-clock-o")
Traceback (most recent call last):
File "/home/shawn/source/planterbox/planterbox/__init__.py", line 196, in run_scenario
step_fn, step_arguments = self.match_step(step)
File "/home/shawn/source/planterbox/planterbox/__init__.py", line 136, in match_step
raise UnmatchedStepException(step)
UnmatchedStepException: But there should not be an element matching $(".fa-clock-o")
----------------------------------------------------------------------
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.