GTXiLib does not catch errors on XCUITests about gtxilib HOT 16 CLOSED

Hey @niksawtschuk !

When I changed override func setUp() to override class func setUp() test start to fail and give me

One or more elements FAILED the accessibility checks: <UIButton: 0x7fde7d11eb00; frame = (16 519.667; 382 31); opaque = NO; layer = <CALayer: 0x600003899020>> + Check "Element has Minimum Tappable Area" failed, Suggest increasing element's frame height to at least 44 for a suggested tappable area of at least 44X44 <UIButton: 0x7fde7d121740; frame = (161.667 620.667; 91 28.6667); opaque = NO; layer = <CALayer: 0x600003899420>> + Check "Element has Minimum Tappable Area" failed, Suggest increasing element's frame height to at least 44 for a suggested tappable area of at least 44X44

Thank you for your help! By the way is this error log is expected? I mean should not we see UIButton name? How can we know which UIButton is failing?

from gtxilib.

Just as an FYI, GTXiLib doesn't work on XCUITests (to the best of my knowledge). It has to be an XCTest-based case (EarlGrey is your best bet here).

Apart from that, I can only see a few things here:

1. GTXiLib needs to be setup and installed once per test suite (do this in the class setUp function.
2. What does testCase() do? Try running this on a simple case that you know will fail. I can easily trigger a failure by running EarlGrey on a view with a UIButton that is too small when using .allGTXChecks().

from gtxilib.

Our UITests inherits from XCTestCase. Should not it work with XCTestCase?

from gtxilib.

testCase() function taps a XCUIElement that opens a page after that it again taps another XCUIElement and asserts page is loaded correcctly. Both of these elements do not have accessibility identifiers.

from gtxilib.

Our UITests inherits from XCTestCase. Should not it work with XCTestCase?

What I meant to say is that your tests have to be only using XCTest. XCUITest is built on top of XCTest, yes, but iirc the tests run in a different process than your app, which does not allow GTXi to hook into its execution.

testCase() function taps a XCUIElement that opens a page after that it again taps another XCUIElement and asserts page is loaded correcctly. Both of these elements do not have accessibility identifiers.

It sounds like you are using XCUITest. If you want to use GTXi, I would recommend using EarlGrey. I ran a quick test on an XCUITest suite and indeed GTXi does not work.

maybe @j-sid can give some insight here, but I don't think XCUITest will work nicely with this any time soon.

from gtxilib.

@niksawtschuk your assessment is correct GTXiLib only works with XCTest at the moment i.e. it works with unit tests but not UI tests (XCUITest)

from gtxilib.

@selin194 One point though GTXiLib currently do not have checks for verifying accessibility identifiers, it can only has the checks listed from this line: https://github.com/google/GTXiLib/blob/master/Classes/GTXChecksCollection.h#L64

One way of getting this to work with XCUITest is to build a "testable" app target that has GTX as a dependency and trigger checks on it (from the app not the test) I may have a work around for this, I'll update this bug when ready.

from gtxilib.

I understand. Yes, we are using XCUITests. I will try EarlGrey, then. By the way, do you have any example project that I can check that you used with EarlGrey? Thank you for your responses!

from gtxilib.

Hey all again! As we talked I start to use EarlGrey for our UI tests and I added new test suite like below:
`
import EarlGrey
import GTXiLib
import XCTest

class AccessibilityTests: XCTestCase {

override func setUp() {
    let checksToBeInstalled: [GTXChecking] = GTXChecksCollection.allGTXChecks()
    let tmp = GTXTestSuite.init(allTestsIn: AccessibilityUnitTests.self)
    GTXiLib.install(on: tmp ?? GTXTestSuite(), checks: checksToBeInstalled, elementBlacklists: [])

    // Put setup code here. This method is called before the invocation of each test method in the class.
}

override func tearDown() {
    // Put teardown code here. This method is called after the invocation of each test method in the class.
}

func testPerformanceExample() {
    let matcher = grey_accessibilityID("signInWithEmailButton")
    let action = grey_typeText("Sample Swift Test")
    let assertionMatcher = grey_text("Sample Swift Test")
    EarlGrey.selectElement(with: matcher)
        .perform(action)
}

}

And I insert a breakpoint to the file 'GTXiLibCore.m' and to the function:
+ (BOOL)_checkElement:(id)element { NSError *error; BOOL success = [gToolkit checkElement:element error:&error]; if (error) { self.failureHandler(error); } return success; }

And I never see the program stops here.

By the way, I also insert a breakpoint to the function:
(void)installOnTestSuite:(GTXTestSuite *)suite checks:(NSArray<id<GTXChecking>> *)checks elementBlacklists:(NSArray<id<GTXBlacklisting>> *)blacklists

And I see program hits to this function. What am I doing wrong? Thanks in advance! @niksawtschuk @j-sid !

from gtxilib.

I changed the size of signInWithEmailButton too small like in here:

Even though GTiXiLib does not catch any error, do you have any idea? @niksawtschuk @j-sid, thank you!

from gtxilib.

@selin194

What do you see on the simulator screen when you run this test? Do you see the expected view? Usually when I've written UI tests like this in the past, each test case will setup the app by setting the root view controller to be the controller it wants to test.

Also, please make sure you are setting up GTXi and installing it on the test suite only once (do this in the static setUp method of your suite), it appears you are doing it in the normal setUp method, which will run that setup code before every test case in your suite.

Besides the fact that the example test you've given us is in the default testPerformanceExample (given by Xcode to give you a place to write a test that measures how long something takes), try making sure your test actually does something. The case you've shown here appears to search for a button, then tries to type text into that button. Make sure you can select an element that exists, then run some assertion on it.

from gtxilib.

Hi All,

I recently tried Earl grey as discussed above.

1.So,I installed Pod Earl Grey in my project.There after I can successfully add the line import EarlGrey in to my UI Tests.But as similar to code above I am not able to write the code.

EarlGrey.selectElement(with: matcher)
.perform(action)

but, I am seeing EarlGreyImpl instead of EarlGrey.selectElement

2.what is the difference between EarlGrey and EarlgreyTests.

As is see the versions of it EarlGrey (1.16.0) and EarlGreyTest (2.2.1)

Will these versions make any difference.

3.In the above code I see there are GTX checks that are enabled is it required or just need to write the EarlGrey tests in test stubs.

from gtxilib.

I used EarlGrey 2.0 and I am facing same issue what @selin194 facing and here I am tapping the button and asserting it and still not able to see failure although I set by button frame to 5*5

can any one help me here @niksawtschuk @j-sid @selin194

from gtxilib.

The current version of GTX is only compatible with XCTest and XCTest based frameworks (aka EarlGrey 1.x) our next release will bring XCUITest support to GTX. Expect this to land by March.

from gtxilib.

Thanks @j-sid
I tried using EarlGrey 2.0 for XCUI tests and I see no accessibility errors.( I tapped button, but still no accessibility errors)
Unit Tests are perfect and I can able to see accessibility Errors.

from gtxilib.

Any Updated on whether GTXlib works for UI automation on its latest version. @j-sid

from gtxilib.