Default to not running sauce labs tests on Travis/CI? about airtap HOT 10 CLOSED

Agree that we should handle this.

How about skipping tests if process.env.TRAVIS_SECURE_ENV_VARS !== 'true'?

From https://docs.travis-ci.com/user/environment-variables/#Default-Environment-Variables:

TRAVIS_SECURE_ENV_VARS:

• set to true if there are any encrypted environment variables.
• set to false if no encrypted environment variables are available.

from airtap.

Sounds good to me! Let's make sure to print a message so the user knows what's happening if they read the logs.

Note, I don't think you want to skip tests if process.env.TRAVIS_SECURE_ENV_VARS is undefined which will happen when e.g. running locally, so how about this test:

var shouldSkip = process.env.TRAVIS_SECURE_ENV_VARS === 'false'

from airtap.

Sounds good to me! Let's make sure to print a message so the user knows what's happening if they read the logs.

👍

Note, I don't think you want to skip tests if process.env.TRAVIS_SECURE_ENV_VARS is undefined

Ah right. SGTM.

from airtap.

So we shouldn't run test/integration/tape-sauce.js on PRs but the rest of the tests should be fine? We need to make a seam somewhere then so we can split up the tests. Maybe just in test/index.js, e.g.:

require('./unit/aggregate-browsers')
require('./unit/browserify-builder')
require('./unit/capabilities')
require('./unit/flatten-browser')
require('./integration/tape-electron')
require('./integration/tape-phantom')

if (process.env.TRAVIS_SECURE_ENV_VARS !== 'false') {
  require('./integration/tape-sauce')
}

from airtap.

@feross Let me know if you're happy with this and I'll get it done.

from airtap.

After this maybe release 0.0.4?

from airtap.

@ralphtheninja This issue is about adding a feature that will allow users of airtap to skip running tests entirely on PRs, since those lack credentials.

So, I could add airtap to a project like buffer without needing to write code to detect if we're in a Travis PR. We want to skip airtap in PRs, because the credentials are missing. See this line: https://github.com/feross/buffer/blob/c0c91b75e43d5ef82c246e4d8f3746b5e779aded/bin/test.js#L7-L8

If we need the same thing for our own tests, it would be nice to just implement this for everyone, either via a --skip-missing-credentials or --skip-pr flag, or just defaulting to this behavior if that's what folks think makes the most sense. Then we can just use the flag in our own airtap test suite :)

from airtap.

@ralphtheninja This issue is about adding a feature that will allow users of airtap to skip running tests entirely on PRs, since those lack credentials.

Oh right. Gotcha!

from airtap.

So we put that check in bin/airtap.js, log something suitable and exit 0.

from airtap.

Yes, I think so.

from airtap.