Git Product home page Git Product logo

vscode-test's Introduction

vscode-test

Test Status Badge

This module helps you test VS Code extensions. Note that new extensions may want to use the VS Code Test CLI, which leverages this module, for a richer editing and execution experience.

Supported:

  • Node >= 16.x
  • Windows >= Windows Server 2012+ / Win10+ (anything with Powershell >= 5.0)
  • macOS
  • Linux

Usage

See ./sample for a runnable sample, with Azure DevOps Pipelines and Github ACtions configuration.

import { runTests, runVSCodeCommand, downloadAndUnzipVSCode } from '@vscode/test-electron';

async function go() {
	try {
		const extensionDevelopmentPath = path.resolve(__dirname, '../../../');
		const extensionTestsPath = path.resolve(__dirname, './suite');

		/**
		 * Basic usage
		 */
		await runTests({
			extensionDevelopmentPath,
			extensionTestsPath,
		});

		const extensionTestsPath2 = path.resolve(__dirname, './suite2');
		const testWorkspace = path.resolve(__dirname, '../../../test-fixtures/fixture1');

		/**
		 * Running another test suite on a specific workspace
		 */
		await runTests({
			extensionDevelopmentPath,
			extensionTestsPath: extensionTestsPath2,
			launchArgs: [testWorkspace],
		});

		/**
		 * Use 1.36.1 release for testing
		 */
		await runTests({
			version: '1.36.1',
			extensionDevelopmentPath,
			extensionTestsPath,
			launchArgs: [testWorkspace],
		});

		/**
		 * Use Insiders release for testing
		 */
		await runTests({
			version: 'insiders',
			extensionDevelopmentPath,
			extensionTestsPath,
			launchArgs: [testWorkspace],
		});

		/**
		 * Noop, since 1.36.1 already downloaded to .vscode-test/vscode-1.36.1
		 */
		await downloadAndUnzipVSCode('1.36.1');

		/**
		 * Manually download VS Code 1.35.0 release for testing.
		 */
		const vscodeExecutablePath = await downloadAndUnzipVSCode('1.35.0');
		await runTests({
			vscodeExecutablePath,
			extensionDevelopmentPath,
			extensionTestsPath,
			launchArgs: [testWorkspace],
		});

		/**
		 * Install Python extension
		 */
		await runVSCodeCommand(['--install-extension', 'ms-python.python'], { version: '1.35.0' });

		/**
		 * - Add additional launch flags for VS Code
		 * - Pass custom environment variables to test runner
		 */
		await runTests({
			vscodeExecutablePath,
			extensionDevelopmentPath,
			extensionTestsPath,
			launchArgs: [
				testWorkspace,
				// This disables all extensions except the one being tested
				'--disable-extensions',
			],
			// Custom environment variables for extension test script
			extensionTestsEnv: { foo: 'bar' },
		});

		/**
		 * Use win64 instead of win32 for testing Windows
		 */
		if (process.platform === 'win32') {
			await runTests({
				extensionDevelopmentPath,
				extensionTestsPath,
				version: '1.40.0',
				platform: 'win32-x64-archive',
			});
		}
	} catch (err) {
		console.error('Failed to run tests');
		process.exit(1);
	}
}

go();

Development

  • yarn install
  • Make necessary changes in lib
  • yarn compile (or yarn watch)
  • In sample, run yarn install, yarn compile and yarn test to make sure integration test can run successfully

License

MIT

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

vscode-test's People

Contributors

amazingguni avatar apupier avatar bpasero avatar connor4312 avatar dependabot[bot] avatar eamodio avatar ext avatar fpoli avatar gabrielbb avatar gkalpak avatar jcreedcmu avatar joaomoreno avatar microsoft-github-policy-service[bot] avatar octref avatar pushkine avatar qiu8310 avatar roblourens avatar sieukrem avatar tal-sapan avatar tyriar avatar vfonic avatar xisui-msft avatar yoshinorin avatar zardoy avatar zimmski avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vscode-test's Issues

unable to import vscode in test using jest

Hi, I'm trying to run some integration tests for my vscode extension using jest. Unit tests work fine since I'm able to mock vscode however, when I'm trying to run an integration test I'm getting Cannot find module 'vscode' in the files when i'm using vscode api (import * as vscode from 'vscode')

jest config looks like this:

module.exports = {
  moduleFileExtensions: ['js'],
  testMatch: ['<rootDir>/out/test/**/*.int.test.js'],
  verbose: true,
};

runTest.ts

import * as path from 'path';
import { runTests } from 'vscode-test';

async function main() {
	try {
		// The folder containing the Extension Manifest package.json
		// Passed to `--extensionDevelopmentPath`
		const extensionDevelopmentPath = path.resolve(__dirname, '../../');

		// The path to the extension test script
		// Passed to --extensionTestsPath
		const extensionTestsPath = path.resolve(__dirname, './suite/index');

		// Download VS Code, unzip it and run the integration test
		console.log(`running tests... ext path: ${extensionTestsPath}`);
		await runTests({
			extensionDevelopmentPath,
			extensionTestsPath,
			launchArgs: ['--disable-extensions']
		});
	} catch (err) {
		console.error('Failed to run tests', err);
		process.exit(1);
	}
}

main();

index.ts

import * as path from 'path';
import { runCLI } from 'jest-cli';
import * as vscode from 'vscode';

export function run(): Promise<void> {

  return new Promise((resolve, reject) => {
    const projectRootPath = path.join(__dirname, '../../../');
    const config = path.join(projectRootPath, 'jest.e2e.config.js');

    vscode.window.showInformationMessage('Run suite');

    runCLI({ config } as any, [projectRootPath])
      .then(jestCliCallResult => {
        console.log(`complete: ${JSON.stringify(jestCliCallResult)}`);
        console.log('print results');
        jestCliCallResult.results.testResults
          .forEach(testResult => {
            testResult.testResults
              .filter(assertionResult => assertionResult.status === 'passed')
              .forEach(({ ancestorTitles, title, status }) => {
                console.info(`  ● ${ancestorTitles} β€Ί ${title} (${status})`);
              });
          });

        console.log('check results');
        jestCliCallResult.results.testResults
          .forEach(testResult => {
            if (testResult.failureMessage) {
              console.error(testResult.failureMessage);
            }
          });
        resolve();
      })
      .catch(errorCaughtByJestRunner => {
        console.error('error in test runner', errorCaughtByJestRunner);
        reject(errorCaughtByJestRunner);
      });
  });
}

I installed both vscode-test and @types/vscode as dev dependencies. I'm not sure why my tests are unable to find the 'vscode' dependency. Vscode is the only dependency that has this issue, other modules work fine. Can anyone help?

Thanks!

Powershell invocation is missing arguments

The powershell invocation at;

			cp.spawnSync('powershell.exe', [
				'-Command',
				`Expand-Archive -Path ${vscodeArchivePath} -DestinationPath ${extractDir}`

is missing some arguments which you should use to avoid some edge cases:

-NoProfile - Stops loading custom users profile scripts. This also has a benefit of speeding up the loading process start time

-NonInteractive - Suppress any UI questions. You have no UI at this point

-NoLogo - May as well suppress the logo while you're at it

  • Probably don't need to set the execution policy (https://docs.microsoft.com/en-us/powershell/scripting/components/console/powershell.exe-command-line-help?view=powershell-5.0)

  • Also note that exit codes from powershell.exe, if you decide to use them, could be different from what you expect as ErrorAction has not been set

  • Both ${vscodeArchivePath} and ${extractDir} could contain spaces in the path so these should be wrapped in double quotes

  • Expand-Archive is ONLY available in PowerShell 5.0 and above. Moving from the old gulp method in vscode-extension-vscode to this could be considered a breaking change as Server 2012 and Windows 7 users will not have Expand-Archive by default. At the VERY least this should be a documentation note for Extension Authors.

Ref - https://github.com/Microsoft/vscode-test/blob/master/lib/download.ts#L90-L92

Can't run extension test when offline

  • Be on a bad connector connection, try to run extension tests
  • Fails with Error: getaddrinfo ENOTFOUND update.code.visualstudio.com update.code.visualstudio.com:443
  • There is a copy of vscode downloaded, so this is checking for an update
  • vscode-test should ignore any network error as long as it can run

Overriding $HOME in tests...

hi again πŸ‘‹

i'm currently overriding the HOME environment variable in order to isolate side effects from my end-to-end tests and provide repeatable fixtures / environment configuration:

const extensionDevelopmentPath = getPackagesDir();
const extensionTestsPath = path.join(getExtensionDir(), 'test');
const testProjPath = getTestProjectDir();
const extensionTestsEnv = {
	HOME: getOverriddenHomeDir()
};

try {
	await runTests({
		extensionDevelopmentPath,
		extensionTestsPath,
		extensionTestsEnv,
		launchArgs: [
			testProjPath
		]
	});
} catch (error){
	return logErrorAndExit(error);
}

seems to work ok but occasionally i get a cryptic error like:

Error: ENOENT: no such file or directory, open '/path/to/my/extension/test/__root__/home/Library/Application Support/Code/storage.json'

...which seems to happen when tests fail (but not always).

is there a recommended way to set a custom $HOME path?

thanks in advance πŸ™

Something appears to mess with PATH on Travis Linux when running tests

I pushed a repo to Travis that is just the TypeScript extension sample with a travis file that echos the PATH in the Travis script, the npm test script and then also inside the sample test. The values are different. Significantly, the path to node has been removed by the time we get into the tests. In my real project, this means my tests are failing because they use the DebugClient and pass node as the executable.

Here's the code:
https://github.com/DanTup/vscode_extension_path_test

Here's the Travis log:
https://travis-ci.org/DanTup/vscode_extension_path_test/jobs/362281254

Here's the relevant parts:

PATH as output in the travis file:

./node_modules/.bin:/home/travis/bin:/home/travis/.local/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.11.1/bin:/home/travis/.kiex/elixirs/elixir-1.4.5/bin:/home/travis/.kiex/bin:/home/travis/.rvm/gems/ruby-2.4.1/bin:/home/travis/.rvm/gems/ruby-2.4.1@global/bin:/home/travis/.rvm/rubies/ruby-2.4.1/bin:/home/travis/gopath/bin:/usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/home/travis/.gimme/versions/go1.7.4.linux.amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin

PATH as output by prefixing the test script in package.json with echo $PATH:

/home/travis/.nvm/versions/node/v8.11.1/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/travis/build/DanTup/vscode_extension_path_test/node_modules/.bin:./node_modules/.bin:/home/travis/bin:/home/travis/.local/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.11.1/bin:/home/travis/.kiex/elixirs/elixir-1.4.5/bin:/home/travis/.kiex/bin:/home/travis/.rvm/gems/ruby-2.4.1/bin:/home/travis/.rvm/gems/ruby-2.4.1@global/bin:/home/travis/.rvm/rubies/ruby-2.4.1/bin:/home/travis/gopath/bin:/usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/home/travis/.gimme/versions/go1.7.4.linux.amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin

PATH as output in tests:

/home/travis/.local/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.kiex/elixirs/elixir-1.4.5/bin:/home/travis/.kiex/bin:/home/travis/.phpenv/shims:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.7.4.linux.amd64/bin:/usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/home/travis/.gimme/versions/go1.7.4.linux.amd64/bin:/usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/home/travis/.nvm/versions/node/v8.11.1/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/travis/build/DanTup/vscode_extension_path_test/node_modules/.bin:./node_modules/.bin:/home/travis/bin:/home/travis/.local/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/.kiex/elixirs/elixir-1.4.5/bin:/home/travis/.kiex/bin:/home/travis/.rvm/gems/ruby-2.4.1/bin:/home/travis/.rvm/gems/ruby-2.4.1@global/bin:/home/travis/.rvm/rubies/ruby-2.4.1/bin:/home/travis/gopath/bin:/usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/home/travis/.gimme/versions/go1.7.4.linux.amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin

Note that /home/travis/.nvm/versions/node/v8.11.1/bin exists in the first and second, but not the last one (the one from inside the tests).

Support running tests using a version of VSCode that is already running

The largest disadvantage to switching to using this library vs the traditional debug launch configuration is the loss of the ability to run tests in the same version of VSCode that you are launching the debug configuration from. The Extension Testing page suggests to just use Insiders for testing extensions, which is fine if you only ever want to test in Stable, but I would personally like to run suites for both. Now the only way to do that is to close VSCode and run the tasks from the terminal.

Downloader should make sure a platform correct version exists

Sorry for the verbose title, anything more succinct is escaping me.

In lib/download.ts's downloadAndUnzipVSCode it runs a check to see if the downloadedPath exists and if it does it just console logs its findings and resolves the promise with the path. This seems to assume that the version of VSCode that exists already matches the platform that is currently running the command, this isn't always true.

I encountered this issue earlier when first running a test suite on Windows which downloaded a stable version of VSCode for Windows. Running the same suite in a Linux docker container, which runs through the same vscode-test\lib\download.ts code returns the path of the previously downloaded VSCode for Windows which causes the test suite to then fail because the downloader doesn't go deeper into the file structure to make sure a platform appropriate version of VSCode exists.

When the test suite is ran in the Linux container, the downloader sees .vscode-test/vscode-1.38.1 and returns the path, if it were to venture into this folder it would find Code.exe instead of the /VSCode-linux-x64 folder that would be needed. I went ahead and opened an issue on the repo whose test suite I encountered this on but I felt like it was potentially an upstream fix.

Perhaps I'm being naive in assuming if process.platform === 'linux' then the version of VSCode inside of .vscode-test/vscode-n.nn.n/* should match that platform?

Setting workspace is not working

I runTests.ts i have code taken from instruction:

// The folder containing the Extension Manifest package.json
		// Passed to `--extensionDevelopmentPath`
		const extensionDevelopmentPath = path.resolve(__dirname, '../../');

		// The path to test runner
		// Passed to --extensionTestsPath
		const extensionTestsPath = path.resolve(__dirname, './suite/index');
		const testWorkspace = path.resolve(__dirname, '../../test-fixtures/fixture1');

		// Download VS Code, unzip it and run the integration test
		await runTests({ 
			extensionDevelopmentPath,
			extensionTestsPath,
			launchArgs: [testWorkspace] 
		});

In the root of the project i have created folder test-fixtures/fixture1 with a file called foo.js. But when i run tests:

console.log(vscode.workspace.name);

returns undefined, hence, the workspace is not set up.
Expected result:

A workspace is set up

Environment variables and PATH not passing through in 1.2.3

Hello! I'm trying to test my VS Code extension which starts a language server by executing a command on the PATH, but none of the system's environment variables seem to be passing through to the test runner. This is causing my tests to fail with an ENOENT error as the executable is not able to be located on the PATH.

Despite passing in the entirety of process.env to runTests(), no environment variables are available in the test runner script specified by extensionTestsPath:

await runTests({
  extensionDevelopmentPath,
  extensionTestsPath,
  extensionTestsEnv: process.env,
  launchArgs: [extensionWorkspacePath, "--disable-extensions"]
});

In the extensionTestsPath file:

export function run(): Promise<void> {
	console.log(process.env)  // prints nothing
}

#20

Github CI - Tests are not running

At our project xlayers we make use of Github CI Actions. When using the vscode-test package for testing purposes the test are not running. Locally I don't have any problems running the test but as the error is not describing an error.

Could you help me out on this topic, here is the Github Action Visisble

If you need any more information let me know.

Test process exits with code 'null'

[Not always] Test process exits with exit code null. Tried to print the thrown error object. But it printed 'Failed',

Exit code:   null
Failed to run tests

All logs look quite normal. Just that it stopped in the middle.

I couldn't reproduce this issue. But it occurs sporadically only in Jenkins CI (unix-like nodes). Is there any way to troubleshoot the signal which triggered this? Or has anyone already faced this issue? Note: I am running the tests using xvfb-run

Sometimes, an additional log is also printed:

Failed to generate minidump
Exit code:   null
Failed to run tests

VSCode-Test appears to be failing on Travis CI linux tests

From microsoft/vscode#70857


Okay...so some initial digging.

  1. The https://github.com/Microsoft/vscode-extension-vscode repo which holds that npm module has changed it's tagging system from 1.1.30 to v.1.1.32 which may be coincidental.

The release dates for vscode on npmjs shows that 1.1.31 and 1.1.32 shortly thereafter were only a day ago so that explains why things have suddenly gone bad. 1.1.30 was a month ago.

A quick look at the diff between the two....
microsoft/vscode-extension-vscode@1.1.30...v1.1.32

Shows some major work on the bin/test file shows that the functionality seems to have moved to a vscode-test module (https://www.npmjs.com/package/vscode-test) which is VERY new (7 days ago)

Interestingly is this line: microsoft/vscode-extension-vscode@1.1.30...v1.1.32#diff-f900621cc6a1719b5473baf0191f297bR12

It says that the testing should be in the stable folder which clearly doesn't exist in my example above. It did exist on my Ubuntu 18.04 desktop though

Given the following log from a job

Downloading VS Code into "/home/travis/build/lingua-pupuli/puppet-vscode/.vscode-test/stable
Downloading VS Code 1.32.3
Downloaded VS Code 1.32.3

My hunch is that unzipVSCode (https://github.com/Microsoft/vscode-test/blob/master/lib/download.ts#L127) is failing silently.'

Okay so there is no error trapping in the unzipVSCode method

https://github.com/Microsoft/vscode-test/blob/master/lib/download.ts#L88-L99

Given this is using spawnSync it may be worth trapping the exitcode, stdout and stderr and throw errors if the exit code is not zero.

Also may be worth raise an error the extractDir doesn't exist after an extract.

[email protected] vscodeExecutablePath issue

To solve issue #29 , you make it point it to VSCode CLI. Now it's ok to install extensions but fail to run tests.

// Here the path should point to Electron, otherwise it won't run the test. 
 await runTests({ vscodeExecutablePath, extensionDevelopmentPath, extensionTestsPath });

throw error `Failed` in GitHub actions

workflow file:Β https://github.com/fi3ework/vscode-antd-rush/blob/master/.github/workflows/test.yml

a failed action:Β https://github.com/fi3ework/vscode-antd-rush/runs/599819681?check_suite_focus=true

I use vscode-test to test my VSCode extension. But recently it directly throws Fail when initializing. The error is throw from hereΒ https://github.com/microsoft/vscode-test/blob/master/lib/runTest.ts#L134, but I don't know why it throw. I can pass the test on my Macbook Pro(masOSΒ 10.14.6, nodejs v10.16.0). I can'tΒ  reproduce it on my own computer no matter how. πŸ€¦β€β™‚οΈ

Please help me or give a hint where I might missed or wrong. Thank you very much. πŸ™πŸ»

Here's the failed screenshot πŸ‘‡

image

and workflow file (pasted for better preview)

name: Test

on: [push]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [macos-latest, windows-latest]
        node: [10, 12]
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - name: yarn - install & compile
        run: |
          yarn
          yarn run compile
          yarn run compile-test
      - name: Run headless tests
        uses: GabrielBB/[email protected]
        with:
          run: npm run test

Also asked in community forum

Not getting latest insiders version, if older insiders version downloaded in the past

If version is 'insiders' and an insiders version has been downloaded before, then the vscode-insiders directory will be present and downloading will be skipped:

if (fs.existsSync(path.resolve(vscodeTestDir, `vscode-${version}`))) {

This means that insiders will never be updated once you have downloaded it the first time, unless you go and manually remove the download directory.

Code Coverage for vscode language extension ?

I have a test suite on language client, that calls language server, and I need to have code coverage of both after running this test suite

I tried (almost)everything ... and succeeded to do nothing

https://github.com/nvuillam/vscode-groovy-lint/blob/master/client/src/test/runTest.ts

Is there somewhere a working example using vscode-test ?
All that I found were very complex examples not using vscode-test, and i'd like to remain close to vscode standard

Thanks :)

Allow to configure 32bit vs 64bit for downloaded version in tests

node ./node_modules/vscode/bin/test --verbose

### VS Code Extension Test Run ###
Current working directory: C:\projects\blockchain-vscode-extension\client
Downloading VS Code into "C:\projects\blockchain-vscode-extension\client\.vscode-test\stable" from: https://vscode-update.azurewebsites.net/1.28.0/win32-archive/stable
Running extension tests: C:\projects\blockchain-vscode-extension\client\.vscode-test\stable\Code.exe C:\projects\blockchain-vscode-extension\client\out\test --extensionDevelopmentPath=C:\projects\blockchain-vscode-extension\client --extensionTestsPath=C:\projects\blockchain-vscode-extension\client\out\test

How do we configure it to pull the 64-bit version?

Current working directory inconsistent across platforms

When running node node_modules/VSCode/bin/test with reporter options to save a file for subsequent upload (see #151 for context), I'm finding that the current working directory is inconsistent. In my src/test/index.ts I have something like this:

const opts = {
  reporter: 'mocha-junit-reporter',
  reporterOptions: {
    mochaFile: './test-results.xml'
  },
  ui: 'tdd',
  useColors: true
};

testRunner.configure(opts).

On Windows, this file ends up in .vscode-test\stable\test-results.xml. On Linux (and I presume MacOS, since Azure DevOps is reporting during test results upload that file cannot be found), the file gets saved to my project root ./test-results.xml. Given I am testing both stable and insiders versions of Code, I prefer the Windows approach but being consistent in any fashion would make CI/CD workflows much easier to create and maintain.

VSCode exits without error before Mocha finishes testing

I have already asked about my problem on Reddit and Stack Overflow.

I am now starting to believe that this caused by a bug in either vscode or vscode-test, because if Mocha was the problem, it would return non-zero return code. Instead, task exits "successfully" in the middle of the test.

What is even more strange is that the task exists at random moments: sometimes the test will complete as it should. It completes more often on my home PC than on my notebook. I guess that this is because my home computer is faster, and the task is able to finish before VSCode exits.

suite("Extension Tests", () => {

    const fixtureFolderLocation = "../fixture/";

    test("Wrap in parentheses", async (done) => {
        const uri = vscode.Uri.file(
            path.join(__dirname, fixtureFolderLocation, "fixture2.html"),
        );

        const document = await vscode.workspace.openTextDocument(uri);

        const editor = await vscode.window.showTextDocument(document);

        editor.selection = new vscode.Selection(new vscode.Position(0, 0), new vscode.Position(0, 4));

        await vscode.commands.executeCommand("nkatex.wrapInParentheses");

        const text = document.getText(new vscode.Range(new vscode.Position(0, 0), new vscode.Position(0, 21)));
        assert.equal(text, "\\left ( word\\right ) ");
        done();
    });

});

When I set a breakpoint somewhere at the beginning of the test, for example at const document = ..., and when I press the "Step Over" button, the test exits immediately.

It seems that there is some asynchronous timeout that closes VSCode during tests. Debug Console just shows the suite name:
image

Run VSCode Integration Tests through typescript files

We currently use Mocha to run all of our tests through the out/ folder within our repository. We want to switch to using ts-node in conjunction with mocha so that we can run the tests and collect coverage information (with NYC) through the typescript files, rather than compiled javscript. This works fine for unit tests, but it doesn't seem to work with vscode-integration tests. I believe due to the way the extension development host is launched, it is looking specifically for a javascript extension so can't interpret our typescript files.
Is it possible to change that so that we are able to also run vscode-integration tests through the typescript files? Thanks!

Support for Opening / Closing Workspace Within Tests?...

hey there πŸ‘‹

thanks for the helpful tool πŸ™

i'm using vscode-test and its supporting bits to drive end-to-end style tests for my extension. one thing i'd like to do is: open a folder / workspace, open a file, scan the file for "problems" and assert against what i find.

the wrinkle i'm running into is that opening a folder / workspace causes my tests to re-run. specifically:

await executeCommand('vscode.openFolder', project.folder.uri);

...when run causes a new window to launch (fine) and my test suite to re-run from the start (not so fine).

i realize i can use the launchArgs option to open a project folder / workspace upon launch but i want to target a number of project fixtures where each represents a different use-case. naively, i thought i could just open each as part of my setup / teardown stages but that doesn't appear to be a viable approach.

can you recommend a work-around or alternative approach?

How to run vscode-test runScript from launch.json?

How can we have an entirely synchronized approach so that I'm running tests in the same way using launch.json?

Following up from my comment at #24 (comment) where I said:

@octref confused by this - vscode is deprecated per https://www.npmjs.com/package/vscode
https://code.visualstudio.com/api/working-with-extensions/testing-extension#migrating-from-vscode does not discuss how to use vscode-test to run from launch.json, does it?

What's the best way to synchronize the approach so that I can use the same script for both?

I pulled down this repo and tried to run the sample Launch Test and I saw this error:
image

Somewhat motivated by prettier/prettier-vscode#940 where there is both the launch.json and yarn test ways to run the tests but the launch.json one doesn't work.

Integration tests improvement

Not sure if it is possible but, would it be possible to have a testing API that lets test click and enter text inside the vscode window? This would allow me to automate all of the checks i have to manually do. Something like puppeteer.

Allow passing environment variables through

Apologies if this exists, I can't find a way. Today when I run my tests I set a bunch of env variables in the CI script for things like log and coverage paths that I'll upload later:

env.DC_TEST_LOGS = path.join(cwd, ".dart_code_test_logs", `${testRunName}_${dartFriendlyName}_${codeFriendlyName}`);
env.COVERAGE_OUTPUT = path.join(cwd, ".nyc_output", `${testRunName}_${dartFriendlyName}_${codeFriendlyName}.json`);
env.TEST_XML_OUTPUT = path.join(cwd, ".test_results", `${testRunName}_${dartFriendlyName}_${codeFriendlyName}.xml`);
env.TEST_CSV_SUMMARY = path.join(cwd, ".test_results", `${dartFriendlyName}_${codeFriendlyName}_summary.csv`);

I can't see any obvious way to do this, if it's not supported would you consider it?

Test runs now failing with "Error: Invalid version *"

My tests on Travis have started failing with this:

Current working directory: /Users/travis/build/Dart-Code/Dart-Code
Failed to run test with error:
Error: Invalid version *
    at /Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:179:31
    at step (/Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:36:23)
    at Object.next (/Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:17:53)
    at fulfilled (/Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:8:58)
    at process._tickCallback (internal/process/next_tick.js:68:7)
      Ended after: 0.511s

I have code that sets the version like this:

const codeVersion = process.env.RUN_CODE_VERSION === "DEV" ? "insiders" : "*";

I think * use to be valid to mean "the latest stable release" but I think in the latest change that has become broken. Looking at the source, maybe not setting a version at all will fix it, though I wonder if for compatibility it should remain supported (I think checking for * in this condition may fix it).

tests don't run in travis or azure pipeline

When i run my tests in travis or the azure pipeline none of the tests get run i get the following output

[main 2019-07-15T19:19:35.386Z]β€Œ update#setState idleβ€Œ

nvm is not compatible with the "npm_config_prefix" environment variable: currently set to "/usr/local"

Run `unset npm_config_prefix` to unset it.

Exit code:   0
Done

It never actually gets to my test runner. It works fine outside of travis or azure pipeline. I also seem to have more success if i run the tests in batches however sometimes batches will get skipped with the same output as above. This happens on mac and linux but mac seems to be worse

Failures to download VS Code are not catchable

On GitHub actions, the VS Code download seems to be quite unreliable, often failing like this:

Downloading VS Code 1.39.2 from https://update.code.visualstudio.com/1.39.2/linux-x64/stable
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27)
Emitted 'error' event at:
    at TLSSocket.socketErrorListener (_http_client.js:392:9)
    at TLSSocket.emit (events.js:198:13)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)

I tried to put some retry logic around it like this:

let currentAttempt = 1;
const maxAttempts = 5;
while (currentAttempt <= maxAttempts) {
	try {
		console.log(`Attempting to download VS Code attempt #${currentAttempt}`);
		await vstest.downloadAndUnzipVSCode();
		break;
	} catch (e) {
		if (currentAttempt >= maxAttempts)
			throw e;

		console.warn(`Failed to download VS Code, will retry: ${e}`);
		currentAttempt++;
	}
}

However this has no effect. It seems like the error during download does not result in the promise returned from downloadAndUnzipVSCode being rejected, but instead the unhandled error brings the process down.

Here's a GitHub actions build log that has this failure:

https://github.com/Dart-Code/Dart-Code/runs/273348206#step:12:85

Pin an insiders version to not auto update

I have pretty slow internet and so it takes a while to re-download vscode insiders every time there is a new release. I know that a pr was made to explicitly introduce this auto-updating behavior: #25 but would it be possible to also have the option to pin an insiders version like: 1.42.0-insider or have an option to not compare to the latest hash?

Disable Update Checking in Tests?...

hey again πŸ‘‹

i'm seeing the following in my test logs:

[main 2019-08-26T20:14:15.598Z] update#setState checking for updates

is there a way to disable update checks across the board (for extensions and vscode itself) within tests?

Publish 1.4.0 to npm

Would you be able to publish 1.4.0 to npm?
We're facing an issue that would benefit from the change in #56

Thanks.

[Feature Request] test webviews

It would be useful if this module could provide utilities for testing webviews.
Here are some things that might be useful for testing an extension that for example provides a preview of markdown or svg or html:

  1. Checking whether or not a webview has opened
  2. Checking if there occurs an error in the console of the webview when the webview is opened
  3. Getting the html content of the webview to check if it is updating correctly
  4. Accessing the webview directly and emulating clicks or mouse movements (document.querySelector("button").click())

Also: Here is an idea what the api for 1. and 3. could look like:

import * as vscode from 'vscode';
import { getWebviews } from 'vscode-test-utils';

test('markdown preview', async () => {
	await vscode.window.showTextDocument('example.md');
	await vscode.window.vscode.commands.executeCommand('markdownPreview.open');
	const markdownWebviews = await getWebviews('markdownPreview');
	expect(markdownWebviews.length).toBe(1);
	const markdownWebview = markdownWebviews[0];
	expect(markdownWebview.html).toContain('<h1>Markdown Example</h1>');
	await setTextDocumentContent('# Markdown Example!!!');
	expect(markdownWebview.html).toContain('<h1>Markdown Example!!!</h1>');
});

Make it possible to run extension tests while VS Code is running

I would like to be able to run extension tests (that use vscode-test) in the Mocha Test Explorer (using a custom "launcher script").
However, when I try to run the tests from this extension, I get the error message "Running extension tests from the command line is currently only supported if no other instance of Code is running.".
Please make this possible, it would be much more convenient to be able to run extension tests in the Test Explorer.

"Unexpected token < in JSON at position 0"

When running tests on Travis just now I saw this:

undefined:1
<html>
^
SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at IncomingMessage.<anonymous> (/Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/request.js:57:42)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

My guess is that the JSON file that's downloaded to check versions was not JSON (but presumably still returned a 200-response, since otherwise it would throw "Failed to get JSON").

Since the error is quite confusing, it may be worth catching the error parsing the JSON and printing a message (eg. including the URL and making it clear that it returned non-JSON content).

VSCode tests fail to launch server despite the extension working in normal condition.

Hi there Shivam here,
So we at hyperledger solang community are developing a vscode extension and server for Solidity language.
Till now, we are able to implement some client-server communication features and wanted to back it up with some tests.
We are launching the server as a child process and it is running fine using this configs: https://github.com/Hyperion101010/sls/blob/master/src/client/extension.ts#L79
The extension and server work fine in normal condition but fail to initialize server while in testing(client is implemented in ts and server is in rust). The test file being:
https://github.com/Hyperion101010/sls/blob/master/src/test/suite/extension.test.ts

Server code: https://github.com/Hyperion101010/sls/blob/master/tower-lsp/examples/server_eg.rs
Now I want some clarification if there is any difference in the process to launch server in testing as compared to normal process.
I mean how can we ensure the server is running in the tests?
The code for same is here(the test function):
https://github.com/Hyperion101010/sls/blob/master/src/test/suite/helper.ts

Along with that here is a screenshot:
testerror

Sir any insight on implementation of test when the client and server are implemented in different languages, testing in general would be very helpful.
Let me know.

Repo relies on globally installed tsc

Shouldn't typescript be a dev dependency?

PS D:\dev\Microsoft\vscode-test> yarn
yarn install v1.10.1
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
$ tsc -p ./
'tsc' is not recognized as an internal or external command,
operable program or batch file.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

Can run tests even when VS Code is running if `--user-data-dir` is specified or, the version is different.

Version: 1.43.0
Commit: 78a4c91400152c0f27ba4d363eb56d2835f9903a
Date: 2020-03-09T19:34:44.548Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 18.7.0

I have noticed that we can run tests from the command line even when VS Code is running if --user-data-dir is specified, or the version is different from the current one.

runTests({
    extensionDevelopmentPath,
    extensionTestsPath,
    launchArgs: [
        testWorkspace,
        '--user-data-dir=' + tmpdirName,
    ]
})

or

runTests({
    version: '1.42.1', // Old version
    extensionDevelopmentPath,
    extensionTestsPath,
    launchArgs: [
        testWorkspace,
    ]
})

If this is an intended behavior, I think it should be documented, which extension authors would want to know. If not, and it leads unexpected problems, it should be documented too.

Especially, --user-data-dir should be documented anyway in https://code.visualstudio.com/api/working-with-extensions/testing-extension since the integration tests should be executed with a default settings.json, not with extension authors' one.

The architecture is so great that we can easily develop integration tests of VS Code extensions. Thank you.

Fix spawn errors caused by testRunnerEnv

Spawn Error: /home/parallels/vscode-test-sample/.vscode-test/vscode-1.30.0/VSCode-linux-x64
/code: error while loading shared libraries: libgconf-2.so.4: cannot open shared object file: No such file or directory

This is caused by using testRunnerEnv directly for spawning process. Correct way should be assigning it to process.env.

https_proxy is not used, when requesting vscodeStableReleasesAPI

I'm developing vscode extension in a proxy environment.
I'm also having trouble downloading vscode because our https proxy address is started with http not https.

This is probably caused due to the code below.

var HttpsProxyAgent = require('https-proxy-agent');
var HttpProxyAgent = require('http-proxy-agent');
var PROXY_AGENT = undefined;
if (process.env.npm_config_proxy) {
    console.log("Using proxy " + process.env.npm_config_proxy);
    if (process.env.npm_config_proxy.startsWith('http://')) { // <-- our proxy is started with http but i want to request https url with https proxy setting
        PROXY_AGENT = new HttpProxyAgent(process.env.npm_config_proxy);
    }   
    else if (process.env.npm_config_proxy.startsWith('https://')) {
        PROXY_AGENT = new HttpsProxyAgent(process.env.npm_config_proxy);
    }   
}
function urlToOptions(url) {
    var options = url_1.parse(url);
    if (PROXY_AGENT) {
        options.agent = PROXY_AGENT;
    }   
    return options;
}

"Failed to download and unzip VS Code insiders"

I'm seeing this on Travis this morning, both on Linux and macOS. The error doesn't really give much info on what went wrong (looks like the real error isn't included in the output), but the URL that's output seems to be 404s, for ex:

https://update.code.visualstudio.com/latest/darwin/insiders

I don't know if this is an issue here (eg. wrong URL) or if something may have gone wrong with the insiders builds? (the link on the website seems to work ok)

Current working directory: /Users/travis/build/Dart-Code/Dart-Code
Downloading VS Code insiders into .vscode-test/vscode-insiders.
Downloading VS Code from: https://update.code.visualstudio.com/latest/darwin/insiders
Failed to run test with error:
Error: Failed to download and unzip VS Code insiders
    at /Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:207:27
    at step (/Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:36:23)
    at Object.throw (/Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:17:53)
    at rejected (/Users/travis/build/Dart-Code/Dart-Code/node_modules/vscode-test/out/download.js:9:65)
    at process._tickCallback (internal/process/next_tick.js:68:7)
      Ended after: 0.881s

Test fail to run when tar or unzip are missing with an unhelpful error message

The function unzipVSCode (

function unzipVSCode(vscodeArchivePath: string) {
) should decompress the downloaded archives. It uses the unzip or tar utilities for that, but does not check the return value of child_process.spawnSync() and if these utilities are missing, then you get a rather unhelpful error message when running the tests:

$ node ./out/test/runTest.js
Downloading VS Code 1.39.2 from https://update.code.visualstudio.com/1.39.2/linux-x64/stable
Downloaded VS Code 1.39.2 into .vscode-test/vscode-1.39.2
Test error: Error: spawn /vscode-obs/.vscode-test/vscode-1.39.2/VSCode-linux-x64/code ENOENT
Exit code:   -2
Done

Failed to run tests
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Solution:
check the return values in

cp.spawnSync('unzip', [vscodeArchivePath, '-d', `${extractDir}`]);
and
cp.spawnSync('tar', ['-xzf', vscodeArchivePath, '-C', extractDir]);
and throw an exception if res.status != null && res.status != 0 or res.signal != null.

I'd submit a PR, but getting a CLA approved takes longer than to actually fix this issue...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.