Issue

When the TestPlatform bits are loaded, they cause our binaries to load when they search for the ITestExtensionComponentProvider instances.

Specifically this is done in the LiveUnitTestingClientProvider class, where Assembly.LoadFrom is explicitly called in the InitializeTestExtension method.

Suggested Resolution

An API should be provided where we can explicitly enable our 'component provider'/'service client' once our package is known to be loaded.

Test explorer uses below config file to read features to enable for TPv2. The default vsix should include testplatform.config.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <appSettings>
    <add key="feature.net40" value="false" />
  </appSettings>
</configuration>

Description

User gets an error when TestPlatform.vsix is installed in a VS preview.

Steps to reproduce

• Build the vstest repo
• Uninstall the existing TestPlatform extension from VS
• Install artifacts\Debug\TestPlatform.vsix

Expected behavior

VSIX installation should be successful.

Actual behavior

VSIX installation shows following warning, it is successful if warning is ignored.

---------------------------
VSIX Installer
---------------------------
This extension is not compatible with Visual Studio 15 and will not be supported in RC. Installation may cause Visual Studio to be unstable.  Do you want to continue?
---------------------------
Yes   No   
---------------------------

Environment

VS preview. VSIX from d9dbd28.

Validate if the existing test project templates work with asp.net core.

Description

There are not templates for VB for Unit testing for .NET core.

Steps to reproduce

VS IDE >> File -> New Project -> There is no Unit Test template targeting VB.

Description

If the tests are already discovered and available in the test explorer. Run All should not discover them again.

Steps to reproduce

• Create a new unit test project
• Add the 10k tests (10KTests.zip)
• Build the tests
• After the tests are discovered, Run All

Expected behavior

Tests should directly run and results are displayed.

Actual behavior

Tests are discovered again. This makes the user wait for two minutes (for the 10k tests project).

Environment

VS: alm/25722.00
TestPlatform: d9dbd28

In .net core tests, user needs to reference 3 packages to run tests:

• Test Framework (mstest, xunit, nunit)
• Test Adapter
• Test Host

Can we reduce this?

Test Framework needs to be separate in the options below since each test framework also provides its own console runner.

Options available:

1. Create a meta package for Test Adapter and Test Host
2. Let Test Adapter require Test Host as a runtime dependency; nuget can download and set it up for the user
3. No op. These dependencies are any ways part of the test project template. User doesn't need to add them manually.

We have observed significant delay when test discovery and execution is requested through VsTestConsoleWrapper. For example, the following log shows discovery took ~7 seconds and execution took ~4 second.

[2016-08-26 09:51:55.084 Info] - TestDriver - [Workflow 1] Received TestRequestEvent
[2016-08-26 09:51:55.084 Info] - TestDriver - [Workflow 1] Start test discovery.
[2016-08-26 09:52:02.724 Info] - TestDriver - [Workflow 1] Discovered 0 tests from ClassLibrary2.dll, updating cached data.
[2016-08-26 09:52:02.724 Info] - TestDriver - [Workflow 1] Discovered 1 tests from TestLibrary.dll, updating cached data.
[2016-08-26 09:52:02.740 Info] - TestDriver - [Workflow 1] Start executing 1 tests from 1 assemblies.
[2016-08-26 09:52:06.762 Info] - TestDriver - [Workflow 1] Handle test result for TestSomething2: Passed
[2016-08-26 09:52:06.762 Info] - TestDriver - [Workflow 1] Tests execution completed successfully.
[2016-08-26 09:52:06.762 Info] - TestDriver - [Workflow 1]  Notify Aggregator about completion of test run 

This behavior was observed on both Preview 3 VS and latest willow based build from ML branch. Note that this doesn't happen all the time. I didn't notice this before upgrading to latest TestPlatform NuGet packages, it was usually around 1 second, not sure if this is related to the upgrade.

@saikrishnav

Issue type

Please specify the type of issue:

• Specification
• Code defect
• Question

Currently, TestPlatform NuGet packages contains Newtonsoft.Json.dll, which should be specified as a NuGet dependency instead. This is causing problem for projects that referencing both Newtonsoft.Json and TestPlatform packages.

User should be able to run multiple test assemblies for .net core. In addition, a sln containing multiple .net core projects should have support for discover/execution of tests.

Description

First build of TestPlatform.sln on VS is failing for SmokeTests because of a relative path for packages directory.

Workaround is to do a commandline build first.

Description

TranslationLayer throws an error on the communication channel if vstest.console exits

Steps to reproduce

1. "Run All" from test explorer on a unit test project
2. Terminate the vstest.console.exe process
3. "Run All" on the same project

Expected behavior

Run All should work on step 3

Actual behavior

Following error is thrown:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.WriteByte(Byte value)
   at System.IO.BinaryWriter.Write(Byte value)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.WriteAndFlushToChannel(String rawMessage) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.CommunicationUtilities\SocketCommunicationManager.cs:line 241
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.SendMessage(String messageType, Object payload) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.CommunicationUtilities\SocketCommunicationManager.cs:line 202
   at Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleRequestSender.DiscoverTests(IEnumerable`1 sources, String runSettings, ITestDiscoveryEventsHandler eventHandler) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.VsTestConsole.TranslationLayer\VsTestConsoleRequestSender.cs:line 107
   at Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper.DiscoverTests(IEnumerable`1 sources, String discoverySettings, ITestDiscoveryEventsHandler discoveryEventsHandler) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.VsTestConsole.TranslationLayer\VsTestConsoleWrapper.cs:line 104

Similar behaviour for Debug All Scenario is observed as well.

Environment

vstest.console from 576f536 bits.

Description

For .netcore portable PDBs are the primary symbol format. OM needs to support parsing these symbols.

In .net core, the adapter can be picked up from the output directory of test assembly. This directory should be probed by test platform. With this change, the user will not need to specify a --testAdapterPath with vstest command.

Description

For a netcore test app, vstest.console must be able to run tests against net46 or any specified platform.

Description

If vstest.console exits abruptly, the child test host processes must exit.

Steps to reproduce

• Create a unit test project
• Add a test with sleep for a minute
• Run test
• Kill vstest.console

Expected behavior

Test host should exit.

Actual behavior

Test host keeps running.

Environment

VS: alm/25722.00
TestPlatform: d9dbd28

Description

At the moment TestPlatform uses two different Newtonsoft.Json version: 7.0.1 (specified in TranslationLayer.Nuspec) and 9.0.1 (specified everywhere else). However VS15 ships with Newtonsoft.Json version 8.0.3.

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, commandline for vstest.console, sample project, target
framework etc.

Expected behavior

Share the expected output

Actual behavior

What is the behavior observed?

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console

Description

When user has selected a testsettings file, unit tests are not discovered by test platform.

Steps to reproduce

• Create a new unit test project
• Select a testsettings file
• Build the project

Expected behavior

Tests should be discovered in test explorer

Actual behavior

Tests are not discovered.

Environment

VS: alm/25722.00
TestPlatform VSIX: d9dbd28

Description

Cancelling Test Debugging Session through Test Window or Toolbar menu is not working.

Steps to reproduce

1. Create a Test Project and build it.
2. Right click on the Test discovered and click on Debug Selected Tests.
3. As soon as the debugging starts, terminate debugging either by clicking on Cancel on Test Window or Stop Debugging from Tool bar menu.

Expected behavior

Debugging session should get terminated, user should be able to start run/debug tests.

Actual behavior

Debugging session is not terminated, cannot start new debug/run session. Have to kill vstest.console or restart visual studio.

Provide a /diag option in vstest.console to capture trace logs. User may specify a log capture file with /diag:log.txt. Traces for testhost processes should be captured as log.testhost.<id>.txt where <id> is a unique id such as timestamp.

Same trace logs should be available in stdout when user sets the VSTEST_TRACE environment variable.

Description

Describe the issue you've observed

Steps to reproduce

1. Use the Nunit project under \sasindesk\share\DifferentConfigs\NunitParallelTests1
2. Turn parallel on.
3. Run tests.

Expected behavior

Test run should complete and report results.

Actual behavior

The run does not complete.

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console

OM has a dependency on json.net, it may lead to diamond dependency problems if the test depends on a different version of json.net.

Reevaluate the dependencies for TL, it needs to package TP.Common etc. at the moment.

Description

If user has a very large project, the discovered tests should show in batches in the test explorer. Currently the test explorer just stays in progress until all tests are discovered.

Steps to reproduce

• Create a new unit test project
• Add 10k tests (10KTests.zip)
• Build the project

Expected behavior

Tests should start showing up in chunks.

Actual behavior

Test explorer remains in progress until all tests are discovered.

Environment

VS: alm/25722.00
TestPlatform: d9dbd28

Description

User cannot re-run a test if debugger is detached while debugging a test. Cancel doesn't cancel the the request.

Steps to reproduce

• Create a new test project
• Right click, debug a selected test
• Detach debugger

Expected behavior

Debugger should detach.
Test run should be aborted.
Run All should be active for next run.

Actual behavior

Debugger detaches.
Test run remains in inprogress.
Run All is not active and cancel doesn't cancel the request.

Environment

VS/Preview/25719.00
VSIX: https://github.com/Microsoft/vstest/releases/tag/v15.0.0-preview-20160914-02

The object model needs to be cleaned up based on 11.0 version. Some utilities can be refactored out.

E.g. RunSettingsUtilities. See the PR for more details: https://devdiv.visualstudio.com/DevDiv/VS.in%20Agile%20Testing%20IDE/_git/MSTest/pullrequest/33962

Description

They are multiple places communicationManager.ReceiveRawMessage(); and communicationManager.SendMessage can throw IOException.

Steps to reproduce

Randomly close testhost and vstest.console process

Expected behavior

Should properly handle all kind of process crashes

Actual behavior

Process has been waiting or execption not properly handled.

See http://dotnet-ci.cloudapp.net/job/Private/job/Microsoft_vstest/job/master/job/Microsoft_vstest_Debug_prtest/38/console

When I replaced direct reference to checked-in assemblies with NuGet package from vs-test myget feed, run test through VSTestConsoleWrapper with current test adapters is broken.

Observed behavior is Runtests returns as Aborted. Below is the log from TestImpact:

[2016-08-22 16:54:47.356 Info] - TestDriver - [Workflow 1] Start test discovery.
[2016-08-22 16:54:47.989 Info] - TestDriver - [Workflow 1] Discovered 1 tests from ClassLibrary3.dll, updating cached data.
[2016-08-22 16:54:47.989 Info] - TestDriver - [Workflow 1] Start executing 1 tests from 1 assemblies.
[2016-08-22 16:54:48.460 Info] - TestDriver - [Workflow 1] Tests execution cancelled/aborted, 1 tests left unexecuted.

Please note that run test through Test Explorer doesn't work either, the behavior is run test never finish.

------ Load Playlist started ------
========== Load Playlist finished (0:00:00.002) ==========
------ Discover test started ------
========== Discover test finished: 1 found (0:00:00.686) ==========
------ Run test started ------

Here some info about my setup:

1. Preview 3 VS (classic setup)
2. TestPlatform installed by using vsix from unzipping \cpvsbuild\drops\dd\nuget\TestPlatform.MSTest.1.0.16081901.nupkg
3. MSTest Test Framework and Adapter from the same nupkg above.
4. XUnit adapter @navin22 shared with me privately awhile ago.
5. TestPlatform referenced though NuGet packages:
   • "Microsoft.TestPlatform.ObjectModel": "15.0.0-preview-20160822-01",
   • "Microsoft.TestPlatform.TranslationLayer": "15.0.0-preview-20160822-01"

I have uploaded all the related bits to help you identify the issue to \mlangfs1\public\gel\TestAdapterIssue

@saikrishnav
FYI @ManishJayaswal @drognanar

Description

In .NetCore vstest.console needs full path of the test dll to run the tests

Steps to reproduce

1. Create a .netcore test project.
2. build the project created in step1.
3. Go to the bin/debug folder of the test project of step1.
4. run command "dotnet.exe vstest.console.dll testDllname.dll /TestadapterPath:Path/Of/Adapter"

Expected behavior

It should run the test

Actual behavior

no test getting discover

MSTest templates should be available for .net core. These templates should be pushed into the dotnet/sdk repo.

The Microsoft.TestPlatform.CommunicationUtilities binary directly references Newtonsoft.Json v9.0.0 while VS has standardized on v8.0.3.

Looking at the project, it references v8.0.3, so the project reference to v9 is indirect and NuGet or MSBuild resolves it to be v9 when the compilation actually occurs (causing the binary to have the direct reference to v9).

Description

Describe the issue you've observed

Steps to reproduce

1. Create a unit test project
2. Add the 10k tests (10KTests.zip)
3. Build

Expected behavior

Share the expected output

Actual behavior

During discovery, vstest.console takes up to 50% of CPU. This will slow down other tasks like editing etc. while tests are getting discovered.
Same CPU usage is observed during execution of tests.

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console

Description

TestPlatform vsix will include feature flags through an artifact. TestWindow can use testplatform features based on what is enabled.

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, commandline for vstest.console, sample project, target
framework etc.

Expected behavior

Share the expected output

Actual behavior

What is the behavior observed?

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console

• Change the OutputType to Lib
• Also, remove the Program.cs from the template.
• Try targeting the project to .NetStandard1.*

Test adapters added as reference to the test app are enlisted in the testapp.deps.json. Test platform should use these adapters for discovery/execution of tests.

Description

On running "Test -> Debug All" from the VS; the execution just hangs.

Steps to reproduce

• Create a new Unit Test project
• Click on Test -> Debug All

Expected behavior

All tests should run under debugger. Breakpoints hit.

Actual behavior

Hang.

Environment

VS/Preview/25719.00
VSIX: https://github.com/Microsoft/vstest/releases/tag/v15.0.0-preview-20160914-02

Description

We observed that the call to RunTests was hanging for ever. This was likely a bug in TP that subsequently got fixed but it would be nice to no hang and return error code so that the caller can continue.

User should be able to Debug All and Debug Selected tests in a .net core project.

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.WriteAndFlushToChannel(String rawMessage)
   at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.ConnectToClientAndProcessRequests(Int32 port, ITestRequestManager testRequestManager)
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.PortArgumentExecutor.Execute()
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.ExecuteArgumentProcessor(IArgumentProcessor processor, Int32& exitCode)
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)

\\mlangfs1\public\kevinh\vstest.console.dmp

Description

vstest.console doesn't run tests from test container with relative path.

Steps to reproduce

Run the following command:

dotnet.exe c:\vstest\vstest.console.dll test\testproject.dll /platform:x64 /testAdapterPath:path\to\adapter

Expected behavior

Tests should run.

Actual behavior

Test run fails with following error:

Could not find file test\testproject.dll

Note that tests run if an absolute path to testproject.dll is provided.

Problem

• Test platform expects all dependencies for a test project to be available in the output directory
• Test platform always starts the testhost.exe with a specific testhost.deps.json. In this case, user's dependencies don't resolve appropriately

While this behavior is fine for desktop clr, for .netcore these premises may not work.

Impact of this bug

• User will be able to run tests in testproject.dll only after a dotnet publish, so that all dependencies are copied to output directory. dotnet build doesn't copy all dependencies to output folder.
• User will not be able to specify the specific dependencies that they want. E.g. because testhost.deps.json is used, System.Xml used by testhost will be loaded, although user wanted to use a later version of System.Xml. Multi-targeting will be hard
• Adapter writers need to use custom lookup (Assembly.LoadFrom) for resolving test assembly dependencies (incl. the framework assemblies)

Proposal

(There is no change in the test platform behavior for test projects targeting net46 or desktop clr. We concentrate only on netcore scenarios below)

Execute testhost.dll with testproject.deps.json

In dotnet core, set of dependencies for a project testproject are resolved with testproject.deps.json file available in the output path of the project. Test platform should invoke testhost.dll with the deps file of testproject i.e. testproject.deps.json.

Since testhost.dll is always invoked with testproject.deps.json, it is not possible to reuse the testhost for running tests of all test projects. For each test project/container, a separate testhost process will be invoked. This will have a performance impact (compared to existing implementation where a single testhost process is used for running tests from all assemblies).

Adapter lookup

Adapters are referenced in the testproject as a exclude: compile dependency. Test platform will probe the adapter from paths in testproject.deps.json. The lookup convention remains unchanged (*.testadapter.dll are considered test adapters).

Note that user may still provide a logger (or data collector) extension with a path. This will require the logger to be published.

Every test project depends on TestHost package

Every test project will include a exclude: compile type dependency on Microsoft.TestPlatform.TestHost nuget package. During project build, nuget takes the responsibility to compute the dependencies required for both testproject and testhost and create the testproject.deps.json.

testhost.dll is always compiled with the lowest netcoreapp runtime. This allows it to be retargeted to any higher runtime as specified in testproject.deps.json.

CI/CDP scenario

If the tests are run on the same machine that has source enlisted and build artifacts, simple dotnet test should work.

For CDP, where the build artifacts are consumed, a dotnet publish is required while publishing the artifacts. This ensures all dependencies are available for dotnet vstest on the test machine. In this case, user needs to provide path to testproject.dll directly.

dotnet publish for a netcore target produces a netcoreapp1.0 directory with all dependencies. A runtime specific directory is created for net46 target.

Open questions

• Does this meet requirements for asp.net test apps?

References

• Dotnet CLI: Runtime configuration file spec
• Dotnet CLI: Corehost spec

Description

Double click on discovered tests doesn't navigate to source code. Source information doesn't show up in the Summary pane.

Steps to reproduce

• Create a new unit test project
• Build solution
• Open test explorer

Expected behavior

Tests should be discovered. Double click on test should navigate to source file.

Actual behavior

Tests are discovered. Double click on test doesn't navigate to source.

Environment

VS: alm/25722.00
Testplatform: d9dbd28

Not sure if related, but OM manifest file for reg free com is not there in TPv2 extension folder, they are available in TPv1 folder.

Description

"dotnet.exe vstest.console.dll testDllName.dll" always return with exit-code 1

Steps to reproduce

run a .net core test with vstest.console runner.

Expected behavior

The process should exit with exit-code 0

Description

VSTest runner does not print help message when run without any arguments.

Steps to reproduce

Run vstest.console.exe without any args

Expected behavior

A help message is shown.

Actual behavior

Just copyright message is shown and the runner exits.

Environment

vstest.console.exe version from 53689b0 commit.

Description

Test description attributes like Description, CssIteration, WorkItem, CssProjectStructure do not show up in the trx log. Only Owner and Priority are part of the log.

Steps to reproduce

1. Create an MSTest V1 based project.
2. Add a DescriptionAttribute on top of the test method.
3. Run tests with trx turned on.

Expected behavior

The trx should have the description string associated with the test in the log.

Actual behavior

The description string does not show up in the log.

Environment

I'm trying this on Dev15 prev-5 and on Windows 10.

Test platform should provide a test target and a task to consume the output of a test project to run it with vstest.console.

The task and targets should be available as a nuget package consumed in CLI repo.

• Publish packages required for test3
• Support for single TFM
• Support for multiple TFM

Tracking issue for blockers to enable test impact:

• Missing dependencies in TranslationLayer nuget package to use it in runtime
• Unit tests in testimpact branch are blocked due to dependency of OM on CoreUtilities

This will help us get test coverage. There are two items in this:

• VSTest repo: tests should run with dotnet test3
• Dotnet\CLI repo: tests should run with dotnet test3

Throw a proper message for the user to understand the state and be able to install the dotnet cli.

Description

C# Static CC is not working

Steps to reproduce

1. Create a C# Unit Test Project.
2. Add a static code coverage file.
3. Run the test method

Expected behavior

1. CC File should be generated.

Actual behavior

1. Test did not get execute and no cc file is generated.

Environment

It actually worked when set environment variable "TestingTools_DisableTestplatformV2" to 1.