My findings are reported in a test project that I've created at https://github.com/temelj/xunit-mstest-demo

I know that .NET Core 2.2 is EOL, I have verified that the same happens under .NET Core 2.1 also.

Downgrading to Microsft.NET.Test.Sdk 16.2.0 seems to be ok (I haven't tried any versions in-between)

I mean project output directory, nuget and etc.

• We could have the identical behavior for full and core .net, I mean when I specify TestAdapterPath for .net full it does not try to find adapters in the project output directory but for .net core it properly finds.

So I should copy my custom adapter to the root of solution directory to run tests for mixed solution (with full .net and core .net projects). And vstest could spend potentially significant time finding all required adapters when agent checkout directory has a lot of files/directories. It is very important for CI.

• It would be great if we could specify several paths separated by semicolon or add new parameter TestAdapterPaths for that to work together with other lookup methods.

Hi Team,

I have added the package for Code Coverage from the Package Manager Console by passing Install-Package Microsoft.CodeCoverage -Version 1.0.3.

I then ran Analyze Code Coverage but I don't see any results in Code Coverage Results window.

I am trying this for .NetCoreApp 1.0, Can anyone help me out here?

Visual Studio Enterprise 2017 Version- 15.2(26430.6)

Also I want to know is there a support for the same on VSTS CI builds? If so, how?

Thanks,
Srivatsa

Hi,
I just created a UWP Unit Test App in VS 2019. The first thing I did was check the packages and see if they were up to date. It appears that the Microsoft.NET.Test.Sdk package was out of date (it was at 16.0.1 and Nuget reports the latest as 16.1.1. So I thought let's just update the package before I start doing anything. I try and immediately get:
Cannot modify an evaluated object originating in an imported file "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\WindowsXaml\v16.0\Microsoft.Windows.UI.Xaml.Common.targets"
and the upgrade fails.

is this a problem with vstest? Why would this happen? It's pretty confusing.

We need to run a bunch of functional tests in Azure DevOps for the Azure SDK for Rust. Would it be possible to use the Azure DevOps VSTest@2 task to call a custom Test Execution Host? What would that configuration look like to run the custom test host? Can the custom Test Execution Host be written in Rust?

Hi,

I’m trying to get the test coverage from a dotnet core test project but I can’t see the results (the tests runs successfully). I followed the steps that you shared on microsoft/vstest#579.

I you can take a look on my test solution i will apreciate it, you can download from this url https://www.dropbox.com/s/qjsqdxtckmauvtt/TestSolution.zip?dl=0

Included a base test, a nunit test project and xunit test project, when runs coverage process the result are from test ptojects not for the base project.

Thanks in advance,

Roger

Data collector doc mention we can access test results from attachment. I still have no clue how to access it? Can you give an example?

The Azure DevOps "VSTest task" has the built in ability to rerun individual tests that fail.

I would like to this everywhere we use VSTest as the test runner but I can't use the YAML task to call VSTest (well call VSTest.consol.exe directly, so we can selectively call other runners for Android and Linux docker containers).

My problem is, I only see this option for the VSO UI and YAML task.
I can’t find the “vstest.console.exe” command line switch that is the equivalent to the VSTest task parameters “rerunFailedTests: true”, and “rerunMaxAttempts: 3”.

Searching through the azure-pipelines-tasks GitHub, I don’t see where they pass that switch on or store a list of failed test to rerun. I think they may be adding it to a block in the .runsettings file they pass to VSTest.console.exe.

Can someone please tell me how to use the "rerunFailedTests" option when calling VSTest.Console.exe directly? This would be a great item to add to the Documentation Repo?

Thanks,

Brian.

Consider the below scenario:
TestClass1:
TestAA - TestCategory - A, 11
TestAB - TestCategory - A
TestClass2:
TestBB - TestCategory - B, 11
TestBC - TestCategory - B
TestClass3:
TestCC - TestCategory - C, 11
TestCD - TestCategory - C

How to create the TestCaseFilter that will run all the tests that have Categories 11, A, B and C?
In our case, it should run only three tests but it is picking up all the tests with my below filter:

First Approach:
"(TestCategory=11&TestCategory=A)&(TestCategory=11&TestCategory=B)&(TestCategory=11&TestCategory=C)"

Second Approach:
"(TestCategory=11&(TestCategory=A&B&C))"

• Add QC for Q4

I have a CMake based C++ app with some CTests.
Test coverage tool works fine (almost) from inside VS Enterprise. The Test Explorer discovers the defined CTests and the Coverage Tool runs them and creates a report.

How can I run the test coverage over the available CTests from command line?
I guess VSTest.Console.exe should be used, but not clear how.
Can you please provide information about it?

Thanks and regards.

Hi there,
I'm trying to add a custom log collector to our tests and I'm not able to get the collector recognized. It very simply just collects failed test cases and prints them out at the end.

I've followed the examples found over at analyize.md and have the following:

// ./LogCollector/LogCollector.cs
using System.Collections.Generic;
using System.Xml;
using Microsoft.VisualStudio.TestPlatform.ObjectModel;
using Microsoft.VisualStudio.TestPlatform.ObjectModel.DataCollection;

namespace LogCollector
{
    [DataCollectorFriendlyName("LogCollector")]
    [DataCollectorTypeUri("datacollector://LogCollector")]
    public class LogCollector : DataCollector
    {
        private readonly List<TestCase> _failedTests = new List<TestCase>();
        private DataCollectionLogger _logger;

        public override void Initialize(XmlElement configurationElement, DataCollectionEvents events,
            DataCollectionSink dataSink,
            DataCollectionLogger logger, DataCollectionEnvironmentContext environmentContext)
        {
            events.TestCaseEnd += TestCaseEnd_Handler;
            events.SessionEnd += SessionEnd_Handler;
            _logger = logger;
        }

        private void SessionEnd_Handler(object? sender, SessionEndEventArgs e)
        {
            _failedTests.ForEach(testCase =>
                _logger.LogError(new DataCollectionContext(testCase), testCase.FullyQualifiedName)
            );
        }

        private void TestCaseEnd_Handler(object? sender, TestCaseEndEventArgs testCaseEndEventArgs)
        {
            if (testCaseEndEventArgs.TestOutcome == TestOutcome.Failed)
            {
                _failedTests.Add(testCaseEndEventArgs.TestElement);
            }
        }
    }
}

and my .runsettings file looks like this:

<!-- ./.runsettings -->

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
        <RunConfiguration>
            <TestAdaptersPaths>./LogCollector/LogCollector.cs</TestAdaptersPaths>
        </RunConfiguration>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="LogCollector" uri="datacollector://LogCollector"/>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

When I run the tests with dotnet test --collector LogCollector I'm faced with the error:

Data collection : Unable to find a datacollector with friendly name 'LogCollector'.
Data collection : Could not find data collector 'LogCollector'

Does anyone have any insight into how I can get the data collector to be picked up by the test runner?

Thanks!

Is there not a generic way to specify "run only tests that don't match these class names"? Ideally want to chain (ClassName !~<Example>)&(ClassName !~<Example>)&....

Allowed operators:
    = implies an exact match
    != implies an exact not match
    ~ implies a contains lookup

Hello all,

We are having an issue regarding Native C++ tests discovery in Visual Studio or in command line. It seems to be new in 15.8. Browsing around I could not find any work around or similar issue opened. I managed to build the same release version and could not find an easy way of fixing it.

vstest.console.exe "D:\TFS\Synergis\Dev\Integration\Main\Source\bin\Debug\AxisExtension.Native.UnitTests.dll" /TestAdapterPath:D:\TFS\Synergis\Dev\Integration\Main\Source\bin\Debug /diag:diag.txt
Microsoft (R) Test Execution Command Line Tool Version 15.8.0-dev
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
Logging Vstest Diagnostics in file: D:\Code\vstest-rel-15.8\diag.txt
Data collection : Logging DataCollector Diagnostics in file: D:\Code\vstest-rel-15.8\diag.datacollector.18-10-25_09-18-01_06900_1.txt
Data collection : Data collector 'UnitTestIsolation' threw an exception during type loading, construction, or initialization: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Length cannot be less than zero.
Parameter name: length
at System.String.Substring(Int32 startIndex, Int32 length)
at Microsoft.VisualStudio.IntelliTrace.TraceLogger.get_InstallDir() in F:\dd\src\edev\diagnostics\intellitrace\tracelog\TraceLogger.cs:line 1222
at Microsoft.VisualStudio.IntelliTrace.TraceLogger.GetProfilerEnvVars() in F:\dd\src\edev\diagnostics\intellitrace\tracelog\TraceLogger.cs:line 937
at Microsoft.VisualStudio.TraceCollector.CommonDataCollector..ctor() in F:\dd\src\edev\diagnostics\intellitrace\collectorV2\CommonDataCollector.cs:line 142
at Microsoft.VisualStudio.TraceCollector.UnitTestIsolationDataCollector..ctor() in F:\dd\src\edev\diagnostics\intellitrace\collectorV2\UnitTestIsolationCollector\UnitTestIsolationDataCollector.cs:line 35
--- End of inner exception stack trace ---
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at System.Activator.CreateInstance(Type type)
at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginManager.CreateTestExtension[T](Type extensionType) in D:\Code\vstest-rel-15.8\src\Microsoft.TestPlatform.Common\ExtensionFramework\TestPluginManager.cs:line 96
at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.LazyExtension`2.get_Value() in D:\Code\vstest-rel-15.8\src\Microsoft.TestPlatform.Common\ExtensionFramework\Utilities\LazyExtension.cs:line 153
at Microsoft.VisualStudio.TestPlatform.Common.DataCollector.DataCollectionManager.TryGetTestExtension(String extensionUri) in D:\Code\vstest-rel-15.8\src\Microsoft.TestPlatform.Common\DataCollection\DataCollectionManager.cs:line 404
at Microsoft.VisualStudio.TestPlatform.Common.DataCollector.DataCollectionManager.LoadAndInitialize(DataCollectorSettings dataCollectorSettings, String settingsXml) in D:\Code\vstest-rel-15.8\src\Microsoft.TestPlatform.Common\DataCollection\DataCollectionManager.cs:line 429.
Logging TestHost Diagnostics in file: D:\Code\vstest-rel-15.8\diag.host.18-10-25_09-18-01_79093_1.txt
Guid should contain 32 digits with 4 dashes (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx).
No test is available in D:\TFS\Synergis\Dev\Integration\Main\Source\bin\Debug\AxisExtension.Native.UnitTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Test Run Failed.

We have the same issue just listing the tests. In some cases, it can list a couple tests before failing. We could not find anything wrong in our tests related to that issue. I included diag files.

Thanks,

diag.host.18-10-25_09-18-01_79093_1.txt
diag.txt
diag.datacollector.18-10-25_09-18-01_06900_1.txt

https://github.com/Microsoft/vstest-docs/blob/master/docs/releases.md

The TestPlatform vsix links point to https://vsdrop.corp.microsoft.com/, which is not accessible. Where can I download those?

Hello,

A little background: I'm working on a programming language implementation. While many aspects of the compiler and VM can be easily tested in e.g. xUnit, there are other things that are much easier to test with other tools similar to LLVM's FileCheck. That said, I'd like for these tests to still be run and reported as part of dotnet test to make the development workflow as simple as possible.

So, from what I gather, the proper way to do this is to implement a test adapter that reports and runs these non-xUnit tests. However, I've not been able to find any current documentation on how to set up a test adapter project and which interfaces I need to implement. Can anyone point me in the right direction?

And a somewhat related question: Is it possible for a test project to declare and export a test adapter directly, such that referencing a NuGet package (e.g. xunit.runner.visualstudio) is not necessary?

Is there a link to docs.microsoft.com where this documentation is available? Or is the only option via this GitHub repo?

Is there a way to know if vstest.console.exe runs in a context of a rerun? Is there something we can use in a custom discoverer (like an environment variable) to detect if is "normal" running or rerunning the failed test?

It looks like currently we don't have any ways to use custom DataCollector (which is implementation of abstract DataCollector class) with .testsettings file. I added the following peace of code to my settings:

<AgentRule name="LocalMachineDefaultRole">
      <DataCollectors>
        <DataCollector uri="datacollector://CustomDataCollector" assemblyQualifiedName="TestLibrary.DataCollector.CustomDataCollector, TestLibrary.DataCollector, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" friendlyName="CustomDataCollector">
        </DataCollector>
      </DataCollectors>
    </AgentRule>

I got an error when I executed the following command:
"[path]/vstest.console.exe" [path]\UnitTestProject1\bin\Debug\UnitTestProject1.dll /Settings:[path]\bin\Debug\Run1.runsettings (All required dlls and the settings files are situated in the [path]/bin/Debug folder)

The diagnostic data adapter 'TestLibrary.DataCollector.CustomDataCollector, TestLibrary.DataCollector, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' on agent 'agent' threw an exception during type loading, construction, or initialization: System.IO.FileNotFoundException: Could not load file or assembly 'TestLibrary.DataCollector, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.

According to documentation:

While the recommended way is to use runsettings or vstest.console args, there are few DataCollectors which only worked with testsettings. E.g.: System Information DataCollector. Below is the sample testsettings for using System Information DataCollector.

So it makes me think that I can't use my custom DataCollector with testsettings. It makes me think that we can use only few pre-defined datacollectors with it. Not custom. I created .runsettings file.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
	<RunConfiguration>
		<!-- Path to Test Adapters -->
		<TestAdaptersPaths>the real path is omitted</TestAdaptersPaths>
	</RunConfiguration>

	<!-- Configurations for DataCollectors -->
	<DataCollectionRunSettings>
		<DataCollectors>
			<DataCollector friendlyName="CustomDataCollector" uri="datacollector://CustomDataCollector">
				<Configuration>
					<LogFileName>DataCollectorLogs.txt</LogFileName>
				</Configuration>
			</DataCollector>
		</DataCollectors>
	</DataCollectionRunSettings>

	<MSTest> 
<!-- I included the previous .testsettings file to .runsettings (The datacollector's declaration was removed from the testsettings) -->
		<SettingsFile>Local3.testsettings</SettingsFile> 
		<ForcedLegacyMode>true</ForcedLegacyMode> 
	</MSTest>

</RunSettings>

But I got the next error:

Data collectors other than Code Coverage and Microsoft Fakes configured via run settings are not supported with embedded test settings. Please see https://aka.ms/vstest-configure-datacollector for more information.

So I'm interesting how can I use my custom datacollector and use .testsettings in the same time. I need the .testsettings file for deployment items.

I've seen an error in docs/analyze.md : if I follow the procedure, I get coverage on test assemblies.

Full should be added to assemblies containing the code under test.

I tried and it is OK for me.

When I run "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "D:\BuildAgent\work\22d15c2ebd171441\MYAPP.Tests\bin\Release\MYAPPTests.dll" /collect:"Code Coverage" the file generated is thrown at a folder within the TestResults folder naming it every time in a different way.

I would like the file to be put in a specific folder, ie: TestResults/CoverageReport, with an specific name, ie: MYAPP.coverage.

Is there a way this can be done using the vstest.console.exe utility through the command line?

I'm looking for further details in to the "In-Proc Data Collection Extensibility API" detailed in this doc - is there anywhere I can read more about how this section of the system works and how we can hook in to it?

Add details regarding the new debugging workflow as specified here to the existing RFC

We have logic in our test codes that hooks up AppDomain process exit event. When running these tests with vstest.console.exe, we found out that the AppDomain was unloaded. Is there clear documentation on when vstest would create/unload new AppDomain? Is it one per test assembly, per unit test or something else?

This documentation did mention the setting of DisableAppDomain. Is it disabling creation of separate AppDomain's for running tests?

when running dotnet test -s test.runsettings, path to runsettings file does not work with relative path.

The example test case filtering code does not compile. Specifically here:

var filterExpression = runContext.GetTestCaseFilter(supportedProperties, (propertyName) =>
{
    TestProperty testProperty = null;
    supportedProperties.TryGetValue(propertyName, out testProperty);
    return testProperty;
});

GetTestCaseFilter() has this signature:

ITestCaseFilterExpression GetTestCaseFilter(IEnumerable<String> supportedProperties, Func<string, TestProperty> propertyProvider);

but supportedProperties in the example is a Dictionary<string, TestProperty>

Would love to see if there is any possible VSTest integration with LinkedIn's BluePill for use with Windows Based Hyper-V Virtual Machines.

It's open source 👍 https://github.com/linkedin/bluepill

I am unable the determine the code coverage of .net core 2.0 code in the TFS Build steps which are built for .Net core.
I tried the following:

"TestPlatform\vstest.console.exe" --collect:"Code Coverage"
But I got this error message:

Data collector 'Code Coverage' message: Cannot find CodeCoverage.exe.

The Not-Contains operator was added in microsoft/vstest#1803, but it is not mentioned in documentation which makes this very hard to discover.

The documentation should probably call out in which version(s) of VSTest this feature is available, so that users can determine if their version is new enough to use it.

In the DataCollector.md under using datacollector there is a sample xml how to use a custom datacollector

<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="NewDataCollector" />
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

The previous commandline arguments that are shown as an alternitive also contain /testadapterpath:<Path to DataCollector>, but there is no mention of the path in the .runsettings sample.

In the documentation in analize.md there the sample contains PathToDataCollector

  ... 
  <RunConfiguration>      
    <!-- Path to Test Adapters -->  
    <TestAdaptersPaths>PathToAdapters;PathToDataCollectors</TestAdaptersPaths>  
  </RunConfiguration> 
  ...

But there is the DataCollector also defined with an URI <DataCollector friendlyName="MyDataCollector1" uri="datacollector://MyCompany/MyDataCollector1/1.0">

Somewere else I also seen an attribute codebase.

What I currently don't understand is,

• when do I need uri in addition to frindlyName,
• can uri be whatever it want (as long as it is the same in the implemented datacollector) or does it need to start with datacollector:// (I've seen samples that started with my://)
• Does PathToDataCollectors is the path to the DLL that contains the implemented datacollector or is it the Path to the folder containing the DLLs

It would be good to document in TestCase filter how to pass a --filter to filter a property for no value.

For example, consider a solution where most tests run without any dependencies, and so have no TestCategoryAttribute. However, some tests have dependencies such as other services running, that make them unsuitable for running on a CI build. These have a TestCategoryAttribute that state the dependencies. It would be useful to define a build that runs all the tests with no category. Something like ---filter TestCategory=''. Currently, there doesn't seem to be any documentation on how to do that.

The format of Exclude values is not documented anywhere.

Is it wildcards? Is it regular expressions?

I have 2 queries:

1. When I run the command vstest.console.exe --collect:"Code Coverage" --framework:".NETCoreApp,Version=v1.0" my test dll path there is no .coverage file generated. However when I run Analyze Code Coverage manually from the Test Explorer, I can see the code coverage results as well as the .coverage file getting generated. Any help here?

2. I would like to use the commandline runner in my VSTS build to collect code coverage results for dotnet core projects. Are there any OOB tasks which is currently there to do so?

Additional Information:-

My Visual Studio version is 15.6.0 Preview 4.0 and dotnet version is as shown below.

In architecture page, section Flow: Test Runner - Test Host, the diagram for discovery phase is not really readable. It would suffice to make it as readable as the execution one.

Thanks!

The docs show how to use 35, 40, 45, but don't mention anything greater. I spent some time (hoping to get it right and create a PR), but haven't gotten it right yet.

I want to run in 4.7.2, how do I do it? I'll add it with a PR once I know.
From the current docs.

<!-- Framework35 | [Framework40] | Framework45 -->
<TargetFrameworkVersion>.NET Framework, Version=v4.6</TargetFrameworkVersion>

Both of these links should be updated.
https://github.com/Microsoft/vstest-docs/blob/master/docs/configure.md
(has <TargetFrameworkVersion>.NET Framework, Version=v4.6</TargetFrameworkVersion> as an example, but in VS I get "
[11/15/2018 2:39:26 PM Error] Failed to configure settings for runsettings plugin 'VSTest Run Configuration' as it threw following exception:
'An error occurred while loading the settings. Error: Invalid settings 'RunConfiguration'. Invalid value '.NET Framework, Version=v4.6.2' specified for 'TargetFrameworkVersion'..'"

https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2017

https://github.com/MicrosoftDocs/visualstudio-docs/issues/1305 for .Net Core should also be clearly mentioned.

Essentially I'm trying to build a script that will run multiple unit tests with only knowing the names of the DLL/file and the test name.

Stuff I have tried:
Assuming that there is 5 tests in the testPath/Tests.dll file

dotnet test "testPath/Tests.dll" --filter "namespace.class.test1" --filter "namespace.class.test2" 
dotnet test "testPath/Tests.dll" --filter "namespace.class.test1&namespace.class.test2" 

Is this something that is even possible?

I'm interested in the SDK to extend the testing framework. However, in this section in the documentation for the SDK that lists compatibility it refers to a version of visual studio that doesn't exist.

In the "Test Platform SDK" document

Compatibility

Version Compatibility
11.0.0 Supports VS 2012 - present
15.0.0 Supports VS 2016 - present

I'm confused, what is VS 2016? Do you mean VS 2015 with some updates that were released in 2016 or VS 2017? Could you please clarify this in the documentation?

Thanks.

Putting the following code in a test case, or executing something like this some where in your application wil cause tests to run indefinitely. My guess it is messes up the communication with VSTest, because JsonConvert will fall back to the settings returned by JsonConvert.DefaultSettings.

JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
    ContractResolver = new DefaultContractResolver
    {
        NamingStrategy = new SnakeCaseNamingStrategy()
    }
}

I've found that JsonConvert is being called here. But replacing it with this.Serialize() doesn't solve the issue so I think it must be used somewhere in TranslationLayer package which isn't open-source(?).

This Reporting test results i found interesting and tried to test it but facing few issues

Reference: https://github.com/microsoft/vstest-docs/blob/master/docs/report.md

Below Steps followed
Created one Unit Test Project with one test and added the nuget reference of package Microsoft.TestPlatform.ObjectModel. And tried to execute the below three lines in developers command prompt

1. vstest.console.exe Tests.dll /logger:html
2. vstest.console.exe Tests.dll /logger:"html;LogFileName=relativeDir\logFile.html"
3. vstest.console.exe Tests.dll /logger:"html;LogFileName=c:\temp\logFile.html"

But getting the below error,

can any one help me on this issue

RE: https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md

You've found the GitHub repository which contains the content for the Visual Studio Test Platform documentation.

Is the "Visual Studio Test Platform documentation" built and online? If yes, is the TestCase filter doc hosted online as a webpage within a web-based document set ... I can't find it hosted anywhere in a website, or is it only hosted on GH at that link?

cc/ @smadala @codito

In the docs it says that with xUnit I should be able to do this:

dotnet test --filter Category=bvt

but it turns out that it doesn't work, I get following exception:

"Exception filtering tests: No tests matched the filter because it contains one or more properties that are not valid (Category). Specify filter expression containing valid properties (DisplayName, FullyQualifiedName) and try again."

Hi
I can not run a local server follow Readme.md docs when I follow the steps to below:
Editing docs

We use docfx for building this documentation. A short primer on editing this repo is below. First, download latest release of docfx and extract it locally. We will use d:\tmp\docfx as destination for these steps. Open a command prompt, git clone this repo. Use the following commands to build and run a local server.

cd d:\src\vstest-docs
d:\tmp\docfx\docfx.exe build
d:\tmp\docfx\docfx.exe serve
Open http://localhost:8080/_site in a browser to see the rendering of the documentation.

I can not find "docfx.exe" in my local File Explorer after latest docfx downloaded and the repo git clone finished.

Am I missed some settings or your docs it not clearly ?
below is my local File Explorer screenshot:

Thanks
Houfa

While creating DevOps build scripts for Visual studio projects VSTest *.trx xml file output includes references to the *.coverage files. One of the build artifacts I believe important to generate is the Unit Test results and code coverage, with the associated full semantic version used for the file names. The process of build those artifacts and enabling VSTO to read the results is unclear.

Here is a sample Cake.Build Task. As you can see there are some attempts to allow the *.trx and *.coverage files to be stored in the same location, but the VS IDE doesn't seem to recognise the changes when I open the test results and then try and navigate to the code coverage file.

`
Task("Artifacts-Tests")
.IsDependentOn("Set-Version")
.IsDependentOn("Run-VSUnit-Tests")
.Does(() =>
{
IEnumerable testArtifacts = GetFiles("./TestResults*/**/.trx");
foreach (FilePath artifact in testArtifacts)
{
FilePath renamedUnitTestResult = new FilePath("./BuildOutput/" + buildConfiguration + "/" + fullSemanticVersion + ".trx");
MoveFile(artifact, renamedUnitTestResult);
// Change the path to the coverage file, as we want them both in the same folder.
XmlPokeSettings trxSettings = new XmlPokeSettings {
Namespaces = new Dictionary<string, string> {
{ / Prefix / "ns", / URI */ "http://microsoft.com/schemas/VisualStudio/TeamTest/2010" }
}};

    XmlPoke(renamedUnitTestResult, @"/ns:TestRun/ns:TestSettings/ns:Deployment/@runDeploymentRoot", "" , trxSettings );
    //  Change the reference coverage file to the semantic version name.
    XmlPoke(renamedUnitTestResult,"/ns:TestRun/ns:ResultSummary/ns:CollectorDataEntries/ns:Collector/ns:UriAttachments/ns:UriAttachment/ns:A/@href", fullSemanticVersion + ".coverage", trxSettings);
}

testArtifacts = GetFiles("./TestResults*/**/*.coverage");
foreach (FilePath artifact in testArtifacts)
{
    FilePath renamedUnitTestResult = new FilePath("./BuildOutput/" + buildConfiguration + "/" + fullSemanticVersion + ".coverage");
    MoveFile(artifact, renamedUnitTestResult);
    break; // Do the first only. Not sure why two *.coverage files are getting generated
}


Information("Collected all the test artifacts.");    

});
`

we are seeing a difference in code coverage results created from Visual studio UI and the command line option dotnet test , even though both using the same runsettings file.

After deep look found out the Ui is using coverable lines to calculate the percentage.

is there any option available in dotnet test to calculate the coverage based on Coverable lines?

Hi,

I'm trying to Migrate Tests from MTM 2012 to TFS 2017 and now I have problem with test run and i'm not sure do i need to have VS Test Agent on PC where i'm going to run UI tests using VsTest.

As I understand I do not need to have it.

I have error when run tests but i'm not sure why it is

Failed CodedUITest_2873_Ranorex_tests_will_run (Data Row 0)
Error Message:
Error calling Initialization method for test class Tests.FM.CodedUITestMain: System.IO.FileNotFoundException: The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
Stack Trace:
at System.Reflection.RuntimeAssembly.nLoadFile(String path, Evidence evidence)
at System.Reflection.Assembly.LoadFile(String path)
at Microsoft.VisualStudio.TestTools.UITest.Framework.UITestExtensionPackageManager.LoadAssembly(String assemblyFile)

After building the solution from sources on a windows machine using build.cmd, I tried running the vstest.console,exe at vstest\artifacts\Debug\net451\win7-x64 results in :
No test is available in EPMLiveCore.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Using /diag, the first exception is
File name: 'Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' ---> System.Security.SecurityException: Strong name validation failed. (Exception from HRESULT: 0x8013141A) The Zone of the assembly that failed was: MyComputer at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks) at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark) at System.Reflection.Assembly.LoadFrom(String assemblyFile) at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyLoadContext.LoadAssemblyFromPath(String assemblyPath) at Microsoft.VisualStudio.TestPlatform.Common.Utilities.AssemblyResolver.OnResolve(Object sender, AssemblyResolveEventArgs args)

Am I missing something to be done after building?

A Microsoft.TestPlatform.TranslationLayer NuGet package is mentioned in the documentation and I can see a few links to a MyGet package source in releases.md such as:

https://dotnet.myget.org/feed/vstest/package/nuget/Microsoft.TestPlatform.TranslationLayer/15.0.0-preview-20170106-08

However it looks like this MyGet feed is private. I had a look on a few other NuGet package sources but could not find the TranslationLayer NuGet package.

Will the Microsoft.TestPlatform.TranslationLayer NuGet package be published somewhere so it is publicly available?

RunSettings arguments has supported setting TestRunParameters now as discussed in this issue microsoft/vstest#862

# cmd
dotnet test  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\")

# powershell
dotnet test --%  -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\") 

# bash
dotnet test -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\"\) 

Please verify and update this document https://github.com/Microsoft/vstest-docs/blob/master/docs/RunSettingsArguments.md#example, thanks!

On the following page...
https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#coverage

There is only code examples for core1.1

Please update w/ examples for core2.0

I am trying to run tests with multiple categories using the vstest task in azure pipeline using the below syntax.
"TestCategory=XXX&TestCategory=XXX"

but it always fails with the below error:
No test matches the given testcase filter TestCategory=XXX&TestCategory=XXX in C:\TestLabAgent_work\1\s\APPIUM\RobotWare7\XXX.XXXXX.XXXXXXXXX.Test.Automation\bin\Release\XXX.XXXXXXXX.XXXXXX.Test.Automation.dll

Please help as I need to urgently execute some test cases with only these two categories.