Test assembly dependency resolution in .netcore about vstest HOT 11 CLOSED

@AbhitejJohn @sbaid @pvlakshm @saikrishnav thoughts?

from vstest.

@codito

User will be able to run tests in testproject.dll only after a dotnet publish, so that all dependencies are copied to output directory. dotnet build doesn't copy all dependencies to output folder.

CI build would require dotnet publish on test project so that all dependencies are bundled; since there isn't a nuget package directory

AFAIK, dotnet publish doesn't work until specific runtime is specified. Do we want to force the users to always specify runtime in their test projects?

from vstest.

@harshjain2 without a RID specified, dotnet publish publishes to netcoreapp1.0 directory for netcore target and net46\win7-x64 (or x86) directory for net46 target.

I've updated original comment. CI build (or strictly speaking CDP) requires publish only if the build step has already run on another machine, and test is running in a separate machine with the build artifacts.

from vstest.

@codito
I'm a bit concerned about dependencies of Test Adapter.

Test Adapter can have dependencies on MSTest.ObjectModel and other nuget packages.

Those dependencies are taken care of if test project adds Test Adapter nuget as relevant dependencies will be added in the test dll's .deps.json file.

I am not sure what will happen if user forgets to add Test Adapter. It might work if, just by luck, test dll has all the dependencies required for test adapter. Otherwise it will bomb.

As Test Platform owner, we should have a strict guideline that Test Adapter nuget packages should always be added to the test project just to ensure the .deps.json of test dll has the required dependencies needed for Test Adapter as well.

from vstest.

@codito

1. Test host is a dependency of each test project
2. Test host is not a dependency, bundled with vstest.console package

On this, in order to ensure the dependencies of testhost is taken care of, option 1 is recommended. That way, dependencies of testhost will be present in test dll's .deps.json which we are passing when starting testhost.

from vstest.

@codito @AbhitejJohn Do we know the requirements for asp.net tests?

from vstest.

In Solution #2, packaging would bloat up the vstest.console bundle since it might end up having a testhost and its dependencies built for a lot of netstandard versions. #1 gives the ability to fragment this with the ability to plug in which ever version is appropriate as long as it understands the protocol. This can also be a nice host extensibility entry point where someone can specify their host via a nuget package and vstest.console can communicate with that host.

One thing thats biting me with solution #1 however is the pain of the user to add 3 nuget dependencies -> Framework, Adapter and the test host. Just a thought, can we somehow tie the adapter to have a dependency on test host? That could possibly work @codito .

@harshjain2 : What requirements are we talking about? Currently ASP.NET Core targets the same core CLR. I do not see any plan for post RTM here though: https://github.com/aspnet/Home/wiki/Roadmap .

@codito : Does the new csproj based project system allow users to add references directly to assemblies as well?

from vstest.

@harshjain2 re the dependencies of test adapter, they will cleanly resolve with testproject.deps.json. dotnet build will ensure this. Dependencies of testhost will always be in the current folder of testhost (a dotnet publish has already been run on testhost before packaging).

I think we should take a staged approach (since this is a architectural change): start with approach 2, then move to approach 1.

@AbhitejJohn @harshjain2 probably meant the assembly loading in asp.net. We will do a small spike for this.

Updated notes on adapter discovery based on deps.json above.

from vstest.

Will a runsettings ever be supported? I ask because that has an AssemblyResolution entry where you can specify locations where dependencies may be found.

from vstest.

@pvlakshm I think we need to support runsettings since it also has platform, test logger, data collector and adapter settings. AssemblyResolution entries may not be applicable to netcore framework (they are also not applicable to C++, Nodejs and other languages the runner supports).

Updated notes with two relevant specifications from dotnet-cli around runtime configuration and corehost.

from vstest.

Resolved with 4f34968.

from vstest.