testura / testura.mutation Goto Github PK

View Code? Open in Web Editor NEW

105.0 105.0 5.0 3.97 MB

Mutation testing tool for C# .NET

License: MIT License

C# 99.81% HTML 0.19%

testura.mutation's People

Contributors

Stargazers

Watchers

Forkers

alexanderjohnston kkerchmar mrksbrg seanliao7 j4w0r

testura.mutation's Issues

Possible localization/UTF bug

On reddit my project was tested using your testing mutator suite here.

Possible Localization Bug

On that attached report, on the function IsUnary in the file Data.cs your program interpreted a part of that function have an equality check against âˆ’ . The actual respective file linked here contains no such character or script which suggests there might be some kind of ASCII/UTF/localization bug in your code.

Incorrect line numbers

The results report that is generated reports line numbers that do not line up with the actual line number in the .cs file.

Problems running Unima

I'm trying to run Unima (downloaded binary from GitHub) from PowerShell, but I'm struggling to get it to work.

The output:

2019-05-17 11:05:09,096: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Opening project at C:\Users\jonas\Downloads\mutation\config.json
2019-05-17 11:05:09,127: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Loading configuration: {
    "SolutionPath":  "C:\\git\\fluentassertions\\FluentAssertions.sln",
    "TestProjects":  [
                         "C:\\git\\fluentassertions\\Tests\\NetCore20.Specs\\NetCore20.Specs.csproj"
                     ],
    "BuildConfiguration":  "Debug",
    "TestRunner":  "dotnet",
    "NumberOfTestRunInstances":  5,
}
2019-05-17 11:05:09,595: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Failed to open project
System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeAssembly.get_DefinedTypes()
   at System.Composition.Hosting.ContainerConfiguration.<>c.<WithAssemblies>b__16_0(Assembly a)
   at System.Linq.Enumerable.<SelectManyIterator>d__17`2.MoveNext()
   at System.Composition.TypedParts.TypedPartExportDescriptorProvider..ctor(IEnumerable`1 types, AttributedModelProvider attributeContext)
   at System.Composition.Hosting.ContainerConfiguration.CreateContainer()
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable`1 assemblies)
   at Microsoft.CodeAnalysis.Host.Mef.MSBuildMefHostServices.get_DefaultServices()
   at Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler.<Handle>d__2.MoveNext()

The project I'm trying to test:
https://github.com/fluentassertions/fluentassertions/

Any hints?

DebugShellView

src/Unima/Unima.csproj has references to three files that are not included in the git repo.

Sections\Debug\DebugShellView.xaml
Sections\Debug\DebugShellView.xaml.cs
Sections\Debug\DebugShellViewModel.cs

HintPath references local user

Found two places that references a specific user.

src/Unima/Unima.csproj:
<HintPath>..\..\..\..\Users\Mille\.nuget\packages\materialdesignthemes\2.4.0.1044\lib\net45\MaterialDesignThemes.Wpf.dll</HintPath>

src/Unima.Console/Unima.Console.csproj:
<PublishUrl>C:\Users\Mille\OneDrive\Dokument\Unima\release\</PublishUrl>

Respect ExcludeFromCodeCoverage attribute

As with other testing frameworks, exclude any class or structure marked with ExcludeFromCodeCoverage from analysis.

Implement mutation testing elements html report

Hey! We also work on a C# mutation testing framework and we are happy to see more people interested in it :) Dropping by to say hello but also to point you in the direction of https://github.com/stryker-mutator/mutation-testing-elements

We created a generic json schema + corresponding html webcomponents that use this json to display a fancy report with the goal of creating a shared standard for reporting in mutation testing land. All you have to do is implement the schema as output and output the html including the json and you'll have a fancy html report basically for free :) You can check out https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/Reporters/HtmlReporter/HtmlReporter.cs and https://github.com/stryker-mutator/stryker-net/blob/417a2b78859425015c7a98ecb86ece301aa7b621/src/Stryker.Core/Stryker.Core/Reporters/HtmlReporter/Files/mutation-report.html for inspiration if you're interested.

SolutionPath in TesturaMutationBaseConfig.json full path

Hello,

I tested the plugin in VS after your post on redit and find it very usefull.
It helped me to find some untested logic in my code.

I noticed that the Solution path in the config.json file is a full path.
But I think it would be usefull if this could also work relative to the location of the config file.
This way you can check the file in and easily share it with your team.

Wrong mutation is removed once you click the "Remove killed mutations" button

Reproduction instructions

Open the Visual Studio 2019
Mutate the whole solution using "Testura.muation" extension
Run all mutations and then immediately click the "Stop mutation run" button
Once the mutations are cancelled, click the "Remove killed mutation" button
==> Only the last mutation is removed (see the video attached below).

MissingManifestResourceException: Could not find any resources appropriate for the specified culture or the neutral culture.

Hi,

I'm trying out the console application on one of our projects at work. However, I'm encountering a MissingManifestResourceException while the baseline is being run.

When I run the tests through dotnet test the tests all pass.

The binary resource is embedded in a separate module but during its compilation it seems to be skipped if I'm correct:

<EmbeddedResource Update="Resources\Messages.resx">
  <Generator>PublicResXFileCodeGenerator</Generator>
  <LastGenOutput>Messages.Designer.cs</LastGenOutput>
</EmbeddedResource>

In the following method, the Include attribute is not present for this resx resource.

        private IList<ResourceDescription> GetEmbeddedResources(string assemblyName, string projectPath)
        {
            var resources = new List<ResourceDescription>();
            var doc = XDocument.Load(projectPath);
            var embeddedResources = doc.Descendants().Where(d => d.Name.LocalName.Equals("EmbeddedResource", StringComparison.InvariantCultureIgnoreCase));
            foreach (var embeddedResource in embeddedResources)
            {
                var paths = embeddedResource.Attribute("Include")?.Value;
                if (paths == null)
                {
                    continue;
                }

                foreach (var path in paths.Split(';'))
                {
                    var pathFixed = path.Split('\\');
                    var resourcePath = Path.Combine(Path.GetDirectoryName(projectPath), path);

                    resources.Add(new ResourceDescription(
                        $"{assemblyName}.{string.Join(".", pathFixed)}",
                        () => File.OpenRead(resourcePath),
                        true));
                }
            }

            return resources;
        }

Which means it seems to be skipping this embedded resource while emitting the compilation into the baseline directory.

The "Testura mutation explorer" window cannot be closed after reopening the Visual Studio

Reproduction instructions

Open the Visual Studio 2019
Mutate the whole solution
Close the Visual Studio 2019
Open the Visual Studio 2019
Click on the "X" icon for the "Testura mutation explorer" window
==> The window does not close (see the video attached).

InvalidOperationException: Cannot modify an evaluated object originating in an imported file

Hello,

I was trying out Unima on the CsvHelper's Test projects ( https://github.com/JoshClose/CsvHelper ) but bumped into the issue described in the title.

Sorry if this is something obvious I'm missing

Here is the contents of the log file:

2019-06-08 16:14:38,467: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Opening project at C:\Users\Eigenaar\Desktop\Test.json
2019-06-08 16:14:38,493: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Loading configuration: {
    "SolutionPath": "C:\\Users\\Eigenaar\\Desktop\\Computer Science\\Projects\\CsvHelper\\src\\CsvHelper.sln",
    "TestProjects": [
        "CsvHelper(netstandard2.0)",
        "CsvHelper(net45)",
        "CsvHelper.Tests",
        "CsvHelper.Performance"
    ],
    "IgnoredProjects": [],
    "Filter": null,
    "NumberOfTestRunInstances": 3,
    "BuildConfiguration": "debug",
    "MaxTestTimeMin": 5,
    "TestRunner": "DotNet",
    "DotNetPath": null,
    "MutationRunLoggers": null,
    "Mutators": []
}

2019-06-08 16:14:38,670: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Loading mutators..
2019-06-08 16:14:38,671: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: ..did not find any mutators in config so loading default ones.
2019-06-08 16:14:39,658: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Opening solution..
2019-06-08 16:19:26,727: Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler: Failed to open project
System.InvalidOperationException: Cannot modify an evaluated object originating in an imported file "C:\Users\Eigenaar\.nuget\packages\msbuild.sdk.extras\2.0.0-preview.14\Build\Inference.targets".
   bij Microsoft.Build.Shared.ErrorUtilities.ThrowInvalidOperation(String resourceName, Object[] args)
   bij Microsoft.Build.Shared.ErrorUtilities.VerifyThrowInvalidOperation(Boolean condition, String resourceName, Object arg0)
   bij Microsoft.Build.Evaluation.Project.VerifyThrowInvalidOperationNotImported(ProjectRootElement otherXml)
   bij Microsoft.Build.Evaluation.Project.RemoveProperty(ProjectProperty property)
   bij Microsoft.CodeAnalysis.MSBuild.ProjectFile.<GetProjectFileInfosAsync>d__16.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.CodeAnalysis.MSBuild.MSBuildProjectLoader.Worker.<DoOperationAndReportProgressAsync>d__17`1.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.CodeAnalysis.MSBuild.MSBuildProjectLoader.Worker.<LoadProjectFileInfosAsync>d__19.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.CodeAnalysis.MSBuild.MSBuildProjectLoader.Worker.<LoadProjectInfosFromPathAsync>d__20.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.CodeAnalysis.MSBuild.MSBuildProjectLoader.Worker.<LoadAsync>d__18.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.CodeAnalysis.MSBuild.MSBuildProjectLoader.<LoadSolutionInfoAsync>d__22.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.<OpenSolutionAsync>d__23.MoveNext()
--- Einde van stacktracering vanaf vorige locatie waar uitzondering is opgetreden ---
   bij System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bij System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   bij Unima.Application.Commands.Project.OpenProject.OpenProjectCommandHandler.<Handle>d__2.MoveNext() in C:\Users\Eigenaar\Desktop\Computer Science\Projects\Unima\src\Unima.Application\Commands\Project\OpenProject\OpenProjectCommandHandler.cs:regel 68

Could not find/load any of the following assemblies

Hello again :)

Could you tell me more what this error is about? It happens to me when I changed xunit test's project framework from core to .net framework (4.5).
If I understand correctly I have to change "Test framework" in config to "xunit". Then I get this error info. Changing back to "DotNet" doesn't help, only when I change test's project framework back to core everything is fine.

Wyjątek nieobsłużony = exception not handled
w = in
wiersz = line

2020-04-16 12:40:36,687[INFO]: Error from test client - WyjĄtek nieobs�uľony: System.InvalidOperationException: Could not find/load any of the following assemblies: xunit.execution.desktop.dll w Xunit.Xunit2Discoverer.GetExecutionAssemblyFileName(AppDomainSupport appDomainSupport, String basePath) w C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\v2\Xunit2Discoverer.cs:wiersz 235 w Xunit.XunitFrontController.CreateInnerController() w C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:wiersz 136 w Xunit.XunitFrontController.EnsureInitialized() w C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:wiersz 163 w Xunit.XunitFrontController.Find(Boolean includeSourceInformation, IMessageSink messageSink, ITestFrameworkDiscoveryOptions discoveryOptions) w C:\Dev\xunit\xunit\src\xunit.runner.utility\Frameworks\XunitFrontController.cs:wiersz 173 w Xunit.Runners.AssemblyRunner.<>c__DisplayClass63_0.b__0() w C:\Dev\xunit\xunit\src\xunit.runner.utility\Runners\AssemblyRunner.cs:wiersz 249 w Xunit.Sdk.XunitWorkerThread.<>c.b__4_0(Object _) w C:\Dev\xunit\xunit\src\common\XunitWorkerThread.cs:wiersz 92 w System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state) w System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) w System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) w System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() w System.Threading.ThreadPoolWorkQueue.Dispatch() w System.Threading._ThreadPoolWaitCallback.PerformWaitCallback().

The "Number of parallel tests" slider is not clearely visible in the "Testura mutation config" window

Unknown error for all mutations

Hey guys, I'm excited, this tool looks great, but I can't figure out one thing, maybe you can help me with that. Mutations look good, but everytime I try to run all those mutations, they all have "unknown error while running". I map this all properly (I think), so it's probably not the issue.
I've created simple example that have the same problems:
SampleProject.zip

Used test framework: xunit
If you need more information please let me know.

Once you mutate the whole solution, "Run survived mutations" and "Remove killed mutations" button should be disabled (clicking on them nothing happens)

Reproduction instructions

Open the Visual Studio 2019
Mutate the whole solution using Testura.mutation extension
After the mutation has finished, click the "Run survived mutations" and "Remove killed mutations" button
==> Nothing happens (see the video attached below).

The "Highlight mutations in code" menu disappears from the "Testura mutation explorer" once you close the Visual Studio while "Testura mutation explorer" is open

Reproduction instructions

Open the Visual Studio 2019
Mutate a file or a whole solution
Once the mutation has finished and the "Testura mutation explorer" window is open, close the Visual Studio 2019
Open the Visual Studio 2019
==> The "..." button with "Highlight mutations in code" menu disappears (see the video attached below).

When you click the "Run all mutations" button and then immediately click the "Stop mutation run" button, 4 mutations are cancelled instead of the 3 ones that were running

Reproduction instructions

Open the Visual Studio 2019
Mutate the whole solution using "Testura.muation" extension
Run all mutations and then immediately click the "Stop mutation run" button
==> 4 mutations are cancelled instead of the 3 ones that were running (see the video attached below).

testura / testura.mutation Goto Github PK

testura.mutation's People

Contributors

Stargazers

Watchers

Forkers

testura.mutation's Issues

Possible Localization Bug

Incorrect line numbers

Reproduction instructions

Reproduction instructions

Reproduction instructions

Reproduction instructions

Reproduction instructions

Recommend Projects

Recommend Topics

Recommend Org