0% coverage reported on netfx target

[saucecontrol]

Migrating this issue over from #578 comments

When running tests locally, I get correct results with netfx target on 32-bit

dotnet test tests\Blake2.Test -c Coverage -s tests\runsettings --logger trx

with the following runsettings:

<RunSettings>
    <RunConfiguration>
        <TargetPlatform>x86</TargetPlatform>
        <DisableAppDomain>true</DisableAppDomain>
    </RunConfiguration>
</RunSettings>

However, if I switch to <TargetPlatform>x64</TargetPlatform>, I get 0% coverage.

On Azure Pipelines, I get 0% coverage under either config.

The netcore targets are correct no matter how I run.

I have tried the coverlet.collector workarounds from https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#2-upgrade-coverletcollector-to-version--100 with no change in results.

My current config can be found here: https://github.com/saucecontrol/Blake2Fast/tree/azure-coverage

And the updated cmdline I'm using with coverlet.collectors is

dotnet test tests\Blake2.Test -c Coverage -s tests\runsettings.collector --logger trx --collect:"XPlat Code Coverage"

[saucecontrol]

@MarcoRossignoli I created my collector config based on an older version of the workaround doc. I see current examples omit the InProcDataCollectionRunSettings element from the config. Is that something that's built in to the newer Microsoft.NET.Test.Sdk versions?

I'm also not understanding the connection between the DataCollectors element in the runsettings and the --collect arg on the cmdline. Is the DataCollectors element defining the collector, or is that supposed to be adding additional settings for a collector that is already recognized by the SDK?

I don't doubt I've missed something in my config, but my recollection is that when I dug into the logs on my previous attempt to solve this, I could see coverlet.collector was being invoked but still failing to collect anything.

[MarcoRossignoli]

I see current examples omit the InProcDataCollectionRunSettings element from the config. Is that something that's built in to the newer Microsoft.NET.Test.Sdk versions?

Yes in-process collector is injected automatically when we use --collect:"XPlat Code Coverage" since .NET Core SDK v2.2.401.

I'm also not understanding the connection between the DataCollectors element in the runsettings and the --collect arg on the cmdline. Is the DataCollectors element defining the collector, or is that supposed to be adding additional settings for a collector that is already recognized by the SDK?

Yes, when you pass --collect:"XPlat Code Coverage" OutOfProc and InProc collectors will be loaded using coverlet.collector.dll you reference. The settings file is needed to specify non default options and they flow to OutOfProc collector.

I don't doubt I've missed something in my config, but my recollection is that when I dug into the logs on my previous attempt to solve this, I could see coverlet.collector was being invoked but still failing to collect anything.

There is a bug in vstest on inproc loading...so we need for now to update to a preview version microsoft/vstest#2221

I cloned and tested your sample and with following command I get coverage

<Project Sdk="Microsoft.NET.Sdk">

	<PropertyGroup>
		<TargetFrameworks>netcoreapp3.0;netcoreapp2.1;net46</TargetFrameworks>
	</PropertyGroup>

	<ItemGroup>
                <!-- To update in future with stable version -->
		<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0-preview-20200116-01" />
		<PackageReference Include="coverlet.collector" Version="1.2.0">
		  <PrivateAssets>all</PrivateAssets>
		  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
		</PackageReference>
		<PackageReference Include="System.Json" Version="4.6.0" />
		<PackageReference Include="System.Runtime.CompilerServices.Unsafe" Version="4.6.0" />
		<PackageReference Include="xunit" Version="2.4.1" />
		<PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" PrivateAssets="all" />
		<DotNetCliToolReference Include="dotnet-xunit" Version="2.3.1" />
	</ItemGroup>

	<ItemGroup>
		<ProjectReference Include="..\..\src\Blake2Fast\Blake2Fast.csproj" />
	</ItemGroup>

	<ItemGroup>
		<EmbeddedResource Include="blake2-kat.json" />
	</ItemGroup>

</Project>

And this is the command line

D:\git\coverletissue\Blake2Fast (master -> origin)
λ dotnet test tests\Blake2.Test --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true

You can enable logging --diag:log.txt and search for [coverlet] and you'll find log for out of process(log.datacollector.20-01-27_20-50-06_02383_4.txt) and in process collector(log.host.20-01-27_20-50-07_67280_4.txt)

Let me know!

[MarcoRossignoli]

@saucecontrol I've updated vstest guide with more details https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md

[saucecontrol]

Thanks again for looking into this and for the doc update. That clears some things up for sure. From what I can see, at worst I had some redundant config values set. I don't see anything different in your example than what I've tried.

However, AFAIK if you don't specify otherwise, the tests run in 32-bit mode, which has been working for me all along on my local machine. I'll give it a shot again, but that config has consistently failed (tests run, but 0 coverage reported) in Azure Pipelines even when the identical config worked locally. When I specify x64 TargetPlatform, I get the same failure locally that I get in CI.

[MarcoRossignoli]

I'll force x64 asap and come back!You're right I didn't do that test.

[saucecontrol]

Cool, let me know. I'm actually not sure if the failure I see locally in x64 has anything in common with the CI failure besides the 0% result; it was just the only way I could get it to fail reliably locally with the same symptom. If you're not able to identify anything, I'll see if I can get some good logs from CI with the current versions of everything since that's the problem I'm actually trying to solve.

[MarcoRossignoli]

I can repro with collectors and it happens only for net46 x64(resolve issue on test host), next days I'll try to understand if happens also with msbuid.
Test branch https://github.com/MarcoRossignoli/Blake2Fast/tree/coverlettest

[MarcoRossignoli]

I did a test and found the issue, is related to coverlet in-proc load failure, vstest plat try to load netcoreapp2.1 lib inside a netfx process and fails

TL;DR;
msbuild/.net tool suffer a know issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test
So we use collector because in that way in-proc coverlet collector class is injected inside test host process and we can persist hit file. In-proc collector is integrated and communicate with vstest platform(datacollector process talk with host process and wait to kill until "they agree").

Now the issue is that in-proc coverlet collector was written with netcoreapp2.1 tfm(I asked to vstest team why, they wrote it) so when vstest plat try to load it a lot of ref failure happens

TpTrace Warning: 0 : 31440, 5, 2020/02/16, 17:39:16.357, 1014902001697, testhost.x86.exe, TestPluginDiscoverer: Failed to get types from assembly 'coverlet.collector, Version=1.2.0.0, Culture=neutral, PublicKeyToken=4beb1ad7f05a1c71'.  Skipping test extension scan for this assembly.  Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos)
TpTrace Warning: 0 : 31440, 5, 2020/02/16, 17:39:16.357, 1014902004338, testhost.x86.exe, LoaderExceptions: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginCache.CurrentDomainAssemblyResolve(Object sender, AssemblyResolveEventArgs args)
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformAssemblyResolver.AssemblyResolverEvent(Object sender, Object eventArgs)
   at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)
...

When this happen and vstest plat kill process we suffer again of know issue above so we cannot persist hit file...and this is randomic, I mean can happen for x64 and not for x86 and vice-versa.

TpTrace Verbose: 0 : 25296, 1, 2020/02/16, 17:39:17.847, 1014916892926, datacollector.dll, [coverlet]Hits file:'...\SauceControl.Blake2Fast_225870d7-66fa-4f8a-91e3-e11c8ef43af2' not found for module: 'SauceControl.Blake2Fast'

I did some test with a collector compiled for netstandard2.0+vstest master compiled plat and it works...btw I found also a past statement of vstest team #466 (comment) so the past idea was support only netcoreapp, but now that we have an unsolvable issue with automatic process kill(above know issue) maybe we should change the idea...or abandon netfx(try msbuild and if you hit issue above...no solution)
I'll wait vstest team thinking.

[MarcoRossignoli]

closed in #970

[saucecontrol]

@MarcoRossignoli I'm just now trying to test that fix, but the latest nightly build I see on MyGet is from August. Are those published somewhere else now?

[MarcoRossignoli]

Yep we're a bit in trouble with nightly we're moving to AzDo but I'm a bit busy at the moment so sorry, for now you can try on my personal unofficial feeds I use for tmp fixing/test https://f.feedz.io/marcorossignoli/coverletunofficial/nuget/index.json

Last version should work on netfx.

Let me know!
If you're in trouble I can load tmp version for your need. Sorry again.

[saucecontrol]

It's no problem at all. I just wanted to try it out before 3.0 lands since I had a free minute. Tried the private feed 3.0.0-preview.1, and it's working great now. Thanks much!

[MarcoRossignoli]

Glad to hear that!