Add possibility to run coverlet in standalone mode

[jakubch1]

It would be great to run coverlet without any tests and collect code coverage for application. What I mean here is to for example having console application, instrument it and then start it (without any vstest or anything). When application finishes collect code coverage.

For example:
Let's assume we have app.exe and references of it ref1.dll and ref2.dll
It would be good to be able to

1. instrument all above through command line for example:
   coverlet instrument app.exe ref1.dll ref2.dll
2. run application
   .\app.exe
3. get code coverage from application run.

It could also work in such way that during application execution it will be "logging" all hits to file:line (it could send it to some "monitor")
And after the process finishes it could generate code coverage report based on such data (by another coverlet produceReport data.txt). Such approach would also give possibility to get coverage report for particular time frame without need of process termination.

Such feature will give possibility to collect code coverage for:

1. backend services when tests are just communicating with it through http etc.
2. any other than vstest test runner

[MarcoRossignoli]

Some notes:

It could also work in such way that during application execution it will be "logging" all hits to file:line (it could send it to some "monitor")

Maybe this is not doable, in past coverlet registered hits on file directly for every call, but it was very slow so we moved to in ram booking in a int array and at the end we flush to disk, so I think that send coverage for every hits realtime won't work well, think for example a loop that would send n hits for every cycle, it was slow on file, I think will be worst with network, BTW I don't have numbers and I'm speculating, maybe tcp buffering works well also in this case.
Maybe some bulk send every some time, when I talk about hits file imagine a simple array of int https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L22

This is the place where instrumentation start, for every module(found or filtered) we generate an InstrumenterResult with all needed data(branches/lines) to calculate coverage from hits. This array is the results of instrumentation in you sample app.exe ref1.dll ref2.dll list of 3 InstrumenterResult object that we need to "save" somewhere at the moment we keep in memory for collectors and .net tool and to file for msbuild.

instrument all above through command line for example:

Some ideas

1. Does it make sense build a coverlet command or maybe it's more useful expose core feature as package and let developers to interact with engine(similar to this experiment, here the exposed interface https://github.com/tonerdo/coverlet/pull/628/files#diff-743457133023fd26c409505936137f4f) and build what they want to it, I mean your scenario address your specific issue.

Take a look at live sample(in PR) https://twitter.com/MarcoRossignoli/status/1201546945589841922 in this case first command instrument dll and generates the instrumentation result and the second command take the instrumentation result and use it to get realtime coverage, because they need to know the mappings.

2. Another idea could be add a new tcp mode tracker, where during instrumentation an address is passed ip:port and bulk parameter(how many time send data) and tracker simply send data to that address realtime and on process exit.

coverlet instrumentation --filter globbingfilter --type tcp --tcpAddress 127.0.0.1 --tcpPort 1000 --frequency 100 --instrumentationResultFilePath file

frequency: send every new records https://github.com/tonerdo/coverlet/blob/528a769c9ab98762935490372e40720ed7219618/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L52

And the protocol cloud be

• Connect() on first tracker access
• RecordedHits(json) <- collection libName/hitCandidate+hitcount
• Disconnect() process exit

Also we need to expose an object model to allow the accounting on server side, pseudo code

Instrumentation inst = new Instrumentation(instrumentationResultFilePath);
inst.ReadCoverage(received json);
inst.ShowSomethingSomeWhere();

Or simply send a binary format that server only need to append to some stream(file, memory) something similar to a database transaction log and another command coverlet reports hitfilesavedbyserver can handle and after pass to generate report part of tool as normal hit file.
Or again...simply send all hits file every some frequency.
Also in this case we should expose some "server" classes to take the stream and write somewhere.

cc: @tonerdo @petli

[MarcoRossignoli]

@jakubch1 one question, it's not clear to me how do you see "the server side part" that receive the hits:

Is implemented in coverlet like a command that stay up on a port until some signal?
Is a model exposed by coverlet and used by server?

[jakubch1]

@MarcoRossignoli I will try to create something based on this what you said and I will see if this works. Maybe you are right that we don't need to have this inside coverlet. Basically people can develop what they need. Any other user asked for such thing?

During the call you said that sometimes there is issue that report is not generated because testhost process is terminated too fast. Do you have any reports for it? In case of vstest it could work in such way that array with hits could be inside shared memory (accessible by both data collector and testhost) and when data collector gets event that all tests are done it (data collector) could generate report.

[MarcoRossignoli]

@MarcoRossignoli I will try to create something based on this what you said and I will see if this works.

What do you mean?Do you use coverlet as coverage engine or not?

@MarcoRossignoli I will try to create something based on this what you said and I will see if this works. Maybe you are right that we don't need to have this inside coverlet. Basically people can develop what they need. Any other user asked for such thing?

You(vstest old team) and some other users asked for standalone release of coverlet core features #212 (check thumbs up), this is an explorative PR #628

During the call you said that sometimes there is issue that report is not generated because testhost process is terminated too fast.

Main issue #210
VSTest issue microsoft/vstest#1900 (comment)
Some issues https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+rabbitmq+is%3Aclosed
Some fix on other repo SixLabors/ImageSharp#1094

In case of vstest it could work in such way that array with hits could be inside shared memory (accessible by both data collector and testhost) and when data collector gets event that all tests are done it (data collector) could generate report.

We don't have issue with collectors because in-process one is synched with test end events, we have that issue with msbuild integration because it's based on process exit event.
I asked for an increment of timeout microsoft/vstest#2379 to mitigate the issue on msbuild(the most used driver)

This is our in-proc implementation https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L46

Collectors integration is the more resilient way to use coverlet but there are some unsupported features due to vstest platform behaviour:

• netfx support 0% coverage reported on netfx target #705
• show results on terminal Show coverage summary on terminal #681
• automatic merge at the end of the tests MergeWith in dotnet test is not working con vstest integration(collectors) #662 (solution wide)
• Specify deterministic output directory Unable to specify output directory when using VSTest integration? #500 (I opened an issue some days ago Avoid guid at the end of outputDirectory microsoft/vstest#2378)

Aggregate collectors issues https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Adriver-collectors

[jakubch1]

What do you mean?Do you use coverlet as coverage engine or not?

I want to try to build something on top of coverlet to check that it allows to replace stuff correctly with different implementation.

We don't have issue with collectors because in-process one is synched with test end events, we have that issue with msbuild integration because it's based on process exit event.
I asked for an increment of timeout microsoft/vstest#2379 to mitigate the issue on msbuild(the most used driver)

I don't understand this. Msbuild integration is internally running vstest so what is the issue?

This is our in-proc implementation https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L46

If I understand correctly this works only for .net Core. So suggestion to use shared memory between OutProcCollector and testhost is still valid. I was checking other threads regarding this issue and Mayank was also suggesting this. By using shared memory and generating report inside data collector you could maybe even not need Coverlet nuget package reference inside test projects. We could make call and discuss this.

* automatic merge at the end of the tests #662 (solution wide)

I'm planning to work on this feature in near future :) The initial idea is to create something as session inside vstest.console.exe (process will run in client mode like in VS) and vstest.console will gather all coverage reports from test projects executions. On session finish it will group by coverage reports based on data collector uri. Then data collectors will again be invoked to merge reports. We will let you know when we agree on design.

[MarcoRossignoli]

I don't understand this. Msbuild integration is internally running vstest so what is the issue?

The issue is that with msbuild version the flush of hits to file is done on process exists and with collectors is done by in-proc collector, so if during process exits vstest plat kill host process msbuild version doesn't flush file correctly.

If I understand correctly this works only for .net Core

At the moment yes...because I don't know why collectors wasn't compiled as netstandard2.0 by @vagisha-nidhi , but will support also netfx if we'll compile for netstandard.
I did same question on vstest repo, can we compile collectors for netstandard?Are netstandard2.0 collectors supported by vstest plat? https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/coverlet.collector.csproj#L3
In that case collectors works also on netfx.

We could make call and discuss this.

Yep a good idea.

I'm planning to work on this feature in near future :)

This is great!

[jakubch1]

At the moment yes...because I don't know why collectors wasn't compiled as netstandard2.0 by @vagisha-nidhi , but will support also netfx if we'll compile for netstandard.
I did same question on vstest repo, can we compile collectors for netstandard?Are netstandard2.0 collectors supported by vstest plat? https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/coverlet.collector.csproj#L3
In that case collectors works also on netfx.

We had internal discussion with code coverage experts regarding this. We believe coverlet should use shared memory between testhost and out proc data collector and generate report inside out proc data collector. This is approach which is used in several code coverage solutions that we know. It fixes all issues with crush/fast kill of test host process.

[MarcoRossignoli]

We had internal discussion with code coverage experts regarding this. We believe coverlet should use shared memory between testhost and out proc data collector and generate report inside out proc data collector. This is approach which is used in several code coverage solutions that we know. It fixes all issues with crush/fast kill of test host process.

Maybe I wasn't clear, we don't have any problem with collectors version of coverlet, it works as expected.

The problem is with msbuild version, where there is no control on process shutdown. I try to explain better the difference between msbuild and collectors(I call "driver" the way to use coverlet runtime, for instance msbuild is a driver, collectors is a driver) run:

1. msbuild

if you want to use msbuild command you have to add coverlet.msbuild package to your test project and you can invoke coverage with command:

dotnet test /p:CollectCoverage=true

This is the most used way https://nugettrends.com/packages?months=12&ids=coverlet.msbuild today.

How it works:

1. msbuild task is injected on build pipeline https://github.com/tonerdo/coverlet/blob/master/src/coverlet.msbuild.tasks/coverlet.msbuild.targets#L49, here we select the dll to instrument and do instrumentation.
2. Test plat span testhost.exe process and start tests. Thanks to injected tracker class we collect in data memory(array of int).
3. Tests end and process testhost.exe will closes and on process exits event we flush in memory hit file to disk https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L76
4. When tests end another msbuild task will run after target VSTest https://github.com/tonerdo/coverlet/blob/master/src/coverlet.msbuild.tasks/coverlet.msbuild.targets#L68 and in this task we load hits file and it generates file report.

Here the "fast kill" issue on point 3 does not allow to process exit to flush file correctly and we don't have any control on where msbuild task will be hosted so we cannot simply open a shared memory, but to fix this issue with your idea we should span our new process that communicates with testhost and in some way read that data on msbuild task after VSTest target. This is a huge change.

2. Collectors

if you want to use collectors integration you have to add coverlet.collector package to your test project and you can invoke coverage with command:

dotnet test --collect:"XPlat Code Coverage"

How it works:

1. When test platform see a collector named XPlat Code Coverage injects out-of-process and in-process coverlet collectors:

https://github.com/microsoft/vstest/blob/master/src/vstest.console/Processors/CollectArgumentProcessor.cs#L176 here in-process injection
out-of-process collector is injected throught friendly name https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L20

2. VStest plat span throught vstest.console process span 2 process datacollector and testhost inside datacollector hosts out-of-process data collector and inside testhost hosts in-process data collector.
3. OnSessionStart out-of-process data collector instruments dlls https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L140
4. Tests run, thanks to injected tracker class we collect in data memory(array of int).
5. On tests end in-process data collector flush hit file, here is the key https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L57, traker code https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L76
6. Test host process closes.
7. OnSessionEnd out of process collector read hit file and generates report file https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L154

So in case of collectors integration we have the guarantee of hit file flush because in-process collector listen test end event and force flush and not rely on process exits https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L57

So there is NO issue with collectors but there is issue with msbuild driver.

If it's not clear we should talk via chat or call, it's hard to explain in a issue post.

[jakubch1]

When we can have a meeting? I can do it today after 7PM or next week Wednesday or Friday after 6PM.

[MarcoRossignoli]

@jakubch1 sorry I read now, if you want I'm here...or ok for Wednesday after 6PM.

[bignis]

I'll chime in with my use case that I think this Issue fits well with.

I maintain a web-based business application that doesn't have very many unit tests, but has plenty of automated tests through Selenium in a web browser. I want to assess how well these Selenium web tests cover my business logic.

If I could instrument my binaries as @jakubch1 described, I could run the Selenium tests in a browser, look at the coverage report afterward, and inform my QA team of where additional Selenium tests are needed to achieve better coverage. I wouldn't need real-time reporting for my use case.

[petli]

Before going down the shared memory road have a look at this comment from the last time it was tried to see if it can be done in a way that doesn't break System.Private.CoreLib:
#276 (comment)

Edit: I'm now caught up and see that this was already noticed in #808, great!

[MarcoRossignoli]

@jakubch1 I did some attempt with the .NET tool idea and seems to work(with a caveat)
We can leverage .NET tool version to start a normal startup server process instead to dotnet test.
For instance suppose to have a normal asp.net core app in a folder named C:\git\coverletissue\FunctionalTests\RestApi

1. Build app

dotnet build

2. Start dotnet process using coverlet .NET tool driver

C:\git\coverletissue\FunctionalTests
λ coverlet C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --target dotnet --targetargs C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --format cobertura --include-test-assembly --verbosity:detailed
Instrumented module: 'C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll'
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\git\coverletissue\FunctionalTests

This command will start server with instrumented libs.
3) In my sample I used bombardier to generate some traffic, so from another console

C:\git\coverletissue\FunctionalTests
λ C:\Users\Marco\Downloads\bombardier-windows-386.exe https://localhost:5001/WeatherForecast

4. After some tests Gracefully shutdown server

coverlet C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --target dotnet --targetargs C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --format cobertura --include-test-assembly --verbosity:detailed
Instrumented module: 'C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll'
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\git\coverletissue\FunctionalTests
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

Calculating coverage result...
Hit file 'C:\Users\Marco\AppData\Local\Temp\RestApi_6c71a5f3-084d-44a5-9936-7882c8dd8800' deleted
  Generating report 'C:\git\coverletissue\FunctionalTests\coverage.cobertura.xml'
+---------+--------+--------+--------+
| Module  | Line   | Branch | Method |
+---------+--------+--------+--------+
| RestApi | 52.72% | 50%    | 38.46% |
+---------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 52.72% | 50%    | 38.46% |
+---------+--------+--------+--------+
| Average | 52.72% | 50%    | 38.46% |
+---------+--------+--------+--------+

Caveat

The only cavet is that server process MUST shutdown in a gracefully way, because today coverlet relies on process exit event to store the hits, so a process kill won't work.
In this tests I've updated a bit startup on purpouse

public static void Main(string[] args)
{
       IHost host = CreateHostBuilder(args).Build();
       var appLife = host.Services.GetRequiredService<IHostApplicationLifetime>();
        _ = Task.Run(() => host.Run());
       Thread.Sleep(TimeSpan.FromMinutes(1.5));
       appLife.StopApplication();
}

I did a simple test with a simple asp.net core app created with scaffolding but I don't see any reason why shouldn't work with other type of app.

[wilbe2013]

Hi everyone.

We try to following some instructions were discussed here, but we don't got good results.

We need get coverage of an assembly .exe(Window form .Net framework 4.6.1 ), in a test project with appium(windows driver), is a UI test project, but we don't know how get coverage of that assembly .exe(we launch .exe from that test)

That Windows form app have reference to other assemblies(.dll, someones in .netframework and others in c++) that we need get the coverage too.

Do you can give us some idea if is possible get coverage with Coverlet in this enviroment?

Thanks in advance.

[MarcoRossignoli]

That Windows form app have reference to other assemblies(.dll, someones in .netframework and others in c++) that we need get the coverage too.

Coverlet supports only IL ECMA-335 (.NET) code coverage, we don't support unmanaged environment.

[wilbe2013]

HI @MarcoRossignoli , that mean even if i exclude that assembly(.dll in c++) i can't get the coverage with the others assemblies?

[MarcoRossignoli]

No for .NET assemblies should work, are you sure that process after test closes gracefully and not with a kill process?coverlet store data in a processExit event, if process closes without that event there is a possibility of missing data coverage, are you able to provide a repro?

[AnthonyIacono]

@jakubch1 I did some attempt with the .NET tool idea and seems to work(with a caveat) We can leverage .NET tool version to start a normal startup server process instead to dotnet test. For instance suppose to have a normal asp.net core app in a folder named C:\git\coverletissue\FunctionalTests\RestApi

1. Build app

dotnet build

2. Start dotnet process using coverlet .NET tool driver

C:\git\coverletissue\FunctionalTests
λ coverlet C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --target dotnet --targetargs C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --format cobertura --include-test-assembly --verbosity:detailed
Instrumented module: 'C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll'
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\git\coverletissue\FunctionalTests

This command will start server with instrumented libs. 3) In my sample I used bombardier to generate some traffic, so from another console

C:\git\coverletissue\FunctionalTests
λ C:\Users\Marco\Downloads\bombardier-windows-386.exe https://localhost:5001/WeatherForecast

4. After some tests Gracefully shutdown server

coverlet C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --target dotnet --targetargs C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll --format cobertura --include-test-assembly --verbosity:detailed
Instrumented module: 'C:\git\coverletissue\FunctionalTests\RestApi\bin\Debug\netcoreapp3.1\RestApi.dll'
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\git\coverletissue\FunctionalTests
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

Calculating coverage result...
Hit file 'C:\Users\Marco\AppData\Local\Temp\RestApi_6c71a5f3-084d-44a5-9936-7882c8dd8800' deleted
  Generating report 'C:\git\coverletissue\FunctionalTests\coverage.cobertura.xml'
+---------+--------+--------+--------+
| Module  | Line   | Branch | Method |
+---------+--------+--------+--------+
| RestApi | 52.72% | 50%    | 38.46% |
+---------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 52.72% | 50%    | 38.46% |
+---------+--------+--------+--------+
| Average | 52.72% | 50%    | 38.46% |
+---------+--------+--------+--------+

Caveat

The only cavet is that server process MUST shutdown in a gracefully way, because today coverlet relies on process exit event to store the hits, so a process kill won't work. In this tests I've updated a bit startup on purpouse

public static void Main(string[] args)
{
       IHost host = CreateHostBuilder(args).Build();
       var appLife = host.Services.GetRequiredService<IHostApplicationLifetime>();
        _ = Task.Run(() => host.Run());
       Thread.Sleep(TimeSpan.FromMinutes(1.5));
       appLife.StopApplication();
}

I did a simple test with a simple asp.net core app created with scaffolding but I don't see any reason why shouldn't work with other type of app.

I tried to follow these steps, but no matter what I do (even with graceful shutdown implemented), I get these errors and a 0% coverage file:

Calculating coverage result...
Hits file:'/var/folders/b_/c5vfxz3j1pzfrrh7nhk6sm_r0000gn/T/Common_58a56b95-a7b3-4ff7-a29b-77a157cfc953' not found for module: 'Common'
Hits file:'/var/folders/b_/c5vfxz3j1pzfrrh7nhk6sm_r0000gn/T/Api_58a56b95-a7b3-4ff7-a29b-77a157cfc953' not found for module: 'Api'
Hits file:'/var/folders/b_/c5vfxz3j1pzfrrh7nhk6sm_r0000gn/T/Data_58a56b95-a7b3-4ff7-a29b-77a157cfc953' not found for module: 'Data'
Hits file:'/var/folders/b_/c5vfxz3j1pzfrrh7nhk6sm_r0000gn/T/Shared_58a56b95-a7b3-4ff7-a29b-77a157cfc953' not found for module: 'Shared'

Do you have any ideas? I added the coverlet.collectors package, but it didn't help.

[iSuper-Ray]

Is it possible to add PID as a target?
My Team wants to get coverage for E2E Tests, but we have a legacy .NET Framework 4.8 application that runs on IIS.
Currently proposed parameters won't allow us to use coverlet for this scenario.

[petli]

@iSuper-Ray Sorry, it's not possible to run against an already executing process, since coverlet works by instrumenting the binaries before they are run.

[iskiselev]

I'd like to try implement that, but for it I'd like to discuss what should be the final shape.

1. Should it be implemented in coverlet.console or should it be separate tool? I'd tend to say it should be separate tool - we need to support multiple sub-commands, and adding it into existing coverlet.console will be compatibility issue. Later existing coverlet.console may be integrated in the new tool. How to name this new tool? Should it be dotnet tool?
2. Suggested commands:

• coverlet instrument assembly1.dll assembly2.dll assembly3.dll
  • Will instrument only assemblies passed as arguments. Should it also support folders? Probably not. Will create mapping file (InstrumenterResult) in current folder (customizable).
• coverlet process HitPath -i InstrumenterResultPath
  • Will calculate code coverage, reading hit files in HitPath. Will require InstrumenterResultPath if it is not located in HitPath.

3. Currently coverlet creates hit file in temp location. That should be customizable to get hit files and use it with process command. I'd suggest read some environment variable for it, and if it was defined, use folder specified in it to store hit files. That folder may support some expansion parameters as %p for PID. I'm looking in Clang code coverage and there LLVM_PROFILE_FILE, when suggesting it. It can be COVERLET_COVERAGE_PATH.
4. Later, coverlet instrument may allow user specify default hit file path.

Is it sounds as plan that I can start work on - or should something here be changed?

[MarcoRossignoli]

Should it be implemented in coverlet.console or should it be separate tool? I'd tend to say it should be separate tool - we need to support multiple sub-commands, and adding it into existing coverlet.console will be compatibility issue. Later existing coverlet.console may be integrated in the new tool. How to name this new tool? Should it be dotnet tool?

I like the idea to have everything inside the coverlet.console and avoid another tool and I think it's ok have a breaking change, we can announce.

we could move the "today" commands under the verb run

coverlet run...

So the fix for current usage will be add run.

Suggested commands:

We should find a good verb for the standalone mode

coverlet instrument...
coverlet standalone --instrument...
coverlet [something] --param1 ... --param2 ...

I like the idea to have a "general" verb name and after specify the parameters so it's possible in future add new complex features on top of the "off line" instrumentation

Should it also support folders?

Yes, usually we "try to" instrument all dlls close to the test container so we should have include/exclude dll and folders and exclude win on include.

Will create mapping file (InstrumenterResult) in current folder (customizable).

It's ok

Currently coverlet creates hit file in temp location. That should be customizable to get hit files and use it with process command.

We should by default create hint file in the current directory(we'll have one for every dll istrumented) with a chance to override, I expected that the user will instrument/execute/shutdown/zip(hints/InstrumenterResult) and move in some folder to generate the report(or generate it in place).
We can use a parameter during the "instrument" phase...that value will be embedded in the instrumented dll.

---

We have always anyway a small issue with the standalone execution, it will work only if we "gracefully" shutdown the execution or we'll lose the hint(coverlet relies on process exits not a great way, we should move to a second strategy using shared memory).

cc: @daveMueller @petli @jakubch1 @fhnaseer

[jakubch1]

You can now use https://www.nuget.org/packages/dotnet-coverage global tool to collect coverage in this scenario.
Example: https://learn.microsoft.com/dotnet/core/additional-tools/dotnet-coverage#use-the-instrument-and-collect-commands-in-server-mode