This is a guest post by Cedd Burge, Software Developer Lead at RES.

CodeCov visualises code coverage and can enforce standards via GitHub and AppVeyor. More information is available on CodeCov.io.

This post is written from the point of view of someone (me) who is already proficient in C# and unit testing, but was new to AppVeyor and integrating CodeCov with GitHub.

It contains from scratch steps to run CodeCov test coverage on a sample C# project using GitHub, AppVeyor, OpenCover and XUnit. You can look at the repo I created to test this post if you get stuck.

Create a new GitHub Repo

If you are new to GitHub, see this getting started guide, otherwise simply create a new repo (YourRepositoryName from now on) and git clone it somewhere convenient.

Create a new System Under Test (SUT) project and a new Test project

In my version of Visual Studio (Community 2015), you can do this by clicking on “File - New - Project” on the main menu, then “Class Library” from “Templates - Visual C#”. Give it a interesting name, which I will assume to be YourSUTProjectName for the rest of this post.

Repeat this for the test project, which I will assume to be YourTestProjectName.

Add the xunit, xunit.runner.visualstudio and xunit.runner.console nuget packages to your test project.

Add an XUnit test to YourTestProjectName and the corresponding thing under test to YourSUTProjectName, or copy and paste mine.

Run this test to make sure it passes.

Setup AppVeyor Integration

You will need to link an AppVeyor account to your GitHub one, so let’s do that:

  • Navigate to your repo in GitHub
  • Click “Settings” on the repo
  • Click “Integrations and services”
  • Click “Browse Directory”
  • Click “AppVeyor”
  • Click “Configure”
  • Click “Grant Access”

Now Log in to AppVeyor.com, probably using your GitHub account

  • Click “Projects”
  • Click “New Project”
  • Choose YourRepositoryName and click “Add”

Sign up with CodeCov

  • Go to https://codecov.io/ and click “Sign up with GitHub”
  • Click “Add new repository” and choose YourRepositoryName.
  • Make a note of the Upload token (YourUploadToken from now on)

Run Coverage Analysis Locally

When working with AppVeyor, it always makes sense to test on your own computer first. The feedback is immediate and you iterate very quickly. It takes a lot longer to modify the appveyor.yml file, push it and wait for a build to go through. Also, if it works locally but doesn’t work on AppVeyor, you know the problem is a configuration difference between your computer and the AppVeyor environment (eg a different version of msbuild).

However, this step is not required, so skip to “Run Coverage Analysis on AppVeyor” if you wish.

CodeCov.io is a tool for visualising and integrating coverage data, which we need to create. We will use OpenCover to do this.

Analyse

Add the OpenCover nuget package to the solution (which will install OpenCover.Console.exe, probably in packages\OpenCover.4.6.519\tools) and then run the following in a command window.

  • YourTestDLL is the relative path to the dll for your test project, eg. Tests\bin\Debug\tests.dll
  • -register[:user] asks OpenCover to register the code coverage profiler under HKEY_CURRENT_USER, which doesn’t require the user to have administrative permissions.
  • -target: asks OpenCover to run XUnit
  • -targetargs: are arguments that OpenCover passes to XUnit
  • -output: is the file to contain the coverage results
  • YourSUTNamepace, YourTestNamespace are the namespaces for your Test and SystemUnderTest projects. If these filters aren’t specified OpenCover will try and analyse everything, including the XUnit dlls.
packages\OpenCover.4.6.519\tools\OpenCover.Console.exe -register:user -target:"packages\xunit.runner.console.2.2.0\tools\xunit.console.x86.exe" -targetargs:"YourTestDLL -noshadow" -output:".\coverage.xml" -filter:"+[YourSUTNamepace*]* -[YourTestNamespace*]*"

Upload

Run the following on a PowerShell command line. This talks to CodeCov and downloads a bash script to upload the coverage.

Invoke-WebRequest -Uri 'https://codecov.io/bash' -OutFile codecov.sh

Then you can run the following in a Bash command line (You can use Git Bash on Windows). YourUploadToken is the CodeCov token that you took a note of earlier, or available on the project settings page on CodeCov (eg https://codecov.io/gh/YourGitHubUserName/YourRepositoryName/settings)

codecov.sh -f "coverage.xml" -t YourUploadToken

The output will show a url with the results (eg https://codecov.io/gh/YourGitHubUserName/YourRepositoryName)

Run Coverage Analysis on AppVeyor

Now that the coverage upload is working locally, we can run it on AppVeyor.

Add and commit an appveyor.yml file to the root of the repository as below.

  • YourSolution is the relative path of the .sln file to build (eg codecov-on-appveyor.sln)
  • YourUploadToken is not required if you are uploading from a public repo with AppVeyor, but is included in case you are using a private repo.
  • %xunit20%. XUnit is installed on the AppVeyor build environment by default, and AppVeyor provide the %xunit20% environment variable for the installation path.
before_build:
 - nuget restore
build_script:
 - msbuild /verbosity:quiet "YourSolution"
test_script:
 - .\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe -register:user -target:"%xunit20%\xunit.console.x86.exe" -targetargs:"YourTestDll -noshadow" -output:"coverage.xml" -filter:"+[YourSUTNamepace*]* -[YourTestNamespace*]*"
after_test:
  - ps: |
      $env:PATH = 'C:\msys64\usr\bin;' + $env:PATH
      Invoke-WebRequest -Uri 'https://codecov.io/bash' -OutFile codecov.sh
      bash codecov.sh -f "coverage.xml" -t YourUploadToken

Add Coverage Graphics to the Repository

There are various badges and graphs available. Click on your project in CodeCov, then “Settings” and “Badge” (eg https://codecov.io/gh/YourGitHubUserName/YourRepositoryName/settings/badge) to see what’s available.

Copy and paste a code snippet from this in to your README.md, such as this one Code Coverage Badge.

You can look at my readme for example badges

Integrate CodeCov with Pull Requests

CodeCov automatically integrates with GitHub pull requests (as long as you signed up to CodeCov via your GitHub account) which you can see on this pull request.

It will show some statistics, such as the increase or decrease in coverage, and by default will fail the build if coverage decreases. You can configure this with a ‘codecov.yml’ file in the root of your repository.

Install the browser plugin

There is a CodeCov browser plugin, for all reputable browsers, that adds coverage when browsing GitHub.com. Code is highlighted in red / green, and a coverage percentage is shown.

Conclusions

It can be difficult keeping control of test coverage for a project, especially with distributed and transient team structures. The combination of GitHub, AppVeyor OpenCover and CodeCov make it very easy to visualise the coverage, and allow you to enforce standards to ensure that it improves over time.

Best regards,
Cedd Burge

Follow Cedd on Twitter: @cuddlyburger
Follow AppVeyor on Twitter: @appveyor