How to Benchmark Your .NET Code Using BenchmarkDotNet

by Chad
Published August 31, 2019
Last updated June 26, 2020

Waves

Sometimes as developers we’re faced with making a certain piece of code perform faster. We often need to decide which piece of code performs the best. Benchmarking provides us concrete measurements between different pieces of code, so we can make verifiably correct decisions based on performance. In this post, I’ll introduce the BenchmarkDotNet library. This library simplifies the process of benchmarking our .NET code, and provides a bunch of cool features out of the box.

Getting Started

To begin performing our benchmarks, we’ll start by creating a new C# .NET Core Console app. If you wanted to test code for other runtimes such as .NET Framework, BenchmarkDotNet supports other runtimes. For the full documentation, see BenchmarkDotNet's website.

Creating the Project

Using Visual Studio 2019, we’ll start by creating a new console app using the .NET Core runtime. With the project created, we’re greeted by the standard “Hello World” console program.

using System;

namespace BenchmarkingIntro
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

Importing the BenchmarkDotNet Library

Next, we need to install the latest BenchmarkDotNet Nuget package. This is easily done via the Nuget package manager or the dotnet add package BenchmarkDotNet command.

Setting Up the Benchmarking Class

Now we need to design a benchmark that we’re eventually going to measure. We can start with a simple benchmark that compares the lookup speeds of two types of collections, a List and a HashSet. I’ve designed a simple class that that creates a List and a HashSet that contain the numbers 0-999. I’ve created two methods, both decorated with the [Benchmark] attribute, that perform a lookup using Contains. The [Benchmark] attribute tells BenchmarkDotNet to measure the method as part of the benchmark. I’ve also used the [MinColumn, MaxColumn] attributes on the LookupSpeed class to include both min and max values for test on the benchmark summary. Below is the code:

[MinColumn]
[MaxColumn]
public class LookupSpeed
{
    private const int N = 1000;
    private const int LOOKUP = 250;

    private readonly IList _list;
    private readonly ISet _set;

    public LookupSpeed()
    {
        _list = new List();
        _set = new HashSet();

        for (int i = 0; i < N; i++)
        {
            _list.Add(i);
            _set.Add(i);
        }
    }

    [Benchmark]
    public bool ListLookup() => _list.Contains(LOOKUP);

    [Benchmark]
    public bool SetLookup() => _set.Contains(LOOKUP);
}

Running the Benchmark

In our program’s Main method, we need to add BenchmarkRunner.Run<LookupSpeed>() to tell BenchmarkDotNet to run a benchmark on the LookupSpeed class we created.

static void Main(string[] args)
{
    BenchmarkRunner.Run<LookupSpeed>();
}

Running this code now generates the following summary output in a console window.

// * Summary *

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.17763.678 (1809/October2018Update/Redstone5)
Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.0.100-preview3-010431
  [Host]     : .NET Core 2.2.2 (CoreCLR 4.6.27317.07, CoreFX 4.6.27318.02), 64bit RyuJIT  [AttachedDebugger]
  DefaultJob : .NET Core 2.2.2 (CoreCLR 4.6.27317.07, CoreFX 4.6.27318.02), 64bit RyuJIT


|     Method |       Mean |     Error |    StdDev |        Min |       Max |
|----------- |-----------:|----------:|----------:|-----------:|----------:|
| ListLookup | 147.012 ns | 1.1715 ns | 1.0958 ns | 145.836 ns | 149.28 ns |
|  SetLookup |   9.967 ns | 0.2299 ns | 0.2647 ns |   9.777 ns |  10.57 ns |

// * Warnings *
Environment
  Summary -> Benchmark was executed with attached debugger

// * Legends *
  Mean   : Arithmetic mean of all measurements
  Error  : Half of 99.9% confidence interval
  StdDev : Standard deviation of all measurements
  Min    : Minimum
  Max    : Maximum
  1 ns   : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// ** Remained 0 benchmark(s) to run **
Run time: 00:00:40 (40.3 sec), executed benchmarks: 2

Important Note About Debug Mode

Make sure your console app is set to run in Release mode, not Debug mode. BenchmarkDotNet will complain if you attempt to run in Debug mode as the results are unlikely to be accurate.

Conclusion

While this is kind of a silly example, and a lot of us probably already know that a HashSet has better lookup performance than a List, benchmarking allows us to get a more definitive, clearer picture of this. Oftentimes we’re forced to make a decision to use one piece of code over another based on performance, and benchmarking our code via BenchmarkDotNet provides a simple way to make an informed decision.

Happy coding!

Read Next

Add TypeScript to Your Project image

May 04, 2020 by Chad

Add TypeScript to Your Project

TypeScript helps bring your JavaScript projects sanity by providing strong type checking and all the latest and greatest ECMAScript features. This article will show you how to quickly and easily add TypeScript to your project, old or new.

Read Article
Unit Testing Exceptions in C# image

January 16, 2020 by Chad

Unit Testing Exceptions in C#

Sometimes there are cases where we want to throw a specific exception in our code. When you are writing your tests, how do you account for this? In this article I will work through examples of how to unit test C# code that's expected to throw exceptions.

Read Article
Unit Test Your C# Code Easily with xUnit and TDD image

January 12, 2020 by Chad

Unit Test Your C# Code Easily with xUnit and TDD

Unit testing your C# code has truly never been easier. Today I will introduce how to get up and running and unit testing your C# code in only a matter of seconds with one of the latest testing technologies, xUnit, using a Test Driven Development (TDD) approach.

Read Article