Running Celeriac on Unit Tests

Introduction

A great way to produce a trace is to run Celeriac in conjunction with a unit test suite. Celeriac is compatible with all testing frameworks since Celeriac instruments the program under test and not the test suite. We've run Celeriac using the Visual Studio, MSTest, xUnit.net, and NUnit frameworks as well as numerous mocking frameworks such as RhinoMock.

The general approach is to create an instrumented copy of the assembly under test by running Celeriac with the --save-program flag. To run the assembly that isn't part of the test suite (i.e. your program or library), you will then either replace the original assembly with the instrumented copy, or modify the test suite setup to use the instrumented assembly. The test suite can then be run as usual, and the Celeriac output files will be generated as they normally would during offline tracing.

From the Command Line or External Testing GUI

Suppose that the assembly under test is Bank.dll, and the test suite is BankTest.dll . To run the test suite:

1. Create an instrumented copy of the assembly under test (see the Getting Started guide).
2. Replace the original assembly under test with the instrumented assembly
3. Run the test suite as normal. For example, when using MSTest:

   MSTest.exe /testcontainer:BankTest.dll /testsettings:Local.testsettings

Some testing frameworks create a new application domain for each test by default. To prevent Celeriac from creating a new trace file for each test, configure the test runner to run the tests in a single domain.

MSTest Configuration

Purity Files

MSTest copies files to another directory in order to perform testing. Therefore, when using a purity file, you must specify that the purity file should be deployed with the tests.

For example, when running Celeriac with the QuickGraph library test suite, we add a DeploymentItem entry for the purity file to LocalTestRun.testrunconfig:

<Deployment>
   <DeploymentItem filename="QuickGraph.Tests\bin\Debug\QuickGraph.pure" />
   <DeploymentItem filename="QuickGraph.Tests\GraphML\" />
</Deployment>

From Visual Studio

Suppose that the assembly under test is Bank, and that the test suite is BankTest.

1. Create an instrumented copy of the assembly under test (see the Getting Started guide).
2. In Visual Studio, navigate in the Solution Explorer to the BankTest project and expand the References folder
3. Delete the reference to the original project (e.g. Bank)
4. Add a new reference to the instrumented assembly. This is accomplished by right clicking on the References folder, clicking Add Reference, clicking the Browse tab, and then navigating to find the program saved previously (InstrumentedProgram.exe, by default).
5. Run the test project as usual.

Best Practices

This section describes best practices for inferring contracts using a test suite.

Inferring Argument Validation

To infer argument validation, test cases should not be run that intentionally pass an invalid input to the method under test (i.e., the client should exhibit correct behavior). Consider, for example, the following method:

public void MethodUnderTest(object x){
  ...
  x.Method();
  ...
}

You should not run a test that provides a null value and then checks that a NullPointerException is thrown. If null is observed as a value for parameter x, Daikon will not infer that x != null is a valid invariant.

However, for non-argument validation exceptions, testing inputs that cause an exception is valuable to inferring exceptional postconditions (which can be written using the Contract.EnsuresOnThrow<T>(...) method). Note that in these cases, the client is behaving properly, but the method under test will still throw an exception (e.g., because a resource cannot be accessed).