In a previous post, we looked the newly introduced Assert.Multiple in the XUnit 2.4.2 assertion library. In this post, we will dive deeper into different ways of expressing tests and especially, test input.
Facts are the simplest possible test-case in XUnit. They are self contained in that they contain the arrangement, act ,and assertion, but a test-suite of only facts would probably lead to duplicating test cases. And you know, when you duplicate code, you create mistakes when adding the difference!
This is where parameterized tests come in, denoted by [Theory]. Parameterization seem like a natural extension of the simple testcase, created using [Fact]. But it is not always obvious how to parameterize tests, because actual values for tests must be constants. Lets start by looking at the trivial simple case:
In the left example, we have three [Fact]s, and in the right listing, the condensed [Theory] equivalent. The translation is simple, because the values are literal values. What if we were to use regular objects?
Complex Theories
As values in [InlineData] must be compilet ime constants, you can’t write [InlineData(new ...)] nor [InlineData(Foo)] with Foo being anything but a constant; either literal constant or constant member, written const int Foo = 10;
If our tests involve class-types rather than simple values, using [Theory] becomes more complicated. In the example below we have simple [Fact] based testing followed by two attempts of using [Theory]:
The first [Fact] tests ordinary sum of two vectors, the second, the sum of two all-zero vectors. This could be done using [Theory] using params or named parameters with instantiation as part of arrangement:
On the left, we parameterize our method using a variable number of parameters, received as an array. Vectors are then instantiated using indices. The test formal parameter is simple in this example, but this approach is error prone caused by the number of parameters in the [Theory] attribute, and the need for disciplined indices into the array when constructing the vectors. On the right alternative, the parameter count is checked at compile time, but also in this example, constructors and method parameters are common sources of error. Both approaches, however, could be considered a violation of the simplicity requirement of test-cases.
TheoryData using MemberData
Instead of the complex solutions above, with the obvious drawbacks, we can use MemberData to point to a static class member, to specify data for our testcase. Using the TheoryData class, to provide static type-safety is the preferred way, as opposed to using IEnumerable<object>, as discussed in a previous post, where I also discuss a different alternative for type safety. In the example below, I’ve changed the above complex [InlineData] to use MemberData:
Data for the theory is now specified using MemberData, with a reference to the static member VectorValues of type TheoryData<Vector,Vector,Vector>. This is done using [MemberData(nameof(VectorValues))] instead of the previous InlineData. VectorValues is a static member, which generates a test-data set on every get. Beware, that it is important, for repeatability and isolation, that the data is newly created on each get!
Why do we use TheoryData? TheoryData will make test data generation simpler and typesafe. However, it will only make test data generation type-safe, but unfortunatly, not provide static type safety in the MemberData-declaration. This is checked only at runtime, but runtime errors will likely be caught at runtime, just as with enumerable set of objects. For example, using TheoryData<Vector,Vector,int> will result in
System.ArgumentException : Object of type ‘System.Int32’ cannot be converted to type ‘MathLib.Vector’
As seen in the example, we can now simply write
new TheoryData<Vector,Vector,Vector>{{new Vector(1,2,3),...,new Vector(4,4,4)}, ...}
or in the later C# versions, we can even write
new {{ new (1, 2, 3), new (3, 2, 1), new (4, 4, 4) }, ...}
Which is clearly a big improvement of the above.
TheoryData using ClassData
Adding static members to our test classes could be considered a drawback and as adding further complexity to our test-code. Furthermore, it it complicates reuse of test-data in other tests; which class should provide the static member? – should it be copied?
A fix for this is to alternatively use the attribute ClassData and provide an implementation of TheoryData. At the time of writing, generic attributes are unsupported, so we use typeof to provide the Class providing the test data. A rewrite of the above looks like this:
The important difference is, that our data source is not tightly coupled to our TestClass, and also, it is now clear, that every test-run will use an untouched fresh copy of the test-data. This ensures that our tests will always run in isolation even if other tests modify test-data.