Using methods that are not under test within a unit test for a different method?

Question

I'm building a Sudoku generator. I have a board class with a number of methods:

public class Board {
    public Board() { /* Creates an empty board */ }
    public bool ValidateRow(int row) { /* Checks for errors in row */ }
    public bool ValidateColumn(int column) { /* Checks for errors in column */ }
    ...
}

I'm following TDD, and as such have a full suite of tests for all these methods. I would like to add two new methods to this class:

public static Board GenerateFilled() { /* Creates a solved board /* }
public bool ValidateBoard() { /* Checks for any error on the board /* }

I'm struggling with how to write my tests for these methods. My first thought was:

[TestMethod]
public void GenerateFilled_GeneratesAValidSolvedBoard() {
    var board = Board.GenerateFilled();
    Assert.IsTrue(board.ValidateBoard());
}

but I realized I wrote the same test for ValidateBoard:

[TestMethod]
public void ValidateBoard_NoErrors_ReturnsTrue() {
    var board = Board.GenerateFilled();
    Assert.IsTrue(board.ValidateBoard());
}

This test relies on both GenerateFilled and ValidateBoard working correctly, though the method under test changes. I've come up with the following ways to avoid this problem:

1. Duplicate the logic of the method not under test into the test. Use that logic to validate my method under test instead of calling the other method.
2. Leave the GenerateFilled test as is and use hardcoded sample data to test ValidateBoard instead of calling GenerateFilled.

I'm not a fan of option 1 because it will make keeping the tests accurate tedious if small changes to the logic in the methods that were duplicated change.

I don't really like option 2 either though because it relies on my sample data case being general of all cases, which is less likely to be true the larger the dataset is.

I suppose in the worst case this just results in any error in either method causing both tests to fail, but it is a bit of a smell. Has anyone come across a similar scenario and found a better solution than the two above?

Answer 1

When going the TDD route, you should not try to add two public methods at once. Instead, pick one of them, and cycle through "test a little, code a little, refactor" until the methods quality is high enough. Only when you are done with that, go to the next public method. If you do this one-by-one, it becomes pretty obvious how to build the tests.

For the shown case, I think it is probably easiest to start with ValidateBoard first. Order matters here, since when going to write a method like GenerateFilled, for which I assume you don't know how the produced board will look exactly, you will need a universal board validator either for creating tests which are robust enough against an anticipated change in the generation logic.

So for testing ValidateBoard, you have virtually no other choice to provide either hardcoded test data, or write some simple generation logic to create valid or invalid boards. The function GenerateFilled is not available at that point in time, and even if it would be, it would not produce specificially shaped test data, like invalid or incomplete boards. This is data you will need to test a function like ValidateBoard completely. So after some TDD cycles, you will end up having enough trust in ValidateBoard to contain no severe bug any more. During this step, ValidateBoard is exclusively the subject under test.

Then, when you start writing tests for GenerateFilled, you can utilize the already well-tested ValidateBoard method in your tests. In this step, GenerateFilled is the subject under test, and you should name the test accordingly, like GenerateFilled_GeneratesAValidSolvedBoard, and not ValidateBoard_NoErrors_ReturnsTrue. Replicating the logic of ValidateBoard in the test again, just for making the test "more isolated", does not bring you any real benefit, only duplicate code.

Having the logic of ValidateBoard duplicated in the tests makes IMHO only sense, if

• there is a short, simple (but maybe slow) implementation possible, one can use for the tests. It should be so simple that the risk for having a bug in there is low.

• the "real" ValidateBoard function will be a highly complex version (for example because of some optimizing), but also more error prone because of increased complexity.

If you encounter such a case, then the simple implementation can be done in the test code and used for testing GenerateFilled as well as ValidateBoard both.

Answer 2

I'm following TDD, and as such have a full suite of tests for all these methods

There's a conflict in this statement. Either you have adopted a TDD approach and have a set of tests and some refactored code that enables those tests to pass, or you have blindly followed the "unit means method" test approach and have thought of a method, then written tests for that method.

TDD is not about "identify a need for a method, think of a test for it...". It's about "think of a test, have it fail, make it pass, refactor". Whether that test means modifying an existing method, creating a new one, stringing together 20 methods or running the entire app within a unit environment makes no difference.

So rather than say:

I would like to add two new methods to this class:

instead, state it in a TDD fashion:

I want to create a solved board

I want to validate the whole board

Now you have you two requirements and can get writing code to make them, one at a time, pass a set of tests. It's very likely that you'll need to write GenerateFilled() and ValidateBoard() to fulfil that first requirement. So be it if you do.

And, as you develop the app, do not be afraid to completely rewrite tests to allow them to better represent the growing functionality. You may wish to look into Recycling TDD to really get an idea of how powerful a TDD tool this can be.

Answer 3

I'm building a Sudoku generator.

You may be interested in Learning From Sudoku Solvers;

I'm following TDD

Not with those methods, you aren't.

If you were doing TDD, the first thing that you would do is figure out the API that you want to test. In other words, what should the client code look like. For a Sudoku solver, you might start with something like

var solvedPuzzle = findTheSolution(unsolvedPuzzle)

Alternatively, you might realize that not all unsolved puzzles have solutions, and the solution may not be unique -- heading you off perhaps toward an API like

var solvedPuzzles = findAllSolutions(unsolvedPuzzle)

Part of the point of the TDD exercise is to be experiencing what it is like to write client code to provide you with a sense of whether or not the API is usable.

I'm struggling with how to write my tests for these methods.

I see two things that might be giving you trouble.

One is that you are testing your code within its own algebra. You've essentially written

var x = MyCode.doTheRightThing()
Assert.assertTrue(x.didTheRightThing())

Of course the code thinks it did the right thing; the trick is to verify that independently.

The other is that your test is relying upon a hidden side effect. If we refactor your original code a little bit, we get something that looks like

var verified = Board.GenerateFilled().ValidateBoard()
Assert.assertTrue(verified)

Which is to say, the API for this "function" looks like

var verified = systemUnderTest( /* takes no arguments */ )

You are missing a dial here (the input) to constrain the behavior of the system under test.

Using methods that are not under test within a unit test for a different method?

This is a perfectly fine thing to do, but there is a caveat -- somewhere in the test suite you need a way to measure the behavior of the system under test that is independent of its own calculus.

This test:

Assert.assertTrue(Board.generateValidBoard().ValidateBoard())
Assert.assertTrue(Board.generateInvalidBoard().ValidateBoard())

Doesn't tell you if anything works, it only tells you that the code is internally consistent.

To get independent confirmation, you need a test that looks something like a spec; when we build a board from these inputs, we get a valid board, when we build a board from those inputs, we get an invalid board.

So your test for ValidateBoard would look like

var validBoard = Board.createBoard( descriptionOfAvalidBoard )
Assert.assertTrue(board.ValidateBoard())

var invalidBoard = Board.createBoard( descriptionOfAnInvalidBoard )
Assert.assertFalse(board.ValidateBoard())

Now that you have independent confirmation that ValidateBoard conforms to its specification, you can turn your attention to the next bit

// This is the thing we are testing
var board = Board.GenerateFilled()

// We have tests up above that demonstrate that this piece works
var isValid = board.ValidateBoard()

// So this checks that we have an implementation that is internally
// consistent with something that we have validated independently.
Assert.assertTrue(isValid)