Bruno Peeters Technical Blog

In my previous blog post I described Behavior Driven Development and specifically the tool SpecFlow. Another area within testing is unit testing, which focuses on testing of classes. This already shows two different areas where to focus on when testing. What kind of testing areas do we have and when to use which strategy when?

To have a decent answer to this question, let’s have a look at the test pyramid.

The test pyramid

Mike Cohn came in his book Suceeding with Agile up with the concept of the test pyramid. It is a methaphore to devide testing in several layers, where each layer describes how much testing is done.

End-to-end testing involves testing the whole application as if a user would operate it. Typically this involves User Interface-tests.

Integration testing focus on testing components of software together. Things like differences between interfaces the components use to communicate with each other is tested or to test if certain version of components will work properly together. An example of an integration test is testing if your hardware layer is still be able to interact with a new hardware driver.

Unit testing is testing of the most smallest functional unit of code of your system. Within OOP, this is writing tests for a class, testing it’s interfaces i.e. the methods and properties of the class.

Up in the pyramid, there is more integration between the different components, whereas below components are more isolated. Typically the complexity of the execution paths that is put into test at the lower levels is much smaller, and therefore execution of the tests is much faster compared to the huge complex paths in the top. In the end-to-end tests, the whole application with all it’s (stubbed) external interfaces need to be set-up, like for example databases. Just creating a simple class instance is much faster and takes only a matter of a couple of milliseconds to get an answer.

It is better to write a lot of small unit tests that run very fast to test your software, write some integration tests and have only a few end-to-end tests.

Testing ice-cream cone

The testing ice-cream cone is the opposite of the test pyramid. It is actually an anti-pattern, where the number of manual tests are very big and the (automated) unit tests are very small. Besides end-to-end testing, integration testing and unit testing, the main part of the testing consists of manual tests. The following image shows this:

The main difference between the testing ice-cream cone and the test pyramid is that initally creating the tests within the test pyramid takes more time, but during time testing using the ice-cream cone costs more time because it is very labor intensive. Poorly designed tests are also difficult to maintain and will break even if you don’t touch any code nearby.

The costs of testing

Testing comes at a cost, this is a fact for both manual as automated tests. Both kind of tests needs to be designed, either in the way of a test specification that can be executed manual or a piece of code that performs the test.

If a test is written, both the manual and the automated test should be maintained. If the UI for example changes this will impact the test specification. As a human tester can adapt fairly easy during execution of the tests, the test-script won’t. Then it comes to the fact how good the tests are written and can be updated to reflect the new situation.

Typically end-to-end tests are more difficult to maintain, as unit tests are fairy easy to maintain. A main part of this is due to the fact end-to-end tests have more dependencies and could therefore having a bigger change to break any of these dependencies.

So given the facts above, it is cheaper to have more unit tests then having UI tests in place.

Code coverage

Code coverage tools can be used to measure the lines of code that is touched by one of your tests. For C# you can use for example Coverlet. Reports out of this can be generated that can be part of your CI-strategy and for example show up in Azure DevOps.

The level of coverage gives a good indication of the health of your test suite. It will show you the gaps, you don’t automatically test.

Even if a higher coverage level is better, it must be not a goal on its own. It will eventually adding tests that in the end doesn’t add real value, only for the sake of increasing the coverage number.

The right tool for the right environment

End-to-end testing

SpecFlow is one of the tools that typically focus on end-to-end testing. Within Gerkhin the specifications are written, so a non technical user can understand what the test actually does. Since the specifications are written from a user perspective, the tests focus more on interacting with the User Interface.

Integration testing

For integration testing of external interfaces, there are several tools on the market. For example Postman is a platform to test your API’s.

For integration testing between your own components, you can use a unit test framework for example.

Unit testing

Unit tests focuses on testing writing small tests for testing your classes. Therefore it interacts with the interfaces of the class to put it into an initial state, performing the test and verifying the outcome.

It also serves as documentation to other developers that shows how a specific class can be used. The unit tests itself serve as examples how to class can be used.

Using a tool like SpecFlow for unit testing would lead to quite some overhead. In the end you have to write down a formal specification, which isn’t read by the real audience. Notice that the end users of your class will be developers. With a SpecFlow they do have some formal specification, but they probably dive quickly into the steps to really see how the class is being called. Leaving out the formal specification, leads back to having a clean unit test that shows code snippets of how to operate a class.

Conclusion

In this post I’ve tried to give you an overview of automated testing and the areas where to look for. Based on this, you have to decide which tools you need to have in place. Each tool has it’s own strengths.

We also saw that testing doesn’t come for free. You have to write the tests, but they should also being maintained as the software it is testing will also evolve over time.