You might know the famous testing pyramid established by Mike Cohn, one of the founders of the Scrum Alliance. This pyramid has been used and modified a bunch of times. Now is my turn to fill up the pyramid!
The testing pyramid
For those who aren’t familiar with the testing pyramid (or who don’t remember), the principle is as follows:
Mike Cohn
The bottom tier has to be automated the most: it gives the best ROI as unit tests can be easily written by developers and gives them an immediate feedback. The more automated unit tests, the stronger will be the foundation.
The middle tier is more difficult to automate than the foundation and tests at this level will be focused on business-facing tests. These functional tests will give an answer to the question “are we building the right thing”. It will ensure functionalities match the requirements.
The top tier will the least automated tests because this level is usually composed of UI tests that are expensive to write or maintain and that often require human control (over visual aspects for instance).
What the pyramid misses
I think the pyramid is very useful and test teams can gain efficiency if they follow its principle. However, something is missing…
Tests in this multi-layer system are being executed by different people, and all these tests are of different nature. This introduces one more difficulty: there is no aggregated vision of all these different tests. For instance, it is hard to get a complete picture of acceptance and GUI tests.
As a result, teams can leave “testing holes” within their application: software areas that are not covered by any tests and it is more likely that bugs will slip through the net of your testing in these areas!
Unknown risk coverage
However it is almost impossible to reach 100% test coverage of the software. So how to assess risks related to these “testing holes” in order to decide which tests should be added?
We suggest 2 ways of doing this assessment in order to have the right focus for additional tests.
1. Focus on regression risk
The point is to compare the current version of the software you have to test with the previous one. This comparison will allow you to identify changes where regression risks are higher.
Mixing this information with test coverage clearly identifies regression risks that have not been covered by any test.
As a picture is worth a thousand words, let me show you that new principle:
Known risk coverage
2. Add business perspectives
Even with a focus on untested changes this could still be too large when you haven’t yet got a high proportion of automated test. So in addition, we recommend adding a business perspective: drill-down to untested changes by features or software functional areas. So you can focus your additional testing activities where bugs should be avoided in order of priority.
Functional view
Ok, but what do we do when we don’t have a high number of automated tests?
Let’s jump to the top of the pyramid, this is where you will need to have more manual tests if your coverage by automated tests is weak. In such a situation you won’t have enough time to execute all the manual tests
every time. You will have to select relevant tests and execute them depending on their priorities and on what has not been tested at lower levels.
Combining changes, coverage and business perspectives, not only will you optimize the time you spend on testing by balancing automated and manual tests, but you will also focus on relevant tests (and thus avoid losing time with useless tests) and ensure you covered the risks in the application.
Adding test footprints
During testing, Kalistick’s agent can record all the execution paths through your application. This is not about record/replay but linking test cases and code so that when changes are made in a release, relevant test cases to cover the corresponding regressions risks can be identified.
As you can imagine, the technology is self-learning. The more tests you execute, the better the software is going to know yours and be able to help you to build more efficient regression test strategy.
The goal of this technology is double:
- Optimizing test efficiency by selecting relevant tests for a test campaign and avoid losing time with useless tests which aren’t testing changed modules
- Ensuring that all risks are covered, even indirect risks dues to indirect impacts of certain changes (mainly regressions thus)
Conclusion
On your way to reach the right balance of automated and manual tests at each level of the pyramid this technology will help you improve your software testing efficiency.
Other posts that you should read:
- Non-Regression Testing: an alternative to automation
- Agile Quality Management for Jira & Greenhopper