Real Users of Model-based Testing
Posted on January 16, 2012 by Robert V. Binder
I think’ve finally found a good visual metaphor for model-based testing. In Aliens, Ellen Ripley uses the power lifter suit to battle the Alien Queen. Model-based testing is like the power lifter suit. It has to be driven with intelligence and skill, but it’s your only hope against the super bugs that like to inhabit today’s huge and complex apps.
Last fall, I conducted a survey of model-based testing users, with the support of the first Model-based Testing User Conference. I’ve finished the survey and posted the report — click here to download.
Here are few highlights:
- MBT usage spans a wide range of application stacks, software processes, application domains, and development organizations.
- MBT is accessible and practical: half of the respondents report becoming minimally proficient with their MBT tool with 80 or fewer hours of training or coaching; 80% with 100 hours or less.
- On average, respondents report MBT reduced escaped bugs by 59%.
- On average, respondents report MBT reduced testing costs by 17%.
- On average, respondents report MBT reduced testing duration by 25%.
I’ve started a discussion post on the MBTUC blog site, model-based-testing.info/2012/01/16/2011-mbt-user-survey/
The survey produced many data points, which the report presents in three dozen graphs. For a more intuitive sense of what present-day MBT users are doing, I created a hypothetical profile, using only the most frequent survey responses. Here’s that composite profile of a typical MBT user.
| I work for a large business organization. We follow an agile process to develop the software we test. The system under test is embedded in another product sold to our customers. It is programmed in C++, runs on dedicated computers, is networked, and uses a Windows OS.
We felt MBT could help reduce testing cost/work and bug escapes. We use a commercial MBT tool that we’ve integrated with our design repository and test harness. We’ve added MBT to the mix of our existing manual testing and traditionally programmed test drivers. Our effort for each testing mode is roughly equal. At present, we’re using MBT in a pilot project to test version 1.0 of our product. We taught ourselves to develop test models and use the tool by reading documentation and experimenting. It took us each about 100 hours of this self-study to become minimally proficient with the tool. We use MBT to test system-scope functionality. The tool generates code and data for an adapter that abstracts the SUT APIs and then drives the APIs and evaluates results from the APIs. We measure requirements coverage for our test runs. The output of this tool is not integrated with our build. We’ve had some difficulty developing test models. Sometimes our model “blows up” (combinatorial runaway) and we’ve found that our MBT test suites are not effective for certain kinds of bugs. However, this is manageable and not any worse than what we expected. With respect to other first generation MBT problems (updating, integration, and inadequate oracle), either we have not seen these problems or they’re not obstacles. In all, we think MBT has been moderately effective so far. For the most part, our expectations for both challenges and improvement have been met. MBT has reduced bug escapes about 60%, testing costs by 15%, and testing time by 30%. Among our co-workers, developers and managers are neutral about MBT but other testers view it as effective. Our users/customers are not aware of MBT. Overall, we are very likely to continue using MBT. |
This entry was posted in Blog, Software Testing. Bookmark the permalink.
3 Responses to Real Users of Model-based Testing
Pingback: 2011 MBTUC Model-Based Testing User Survey is Out | Conformiq
Pingback: Top Three Problems with Model-Based Testing | Conformiq
Pingback: Model-Based Testing Users Report a Triple Benefit, 59% Reduction in Escaped Bugs | Conformiq