A dartboard and levels of testing?

"We need to create a regression test matrix" is a statement that I have heard several times. In conversations prior to todays I've nodded wisely and agreed, but not acted. Today though I nodded and smiled, there may have been even a little bouncing!

Why? I hear you ask well today a potential solution appeared before my eyes whilst I was trying to solve another problem.

Taking a step back, today I was tinkering with various diagrams to display the different levels of our manual, functional testing. The diagram was to be included in the release test plan. I created three (3) models which displayed Acceptance aka ‘Full Set’, Regression and Sanity testing covering different proportions of the system...

The models are simple in construction and there purpose is to highlight the differences coverage and that none of the testing covers ‘all’ of the system functionality.

The relative size of the objects in the diagram, though not explicitly, gives an indication of effort to complete that level of testing. Relatively speaking the full set of tests is somewhere around 50-100% larger than the regression suite, which is 50-100% larger than the sanity test suite.

At this point I discounted the excel pyramid graph and started to focus on the boxes and circles. I feel that both pictures provide a reasonable representation, but there was something about the circles that I kept coming back too.

Fast forward an hour and I’m into the weekly status meeting and due to our recent production release and impending production patch cycle the “we need a regression test matrix” comment was made. We discussed the matrix should take the form of a excel spread sheet and the need to conducts tests across the system as well as focus on the areas of change was required. My eyes began to widen! Circles was the choice and I thought the easiest way to explain where we’d go into more detailed testing was to draw smaller circles around functionality and then bang, I hit the bulls eye so to speak. If you picture a dart board it’s made up of wedges (20), each with two large sections and two bands.

So to my original circle diagram I added several wedges, as I already had the rings (test levels) present. The result is shown below. Each of the wedges represents a slice of system functionality or grouping of functionality.

Now each time it comes to patch testing I can plot which areas of the system are going to be tested more thoroughly using a set of darts, along with a steady hand and intense concentration.