My thought for the day - just because one test step fails, should the whole test fail?
Today I had cause to ponder the status of failed test cases. I'm sure you've been in the situation were as a test manager you've been asked if the system is 'ready' or 'is testing finished'. I was asked this question today for the umpteenth time. The situation was this, we are currently testing an environment which has been refreshed to the current production baseline (without data!) and we ran our regular 'sanity test' to see if everything was in the correct state. Situation normal. The sanity test set that we run are a high level set of tests that cover all the key functionality and integration points. Often we run this set of tests and there is failures i.e. Exchange isn't working or the search crawl hasn't run. Today when I reported the results back to the Mgt team for some reason the words coming out of my mouth didn't seem to make sense. The stat's were something like 22 passed, 4 failed and we are good to go. short pause, we've raised 3 defects and we are good to go. hmmmm, this is were I started thinking so we've got failed tests and defects but we are still good to go? The fact of the matter is, this is a risk based decision in that the risk of the failures and extant defects in the system causing a loss of functionality or adversely effecting the user experience is low. Still pondering I think that the floor in the presentation of the results (I have) is that all too often we consider the results of the tests somewhat independent of the outstanding defects.
My thoughts today brought me too this conclusion: Each of the test step failures should be given a severity which aligns to the defect which should be raised as a result of the failure. Then at the conclusion of the test (ie when all the test steps are executed) the calculation of Pass-Fail should be an aggregate of the step failures (or not). For each organisation this matrix and algorithm would need to be configured & tweaked but I think it has merit :-)
By reporting on the test status in terms of failure severity I think will bring more meaning to test results (another dimension!). We could further enhance the reporting by assigning each of the test scripts a priority and then reporting on the number of high priority tests with severe failures. Oh the possibilities!
Today I had cause to ponder the status of failed test cases. I'm sure you've been in the situation were as a test manager you've been asked if the system is 'ready' or 'is testing finished'. I was asked this question today for the umpteenth time. The situation was this, we are currently testing an environment which has been refreshed to the current production baseline (without data!) and we ran our regular 'sanity test' to see if everything was in the correct state. Situation normal. The sanity test set that we run are a high level set of tests that cover all the key functionality and integration points. Often we run this set of tests and there is failures i.e. Exchange isn't working or the search crawl hasn't run. Today when I reported the results back to the Mgt team for some reason the words coming out of my mouth didn't seem to make sense. The stat's were something like 22 passed, 4 failed and we are good to go. short pause, we've raised 3 defects and we are good to go. hmmmm, this is were I started thinking so we've got failed tests and defects but we are still good to go? The fact of the matter is, this is a risk based decision in that the risk of the failures and extant defects in the system causing a loss of functionality or adversely effecting the user experience is low. Still pondering I think that the floor in the presentation of the results (I have) is that all too often we consider the results of the tests somewhat independent of the outstanding defects.
My thoughts today brought me too this conclusion: Each of the test step failures should be given a severity which aligns to the defect which should be raised as a result of the failure. Then at the conclusion of the test (ie when all the test steps are executed) the calculation of Pass-Fail should be an aggregate of the step failures (or not). For each organisation this matrix and algorithm would need to be configured & tweaked but I think it has merit :-)
By reporting on the test status in terms of failure severity I think will bring more meaning to test results (another dimension!). We could further enhance the reporting by assigning each of the test scripts a priority and then reporting on the number of high priority tests with severe failures. Oh the possibilities!
Drew
ReplyDeleteI more frequently report on test case execution percentage (have all test cases achieved a PASS or FAIL) and separately report on the unfinalised defects (with priority breakdown). 2 sides of the same coin, but 2 sides, nonetheless. I find that higher level management get confused when test case failures and defects get reported. I do monitor and report test case failures as a percentage of the execution results as execution progresses, but am moreso interested in the "Executed", "Blocked" and "NA" metrics.
Whilst test steps are provided a PASS/FAIL result, I require my testers to relate the overall test case PASS/FAIL result based on the purpose of the test case (although as I'm reporting on the failures within the context of defects - I don't *really* care as long as execution results are achieved).
Your thoughts?
just noticed something else... I've been in situations where PMs want to consider a test result is completed when sev 1 and 2 defects are resolved (regardless of the qty or combined impact of sev 3 and 4 defects). Reporting that a test case has passed but the test steps have failed (albeit against sev 3 or 4 defects) can easily mislead/misinform stakeholders as to the nature of the results. You're also giving management the opportunity to misquote by only reporting the results of test execution with a reflection of the test case results (without noting the residual defects).
ReplyDeleteThe other element I have been looking into this week is the big R. Requirements coverage is all to often over looked (I am guilty) in reporting. I need to build a cube of some sort that means I can look at the number of requirements implemented, successfully those that are partially implemented and those missed. But also requirements should also be considered according to there priority. Then on the other side's of the cube will be the test case/script results and defects sort of woven throughout...
ReplyDeleteIn terms of 'Execution results achieved' I've seen all too often heard 'All my tests passed' are we are good to go? My question to them is 'Have you run the right tests?'. As you say a focus on execution results can often lead to misaligned expectations. It's a great challenge we face and I haven't found a sliver bullett yet!
I think both are suitable for different purposes. If you are trying to find defects then report of defects. If the purpose is to pass to another phase or get client sign off I would use requirements verified. The difference is I can have a test that fails, but still meet a requirement or only partially meet a requirement and the user still accept the system. Where if you only report on defects the user feels the whole system failed even if the defect is a sev 3. From my perspective it is more important that a requirement passed then a test passed.
ReplyDelete