OzAgile cancelled

I learnt during the week that the OzAgile conference has been cancelled for 2010, and hopefully to be run in 2011. I was going to be a presenter, along side many of the Agile 'Rock Stars' who've written many of the books that are referenced throughout the agile world... Stay tuned as now I'm looking to present at another conference some time soon!

- Posted using BlogPress from my iPhone

A simple example of the escalating cost of defect detection

We've all been taught that the cost of finding defect later in the development/testing cycle is larger, than the cost of remediating those defects detected early. I've seen the statistics, heard the teachings and also taught this lesson once or twice myself.

Last week I came across an excellent, but simple example of how the cost escalates, and it's not just because the dev and test team need to work extra hours to effect the fix!

The example of cost came as I participated in/observed the process of investigating 2 defects that were found in production a few days after a release.

Upon reflection I've looked back at the regular defect triage meetings we held during the testing cycle, on average there were 3 participants; the development manager, either a senior developer or senior tester and myself. The meetings often lasted 30 mins, so to keep the calculations simple if each of the resources who attends cost $100 per hour, then the cost of these triage meetings was $150.

Before UAT commenced, we had a meeting with the users to walk through the outstanding defects going into the UAT phase (and our plan to address them). At this meeting we had the usual triage team, plus 3 users and an extra 2 from the dev/test space. This meeting went for 1 hour and using the $100 per hour per resource, it cost $800.

At the conclusion of UAT another two meetings were held, with an additional user representative (senior stakeholder) and our project manager. This meeting lated an hour, and therefore cost (in our simple model) $1000.

Now when the production defects were discovered, on Tuesday we had several more meetings with between 12 and 18 attendee's costings $1200-$1800 per meeting.

As can be seen in the table below, the cost of the defect triage meeting increases by 500% by the time we were triaging the defects found in the production environment!

The columns on the right hand side of the table above records the number of defects discussed at each of the meetings (made up of course!), and as you would expect the number of defects discussed at each of the meetings decreases. However the most interesting point is that the triage cost per defect spirals to be 37.5 times more expensive to triage the defects found in production....

My final point is note that the figures described above don't include ANY actual coding/configuration or testing effort - so it's easy to see how the costs spiral upwards!

Test early, test often. Make the smart choice, test wisely and make an early investment to fix defects rather than accumulate the technical debt :-)

Certification in software testing and a drivers license...

Recently I have been thinking about certification of software testers, for no particular reason other than it's been on my mind. I've been thinking along the lines of 'what lessons can certification learn from the evolution of the driving test?'

So it came to me very late one night, the process of gaining a certification in software testing has many similarities to gaining a driver’s license. Regardless of the level of the license (Learners, Provisional or higher) it's just the beginning of greater learning (through experience and further training).

If you think back to when you first got your license, it would have involved some study (hopefully!) of the road rules, a test -maybe written, online or practical; or all combination of all of three?

After you 'made the grade' you were allowed with some varying level of supervision on to the open road. Yes, the police and road rules are a form of supervision!

So, what does this have to do with software testing? Well I've come across many a tester who has a certification and believes that that puts them ahead of the pack. But if we compare this to the newly licensed driver - in the same way the certification identifies you as having a 'known' level of understanding in the subject area. It doesn't mean you know it all, and it most certainly doesn't mean you shouldn't continue learning :-)

The other aspect of the driver’s license analogy I thought a lot about was how the process of getting a license has evolved. I recall a conversation with a developer some time ago when I was going through the process of getting my motorbike license. He said back in the day when he got his license the process involved meeting up with the local policeman and demonstrating that he was able to start, stop and turn the bike. The final part of his test was conducted in a car park which had a gravel surface - just to mix it up little! All in all this process lasted about 30 mins. When I compare this to my experience they are worlds apart. The process I went through to get my learners involved a full days training - the quarter of which we didn't even sit on a bike, let alone ride it. Once I passed that course I was allowed to ride a little bike, at a restricted speed - I now had my learners.

So the point is? Well the evolution of the license tests, including extended learning before the granting of a license is traced directly back to the correlation between the level of driver training and the frequency of accidents. I believe that sometime in the future, that certification in software testing will evolve to be more practical as we realize that being certified doesn't guarantee testing results.

It's my belief that as an industry embracing certification how we need to evolve our thinking and take some lessons from other industries and certification processes. The lesson I'd like to see taken up are that newly certified testers should be paired with an experienced mentor to help them grow into polished professionals :-)

What a way to celebrate all that is testing by going to watch one of the hardest and most physical types of testing there is - an international rugby test match! As the temperature dipped to a chilly 7 degrees, a group of K.J.Ross & Associates staff, partners and clients made the small trek to Bruce stadium. It's the first time the Wallabies have played in Canberra for several years and it was great to be a part of it.

The rumour on the radio in the morning was that the hotel where the Fijian team was staying had run out of blankets! Luckily for Michael Larsen and I, one of our guests, who happened to sit between us was a die-hard rugby fan who'd sat through many a cold rugby encounter. He waited about 5 minutes and then produced a blanket of his own and kindly offered to share - thanks Pete! One of the other guests also showed some good early form, pulling out a stubbie cooler, or warm hand preservation devise - sadly in this instance there wasn't enough to share :-(

Unlike the teams we were watching, the starting line up for team KJRA had several last minute changes! We lost one of our original guests to injuries sustained during a half marathon! and another with family commitments. Luckily, like all good teams we had depth to call on and the final team was decided about 60 minutes before kick off.

The first half was an arm wrestle with the wallabies only slightly in front at the half time break (14 - 3). The half time break entertainment was hardly noticed by team KJRA. Instead of oranges we opted for hot chips! really a MasterCard moment.

The second half was not so close, with the wallabies clicking into gear and running away 43 points to 3.

Needless to say a great time was had by all :-)

Interesting quotes from the books I've read recently...

I've not long finished 'The Speed of Trust' - by Robert M. R. Covey. I came across this book by referral (of sorts). I was attending a session at the KJRA Summer School, presented by Dr. Mark Pedersen on 'Test Project Management'and Mark referred to this book to demonstrate a point he'd made.

I can't remember the specific point, but I liked the sound of the book, and identified through my own experiences (professionally) were my managers had just trusted me to 'do testing'.

I'm convinced (imo) that within the software development industry, testing is still considered a 'dark art' and/or a 'necessary evil' and therefore not well understood (a topic for another time)! So trust in the test manager and the testing team is critical to success. So often the stakeholders we test on behalf of, take our word (trust us) on the assessment of the defects we've found, and the results of the test cases we've run.

I also found myself nodding my head and agreeing out loud when the book described the effect of high and low trust on speed and cost of everyday transactions, and the notion of 'trust taxes' being applied through either lower speed or increased costs - all true.

Here's a couple of quotes I noted, and the reasons why they made sense or I identified with them - six (6) to be exact:

1) "You should not be satisfied with being a victum, nor with being a survivor. You should aim to be a conqueror." Dr. Laura Schlessinger

Recently I worked on a gig where I was so not seeing eye to eye with all whom I have should been and it made my job soooo much harder to do. It was a lowest of low trust environments, I an outsider - even worse, a contractor - oh no. But anyway the role nearly broke me, but it didn't kill me and I am certainly stronger for it. At first I thought I just survived, but then after returning I found out that I was part of a watershed in perception, and assisted to change the direction for the better. I liken my influence to a tug boat assisting a large ship change course, the ship with its rudder fully starboard, will eventually turn around, but with a little tug boat pushing at the bow, it turns a lot quicker!

The concqueror bit came through another conversation which went something like "the boss said in a meeting, I want reports like Andrew used to give me, ones that actually give me information... and I also want them daily like he used to do!" I thought, that's great after all the resistance I encountered obtaining the data for those reports, tis great to know that now they have to do it my way!

2) In reference to training staff - Question: "What if you train everyone and they all leave?" CEO Response "What if we don't train them and they all stay?" - anon CEO.

It's often been a discussion topic, about the risk involved if you train up the young talent and then watch them walk out the door, and the same is said for contractors in an organisation - they should train themselves. But how does this assist your organisation to grow and become more efficient? It's an interesting point, all to often I've seen staff leave because another company has offered/promised better training and/or options for career progression. My personal experience has been that by allowing staff to go on training has been win, win. The staff have gained some skills, and that means I can push them into areas where I couldn't previously...

3) "we all make mistakes. If you can't make mistakes, you can't make decisions." Warren Buffett.

This is a great comment, all decisions envolve risk and if people are not empowered to take some risks then there is little chance of reward.

4) "There are no facts, only interpretations." Friedrich Nietzsche

I've always been of the opinion that there are three (3) sides to a story, his, mine and the truth somewhere in the middle. This quote challenges that view a little. I think it might also be equally as true as the "There are no defects, only interpretations of software features!"...

5) "We judge ourselves by what we feel capable of doing, while others judge us by what we have already done." - Henry Wadsworth Longfellow,

Everyone who's ever been knocked back for a job at the 'next level up' knows how one feels...

6) Tom Watson "If you wanted to increase your success rate, double your failure rate."

This reminded me about an Agile development comment I heard once, in terms of failing it was 'Fail fast, Fail often, Fail better' and was along the lines "If at first you don't succeed, try again".

There was so many more pages that I folded over, with highlighter or pen underlining just like I used to do while studying at university, but these would have to be the top 6.

New(ish) Testing books... Part 1

Whilst preparing to present a course recently I stumbled upon several testing books that are relatively new (published 2009). The first book ' Exploratory Software Testing' is the latest (?) release by James Whittaker, author of titles such as 'How to Break Software' and 'How to Break Security Software'

Initially I came across this book late at night while watching the keynote presentation from StarWest 2009 on stickyminds.com. Loving the ideas that James presented in the keynote, I searched the web, and ordered the book that same night (well it was early the next morning by then!).

I have mixed feelings about this book, I love the metaphor 'Tours' that James describes as the basis of the testing approach he implemented at Microsoft, and then Google. It's (the metaphor) great, because everyone one has travelled and been on a tour of some sort - be it a school trip or an overseas adventure. This means that instantly when speaking to someone about creating a 'highlights tour' of their application there is a connection and mental picture created.

It was the definition of the tours, there derivation that I thought that the book would have gone into in more detail. The webinar touched on how the tours where created, and the book gives a few paragraphs to each of the established tours, but didn't go into much further detail (that I could find).

I understand each application is different so therefore each time a tour is created it will be unique. But I was expecting some more detail on James' experience in creating the tours. Did they whiteboard the tour outline and then overlay the 'stops' or 'highlights' of the application they were testing on it? Or was it in reverse were all of the application functions identified first and then categorized?

One of the thoughts I had, was this the intent of the book was expose the thought process and idea, rather than be text book with specific examples... Anyway, I'd recommend this book for any tester it's covers some really interesting topic related to exploratory testing, and testing in general. James' vision for the future of testing is very exciting!

ISBN-13: 978-0-321-63641-6

Test estimation hokey poky

Recently I've been involved in estimating the testing component of several tender responses. As is usually the case, when responding to tenders, the amount of information you have to work with is minimal. This is not just restricted to tenders, as I recall being involved in the estimation of testing for projects in the proposal stage where the business case is all I had to work with. So as with all estimation activities there are assumptions that have to be made, and importantly declared in your estimation model.

In this post I'm focusing on how I estimated the testing effort for a tender, for which I had access to an immature functional Performance specification (FPS). The FPS documented each of the system requirements, and also had an annex which detailed a series of fleshed out business scenarios(usecases). Each of the use cases outlined via a process diagram the main steps in the 'happy path' and also defined the most likely (but not every) alternate path. This information, even though immature and incomplete proved invaluable with constructing my testing estimate model.

The way that I approached the estimation was using the assumption (well educated & researched guess) that each of the requirements would need at least one test case to be created in order to verify its compliance. I also allowed for an additional test case per alternate path identified in the use case. This brought the number of estimated test cases out at 160. History tells me that (if we win) once we start the test analysis and design there will be instances where several requirements are covered by a single test, and other requirements that will demand several tests.

Of course, in the situation were you are estimating for a known application, then there are heaps of metric's surrounding test cases and requirements that you should be able to draw on to assist with your estimation.

Next, I calculated the amount of time it would take to analyse, design, document and verify the test cases (on average). Based on a verbal description of the system we'd be testing, and using all the information I could find I determined that 2 hours per test case should be enough. And this is where the hokey poky started!

Some of the estimation team disagreed with my estimate, stating that its not possible to analyse, design, document and verify a test in 2 hours. I agreed to disagree, as there are a number of reasons why you might not be able too, but equally as many reasons why you certainly could it all depends on the complexity of the requirement being tested and the system implementation and so on. So using the principal of 'you can never have enough time to test, the team then agreed that 4 hours or 1/2 a day would be a more palatable estimate, and so I increased the estimate...

Testing is like a balloon being filled with air, the more air you put in the bigger the balloon. Similarly if you let some of the air escape the balloon decreases in size and takes up less space. Determining what the 'optimal size' of the balloon all depends on where its to be used, and its purpose.The same can be said for testing it all depends on the type of testing, and what risks you are trying to mitigate. In the event that the balloon bursts, I believe it more a case of poor management of the testing process. By trying to squeeze to much testing into a confined time box, BANG, the result can end in a catastrophe. It's often not the testing that suffers, its the quality of the application released. One of the biggest lessons I learnt when first estimating testing, was the testing schedule had to include time for defect remediation. I found it necessary to include because all to often the dev team's schedule finishes once the application is delivered to testing!

Back to the hoky poky, "my estimates in" and then after having submitted the initial adjusted estimates, the review team came back to say the cost of testing exceeded the cost of development and its a COTS product! They asked if there was any activities that we could cut back? LOL, I said well we could go back to the original estimates I gave? Of course this slashed the estimate by 50% which (due to the original doubling!). The estimate was resubmitted.

Just like the hoky poky, my estimate when in, it came out, back in again and then I had to shake it all about! Now we have to win the rest of the work to see who was closed at pinning the tail on the estimation donkey! stay tuned.