Interesting quotes from the books I've read recently...

I've not long finished 'The Speed of Trust' - by Robert M. R. Covey. I came across this book by referral (of sorts). I was attending a session at the KJRA Summer School, presented by Dr. Mark Pedersen on 'Test Project Management'and Mark referred to this book to demonstrate a point he'd made.

I can't remember the specific point, but I liked the sound of the book, and identified through my own experiences (professionally) were my managers had just trusted me to 'do testing'.

I'm convinced (imo) that within the software development industry, testing is still considered a 'dark art' and/or a 'necessary evil' and therefore not well understood (a topic for another time)! So trust in the test manager and the testing team is critical to success. So often the stakeholders we test on behalf of, take our word (trust us) on the assessment of the defects we've found, and the results of the test cases we've run.

I also found myself nodding my head and agreeing out loud when the book described the effect of high and low trust on speed and cost of everyday transactions, and the notion of 'trust taxes' being applied through either lower speed or increased costs - all true.

Here's a couple of quotes I noted, and the reasons why they made sense or I identified with them - six (6) to be exact:

1) "You should not be satisfied with being a victum, nor with being a survivor. You should aim to be a conqueror." Dr. Laura Schlessinger

Recently I worked on a gig where I was so not seeing eye to eye with all whom I have should been and it made my job soooo much harder to do. It was a lowest of low trust environments, I an outsider - even worse, a contractor - oh no. But anyway the role nearly broke me, but it didn't kill me and I am certainly stronger for it. At first I thought I just survived, but then after returning I found out that I was part of a watershed in perception, and assisted to change the direction for the better. I liken my influence to a tug boat assisting a large ship change course, the ship with its rudder fully starboard, will eventually turn around, but with a little tug boat pushing at the bow, it turns a lot quicker!

The concqueror bit came through another conversation which went something like "the boss said in a meeting, I want reports like Andrew used to give me, ones that actually give me information... and I also want them daily like he used to do!" I thought, that's great after all the resistance I encountered obtaining the data for those reports, tis great to know that now they have to do it my way!

2) In reference to training staff - Question: "What if you train everyone and they all leave?" CEO Response "What if we don't train them and they all stay?" - anon CEO.

It's often been a discussion topic, about the risk involved if you train up the young talent and then watch them walk out the door, and the same is said for contractors in an organisation - they should train themselves. But how does this assist your organisation to grow and become more efficient? It's an interesting point, all to often I've seen staff leave because another company has offered/promised better training and/or options for career progression. My personal experience has been that by allowing staff to go on training has been win, win. The staff have gained some skills, and that means I can push them into areas where I couldn't previously...

3) "we all make mistakes. If you can't make mistakes, you can't make decisions." Warren Buffett.

This is a great comment, all decisions envolve risk and if people are not empowered to take some risks then there is little chance of reward.

4) "There are no facts, only interpretations." Friedrich Nietzsche

I've always been of the opinion that there are three (3) sides to a story, his, mine and the truth somewhere in the middle. This quote challenges that view a little. I think it might also be equally as true as the "There are no defects, only interpretations of software features!"...

5) "We judge ourselves by what we feel capable of doing, while others judge us by what we have already done." - Henry Wadsworth Longfellow,

Everyone who's ever been knocked back for a job at the 'next level up' knows how one feels...

6) Tom Watson "If you wanted to increase your success rate, double your failure rate."

This reminded me about an Agile development comment I heard once, in terms of failing it was 'Fail fast, Fail often, Fail better' and was along the lines "If at first you don't succeed, try again".

There was so many more pages that I folded over, with highlighter or pen underlining just like I used to do while studying at university, but these would have to be the top 6.

New(ish) Testing books... Part 1

Whilst preparing to present a course recently I stumbled upon several testing books that are relatively new (published 2009). The first book ' Exploratory Software Testing' is the latest (?) release by James Whittaker, author of titles such as 'How to Break Software' and 'How to Break Security Software'

Initially I came across this book late at night while watching the keynote presentation from StarWest 2009 on stickyminds.com. Loving the ideas that James presented in the keynote, I searched the web, and ordered the book that same night (well it was early the next morning by then!).

I have mixed feelings about this book, I love the metaphor 'Tours' that James describes as the basis of the testing approach he implemented at Microsoft, and then Google. It's (the metaphor) great, because everyone one has travelled and been on a tour of some sort - be it a school trip or an overseas adventure. This means that instantly when speaking to someone about creating a 'highlights tour' of their application there is a connection and mental picture created.

It was the definition of the tours, there derivation that I thought that the book would have gone into in more detail. The webinar touched on how the tours where created, and the book gives a few paragraphs to each of the established tours, but didn't go into much further detail (that I could find).

I understand each application is different so therefore each time a tour is created it will be unique. But I was expecting some more detail on James' experience in creating the tours. Did they whiteboard the tour outline and then overlay the 'stops' or 'highlights' of the application they were testing on it? Or was it in reverse were all of the application functions identified first and then categorized?

One of the thoughts I had, was this the intent of the book was expose the thought process and idea, rather than be text book with specific examples... Anyway, I'd recommend this book for any tester it's covers some really interesting topic related to exploratory testing, and testing in general. James' vision for the future of testing is very exciting!

ISBN-13: 978-0-321-63641-6

Test estimation hokey poky

Recently I've been involved in estimating the testing component of several tender responses. As is usually the case, when responding to tenders, the amount of information you have to work with is minimal. This is not just restricted to tenders, as I recall being involved in the estimation of testing for projects in the proposal stage where the business case is all I had to work with. So as with all estimation activities there are assumptions that have to be made, and importantly declared in your estimation model.

In this post I'm focusing on how I estimated the testing effort for a tender, for which I had access to an immature functional Performance specification (FPS). The FPS documented each of the system requirements, and also had an annex which detailed a series of fleshed out business scenarios(usecases). Each of the use cases outlined via a process diagram the main steps in the 'happy path' and also defined the most likely (but not every) alternate path. This information, even though immature and incomplete proved invaluable with constructing my testing estimate model.

The way that I approached the estimation was using the assumption (well educated & researched guess) that each of the requirements would need at least one test case to be created in order to verify its compliance. I also allowed for an additional test case per alternate path identified in the use case. This brought the number of estimated test cases out at 160. History tells me that (if we win) once we start the test analysis and design there will be instances where several requirements are covered by a single test, and other requirements that will demand several tests.

Of course, in the situation were you are estimating for a known application, then there are heaps of metric's surrounding test cases and requirements that you should be able to draw on to assist with your estimation.

Next, I calculated the amount of time it would take to analyse, design, document and verify the test cases (on average). Based on a verbal description of the system we'd be testing, and using all the information I could find I determined that 2 hours per test case should be enough. And this is where the hokey poky started!

Some of the estimation team disagreed with my estimate, stating that its not possible to analyse, design, document and verify a test in 2 hours. I agreed to disagree, as there are a number of reasons why you might not be able too, but equally as many reasons why you certainly could it all depends on the complexity of the requirement being tested and the system implementation and so on. So using the principal of 'you can never have enough time to test, the team then agreed that 4 hours or 1/2 a day would be a more palatable estimate, and so I increased the estimate...

Testing is like a balloon being filled with air, the more air you put in the bigger the balloon. Similarly if you let some of the air escape the balloon decreases in size and takes up less space. Determining what the 'optimal size' of the balloon all depends on where its to be used, and its purpose.The same can be said for testing it all depends on the type of testing, and what risks you are trying to mitigate. In the event that the balloon bursts, I believe it more a case of poor management of the testing process. By trying to squeeze to much testing into a confined time box, BANG, the result can end in a catastrophe. It's often not the testing that suffers, its the quality of the application released. One of the biggest lessons I learnt when first estimating testing, was the testing schedule had to include time for defect remediation. I found it necessary to include because all to often the dev team's schedule finishes once the application is delivered to testing!

Back to the hoky poky, "my estimates in" and then after having submitted the initial adjusted estimates, the review team came back to say the cost of testing exceeded the cost of development and its a COTS product! They asked if there was any activities that we could cut back? LOL, I said well we could go back to the original estimates I gave? Of course this slashed the estimate by 50% which (due to the original doubling!). The estimate was resubmitted.

Just like the hoky poky, my estimate when in, it came out, back in again and then I had to shake it all about! Now we have to win the rest of the work to see who was closed at pinning the tail on the estimation donkey! stay tuned.

Defects are requirements?

It was a throw away line over a beer "Defects should be written as requirements..." The context of the conversion was talking about delivering a testing training course and the topic of defects.

At the time I had "Ding" moment where I thought its such a simple concept. All testers understand that defects are important, but as possibly the only tangible outcome of testing we should give them more focus.

When I think of the conversations about testing I've had of late at go, no go meetings, almost all have glossed over the test results and focused on the outstanding defects. If I calculated the amount time I spent translating and interpreting what the functional impacts were, the associated effects on users and risk to data - well it would be days. I think that if I approached the documentation of defects with much of the discipline I'd expect to find in the documentation of requirements then I'd be that much better off.

Speaking of requirements, there is a lesson we (as testers) should learn and its one we try to teach on a daily basis. The quality of requirements has a major impact on the quality of the system built and the testing conducted. Well, our defects are mini specifications on how particular system function should work and its content is of poor quality, then the probability that the fix implemented will be sub optimal is increased.

Lastly, reason for making an effort to document defects thoroughly, is that one day down the track you might have to retest or reproduce the defect. If you've only entered scant on details in your hast to raise the defect, then it makes your job that much harder! I know I've had many instances where I've read a defect report that I'd raised and had to scratch my head to fill in the gaps I left.

It's the People; It's Always the People

An email arrived in my inbox I came across an email announcing the conference program and this particular line caught my eye. "It's the People; It's always the People" which turns out to be the title of a Key Note presentation to be presented by Johanna Rothman at the Better Software Conference 2010.

Without even reading the abstract of the presentation I was in violent agreement. So far through my career I've found that the development of relationships with those whom I need to work closely with, and those who's services I consume invaluable. In many instances its been the relationship with people which has enabled me to achieve my goals, and the lack of relationships which as hampered my progress.

All relationships have a foundation build on respect, trust and symbiotic needs; All relationships require maintenance as they change over time. The ability to form and maintain good working relationships is an essential skill for any testing professional, and is a key factor in success....

A dartboard and levels of testing?

"We need to create a regression test matrix" is a statement that I have heard several times. In conversations prior to todays I've nodded wisely and agreed, but not acted. Today though I nodded and smiled, there may have been even a little bouncing!

Why? I hear you ask well today a potential solution appeared before my eyes whilst I was trying to solve another problem.

Taking a step back, today I was tinkering with various diagrams to display the different levels of our manual, functional testing. The diagram was to be included in the release test plan. I created three (3) models which displayed Acceptance aka ‘Full Set’, Regression and Sanity testing covering different proportions of the system...

The models are simple in construction and there purpose is to highlight the differences coverage and that none of the testing covers ‘all’ of the system functionality.

The relative size of the objects in the diagram, though not explicitly, gives an indication of effort to complete that level of testing. Relatively speaking the full set of tests is somewhere around 50-100% larger than the regression suite, which is 50-100% larger than the sanity test suite.

At this point I discounted the excel pyramid graph and started to focus on the boxes and circles. I feel that both pictures provide a reasonable representation, but there was something about the circles that I kept coming back too.

Fast forward an hour and I’m into the weekly status meeting and due to our recent production release and impending production patch cycle the “we need a regression test matrix” comment was made. We discussed the matrix should take the form of a excel spread sheet and the need to conducts tests across the system as well as focus on the areas of change was required. My eyes began to widen! Circles was the choice and I thought the easiest way to explain where we’d go into more detailed testing was to draw smaller circles around functionality and then bang, I hit the bulls eye so to speak. If you picture a dart board it’s made up of wedges (20), each with two large sections and two bands.

So to my original circle diagram I added several wedges, as I already had the rings (test levels) present. The result is shown below. Each of the wedges represents a slice of system functionality or grouping of functionality.

Now each time it comes to patch testing I can plot which areas of the system are going to be tested more thoroughly using a set of darts, along with a steady hand and intense concentration.

One from the archives

Have you ever been asked ‘why do you write test cases?’ what’s your answer? There are several that spring to mind; because that’s the way we test software, because the contract said we had to, the customer demands we do, for auditing, so they can be given to the testers to execute… My believe is that regardless of the reason you think you write test cases, the fundamental reason behind test case documentation is to build and capture knowledge about how a system (or function) should work.

A colleague and I were once debating the merits of getting test cases peer reviewed, his position was that the peer reviews provided demonstrable value to the QA process and mine was that peer reviews provided questionable return on investment! After what seemed like a rather long time, the point came were my colleague said ‘well why do you think like that?’ The answer I gave was this ‘well, I believe that the process of writing test cases is how the tester becomes one with the system, learning about what its meant to do, how it should do it and so on...’ And therefore the reason that I question the value of a peer review is that more often that not, the reviewer doesn’t have enough detailed knowledge of the system or functionality to critic the test cases.

To which my colleague replied ‘that a bit Zen Drew!’

From that moment on I’ve seen the process of developing tests as a learning exercise! Needless to say that many years down the track I have used this apifenany as the foundation of my approach to software testing. I still believe the fundamental reason for documenting test cases it to acquire the knowledge about the system so that when it comes to execution the tester is able to evaluate the output and determine if it’s valid.

I’ve used these fundamental believe to underpin the methods and processes I’ve implemented at various sites, and no more so than when I started a new job at an organization that uses ‘Agile Principles’. My introduction to Agile was rather confronting at the time, coming from a predominantly ‘waterfall’ back ground.

Being somewhat new to ‘Agile-iterative’ development I search the internet and text books reading about Agile and principles of developing software using these methods. I learnt very quickly that it’s important to note that no two implementations of Agile are the same, Agile is a philosophy which embraces’ certain values. So when I talk about our Agile, it’s going to be different to your Agile and also different to their Agile, but look beyond the detail at the bigger picture about the process of learning.

The most important lesson I learnt rather early on in this new job was who to turn too if something didn’t make sense, sometimes it was the development team and other times it was the business customer. This isn’t usual – in every project I’ve worked on there was a (or several) key people whom where the fountain of knowledge, the difference here is that there was no specification document to scribble on! It’s not that documentation doesn’t exist, rather that it doesn’t exist in the same format as most people are used too! In many ‘Agile Shops’ the specifications are living – contained in stories that grow as the system grows – this is a story for another time. As the months past and I came to terms with ‘Our Organsiations’ development model I found myself questioning and comparing, trying to align the processes we were using with the ones that I’d used in previous testing engagements’. Other than conducting the testing of our applications I was also attempting to teach a new tester the fundamentals of testing, trying to apply the teachings’ of my classically trained testing methods in a world where the fundamentals’ are rather different. It was through this teachings that I came to the following conclusion; the differences in our development model when compared to the previous testing models was that 1) the time to learn the system (i.e. develop test cases) was far reduced and 2) the primary source of information for the learning’s was from the master’s within the project.

And so was born the analogy which I use regularly, in regards to learning the system I like to describe the difference between test case development in an Agile type in environment is that the knowledge is acquired through the learning by doing, somewhat hands on akin to an apprentice learning through an apprenticeship. The similarities are many, an apprentice begins there learning’s with an idea about the job they are going to be doing, and then is set small tasks by the master. As the apprentice learns and grows as a trade’s person, the master increases the complexity of the tasks and the apprentice is more capable to complete the tasks. The hands on learning is supplemented by small (in comparison) structured teachings, which are then reinforced on the job. Testing within our development process involves a very similar structure, at the commencement of the iteration the testers gain an understanding of the features that are to be implemented. Then as the iteration progresses and the stories begin to be delivered the tester tests and learns, then tests some more. The job that the tester does is overseen by the business customer and further lessons are provided by the developers. As the system grows, so does the testers understanding. The other important point to note is that through the apprenticeship, the number of text books studied and volume of notes created is minimal and when you compare this to the number of texts and volume of notes that a University produces and the picture is somewhat complete. You see like a University student, a tester on a waterfall type project the tester attends design & information sessions (like lectures) takes notes and then studies the design documents (text book or course notes) and then produces test cases (assessment tasks). During the design and build stage (semester), this process is repeated several times and then when it comes to test execution it’s when the testers knowledge of the system is tested (Exams!). One other important comparison is the ratio of ‘students to teachers’ and apprentices to masters, in the famous words of Jedi Master Yoda ‘Always there are 2, no more, no less, a master and an apprentice…’ in our development model the number of people involved is much smaller to ensure the lines of communication are short and fast. Compare this with the university model where each lecturer may have anywhere up to 100 or more students and several tutor’s to assist with the learning. An interesting point.

So to summarize, if documenting test cases is about learning the system, then the some of the differences between Agile and Waterfall testing environments can bee seen as the differences in learning philosophies used in Universities and apprenticeships. Focus on the process of learning, not the outcomes that could be achieved through the learning method.