Tuesday, February 23, 2010

Defects are requirements?

It was a throw away line over a beer "Defects should be written as requirements..." The context of the conversion was talking about delivering a testing training course and the topic of defects.

At the time I had "Ding" moment where I thought its such a simple concept. All testers understand that defects are important, but as possibly the only tangible outcome of testing we should give them more focus.

When I think of the conversations about testing I've had of late at go, no go meetings, almost all have glossed over the test results and focused on the outstanding defects. If I calculated the amount time I spent translating and interpreting what the functional impacts were, the associated effects on users and risk to data - well it would be days. I think that if I approached the documentation of defects with much of the discipline I'd expect to find in the documentation of requirements then I'd be that much better off.

Speaking of requirements, there is a lesson we (as testers) should learn and its one we try to teach on a daily basis. The quality of requirements has a major impact on the quality of the system built and the testing conducted. Well, our defects are mini specifications on how particular system function should work and its content is of poor quality, then the probability that the fix implemented will be sub optimal is increased.

Lastly, reason for making an effort to document defects thoroughly, is that one day down the track you might have to retest or reproduce the defect. If you've only entered scant on details in your hast to raise the defect, then it makes your job that much harder! I know I've had many instances where I've read a defect report that I'd raised and had to scratch my head to fill in the gaps I left.

Friday, February 19, 2010

It's the People; It's Always the People

An email arrived in my inbox I came across an email announcing the conference program and this particular line caught my eye. "It's the People; It's always the People" which turns out to be the title of a Key Note presentation to be presented by Johanna Rothman at the Better Software Conference 2010.

Without even reading the abstract of the presentation I was in violent agreement. So far through my career I've found that the development of relationships with those whom I need to work closely with, and those who's services I consume invaluable. In many instances its been the relationship with people which has enabled me to achieve my goals, and the lack of relationships which as hampered my progress.

All relationships have a foundation build on respect, trust and symbiotic needs; All relationships require maintenance as they change over time. The ability to form and maintain good working relationships is an essential skill for any testing professional, and is a key factor in success....

Wednesday, February 17, 2010

A dartboard and levels of testing?

"We need to create a regression test matrix" is a statement that I have heard several times. In conversations prior to todays I've nodded wisely and agreed, but not acted. Today though I nodded and smiled, there may have been even a little bouncing!

Why? I hear you ask well today a potential solution appeared before my eyes whilst I was trying to solve another problem.

Taking a step back, today I was tinkering with various diagrams to display the different levels of our manual, functional testing. The diagram was to be included in the release test plan. I created three (3) models which displayed Acceptance aka ‘Full Set’, Regression and Sanity testing covering different proportions of the system...

















The models are simple in construction and there purpose is to highlight the differences coverage and that none of the testing covers ‘all’ of the system functionality.

The relative size of the objects in the diagram, though not explicitly, gives an indication of effort to complete that level of testing. Relatively speaking the full set of tests is somewhere around 50-100% larger than the regression suite, which is 50-100% larger than the sanity test suite.

At this point I discounted the excel pyramid graph and started to focus on the boxes and circles. I feel that both pictures provide a reasonable representation, but there was something about the circles that I kept coming back too.

Fast forward an hour and I’m into the weekly status meeting and due to our recent production release and impending production patch cycle the “we need a regression test matrix” comment was made. We discussed the matrix should take the form of a excel spread sheet and the need to conducts tests across the system as well as focus on the areas of change was required. My eyes began to widen! Circles was the choice and I thought the easiest way to explain where we’d go into more detailed testing was to draw smaller circles around functionality and then bang, I hit the bulls eye so to speak. If you picture a dart board it’s made up of wedges (20), each with two large sections and two bands.

So to my original circle diagram I added several wedges, as I already had the rings (test levels) present. The result is shown below. Each of the wedges represents a slice of system functionality or grouping of functionality.

Now each time it comes to patch testing I can plot which areas of the system are going to be tested more thoroughly using a set of darts, along with a steady hand and intense concentration.

Thursday, February 4, 2010

One from the archives

Have you ever been asked ‘why do you write test cases?’ what’s your answer? There are several that spring to mind; because that’s the way we test software, because the contract said we had to, the customer demands we do, for auditing, so they can be given to the testers to execute… My believe is that regardless of the reason you think you write test cases, the fundamental reason behind test case documentation is to build and capture knowledge about how a system (or function) should work.

A colleague and I were once debating the merits of getting test cases peer reviewed, his position was that the peer reviews provided demonstrable value to the QA process and mine was that peer reviews provided questionable return on investment! After what seemed like a rather long time, the point came were my colleague said ‘well why do you think like that?’ The answer I gave was this ‘well, I believe that the process of writing test cases is how the tester becomes one with the system, learning about what its meant to do, how it should do it and so on...’ And therefore the reason that I question the value of a peer review is that more often that not, the reviewer doesn’t have enough detailed knowledge of the system or functionality to critic the test cases.

To which my colleague replied ‘that a bit Zen Drew!’

From that moment on I’ve seen the process of developing tests as a learning exercise! Needless to say that many years down the track I have used this apifenany as the foundation of my approach to software testing. I still believe the fundamental reason for documenting test cases it to acquire the knowledge about the system so that when it comes to execution the tester is able to evaluate the output and determine if it’s valid.

I’ve used these fundamental believe to underpin the methods and processes I’ve implemented at various sites, and no more so than when I started a new job at an organization that uses ‘Agile Principles’. My introduction to Agile was rather confronting at the time, coming from a predominantly ‘waterfall’ back ground.

Being somewhat new to ‘Agile-iterative’ development I search the internet and text books reading about Agile and principles of developing software using these methods. I learnt very quickly that it’s important to note that no two implementations of Agile are the same, Agile is a philosophy which embraces’ certain values. So when I talk about our Agile, it’s going to be different to your Agile and also different to their Agile, but look beyond the detail at the bigger picture about the process of learning.

The most important lesson I learnt rather early on in this new job was who to turn too if something didn’t make sense, sometimes it was the development team and other times it was the business customer. This isn’t usual – in every project I’ve worked on there was a (or several) key people whom where the fountain of knowledge, the difference here is that there was no specification document to scribble on! It’s not that documentation doesn’t exist, rather that it doesn’t exist in the same format as most people are used too! In many ‘Agile Shops’ the specifications are living – contained in stories that grow as the system grows – this is a story for another time. As the months past and I came to terms with ‘Our Organsiations’ development model I found myself questioning and comparing, trying to align the processes we were using with the ones that I’d used in previous testing engagements’. Other than conducting the testing of our applications I was also attempting to teach a new tester the fundamentals of testing, trying to apply the teachings’ of my classically trained testing methods in a world where the fundamentals’ are rather different. It was through this teachings that I came to the following conclusion; the differences in our development model when compared to the previous testing models was that 1) the time to learn the system (i.e. develop test cases) was far reduced and 2) the primary source of information for the learning’s was from the master’s within the project.

And so was born the analogy which I use regularly, in regards to learning the system I like to describe the difference between test case development in an Agile type in environment is that the knowledge is acquired through the learning by doing, somewhat hands on akin to an apprentice learning through an apprenticeship. The similarities are many, an apprentice begins there learning’s with an idea about the job they are going to be doing, and then is set small tasks by the master. As the apprentice learns and grows as a trade’s person, the master increases the complexity of the tasks and the apprentice is more capable to complete the tasks. The hands on learning is supplemented by small (in comparison) structured teachings, which are then reinforced on the job. Testing within our development process involves a very similar structure, at the commencement of the iteration the testers gain an understanding of the features that are to be implemented. Then as the iteration progresses and the stories begin to be delivered the tester tests and learns, then tests some more. The job that the tester does is overseen by the business customer and further lessons are provided by the developers. As the system grows, so does the testers understanding. The other important point to note is that through the apprenticeship, the number of text books studied and volume of notes created is minimal and when you compare this to the number of texts and volume of notes that a University produces and the picture is somewhat complete. You see like a University student, a tester on a waterfall type project the tester attends design & information sessions (like lectures) takes notes and then studies the design documents (text book or course notes) and then produces test cases (assessment tasks). During the design and build stage (semester), this process is repeated several times and then when it comes to test execution it’s when the testers knowledge of the system is tested (Exams!). One other important comparison is the ratio of ‘students to teachers’ and apprentices to masters, in the famous words of Jedi Master Yoda ‘Always there are 2, no more, no less, a master and an apprentice…’ in our development model the number of people involved is much smaller to ensure the lines of communication are short and fast. Compare this with the university model where each lecturer may have anywhere up to 100 or more students and several tutor’s to assist with the learning. An interesting point.

So to summarize, if documenting test cases is about learning the system, then the some of the differences between Agile and Waterfall testing environments can bee seen as the differences in learning philosophies used in Universities and apprenticeships. Focus on the process of learning, not the outcomes that could be achieved through the learning method.

Tuesday, January 19, 2010

KJRA Summer School 2010, come fly with me...

Great food, great company and thought provoking discussions what more could you ask for?This was my first KJRA Summer School and for me it brought back many memories of my formative years attending a boarding school in western NSW. The location of summer school was the 'The Women's Collage' at the university of Queensland, which is steeped in history. The halls of residence where we stayed opened in the late 1950's and reminded me of the accommodation I lived in whilst completing year 12. At the time, the year 12 accommodation was the pinnacle, single rooms with a balcony. Oh I waited patiently "doing my time" for 5 years to ascend into "Hindmarsh". The memories of hot summers night with only a fan for cooling, sleeping without any covering flooded on back. However the most confronting thing about attending summer school at a women's collage was that in the bathrooms there was full length mirrors, and round mirrors over the sinks which didn't just reflect, but magnified ones face so that all the blemished became blindingly obvious! Back in the day, the bathrooms where I went to school had only a single mirror over one of the sinks for those of us that needed to shave! So being confronted with ones unshaven, just woken up reflection was a memory I'd rather not have gained!

I do love the boarding house/summer school/residential school setup where the focus is on community and communal activities. At school, as at summer school you never ate alone there was always someone too share a conversation at meal time. Talking about the events of the day, sports or conquering great plans for the nights activities! and that's one of the other great things group activities that inspire laughter and cooperation. Back in the day we played pool in the common room, or went 'out bush' into the forest in our younger years building cubby houses for all manner of adventure, and during summer swimming in the mighty Murrumbidgee river was a great way to pass the time and keep cool. At summer school some of the old favourites were played the egg and spoon race, sack races and the three legged race began the evening. Then we fast forwarded to what seems to be the in thing at the moment with Guitar Hero - World tour. After several rounds, each increasing in difficulty Todd was crowned the guitar hero hero! That was Thursday night, Friday night was 80's night and it was fabulous to see some of the attendees dress up! Oh the fashion crimes that occurred in the 80's :-)

I finished summer school on Saturday, checking out of my room which incidentally bore a plarke on the door frame "The John Graham Miller room, endowed by Mrs Rina Miller" which I pondered for some time. It wasn't the only room to be named after a bloke oh if only those walls could talk!!!

I sat in on the Test Project Management course for the morning which was the source of much inspiration on the trip home! On Friday whilst delivering the Agile Testing Course I quoted a Prince2 trainer (somewhat strange I know given some peoples views on Prince2 and Agile) that I'd heard say "The Project Management should be like a modern day pilot (with all due respect) in that he/she should plan the flight, get the plane ready, take off and then switch on the auto pilot - monitoring the progress at key points making corrections as necessary. Then plan the landing, land and disembark the passengers". In the context of Prince2 it made a lot of sense and on the flight back from Brisbane I juxtaposed some of my experiences as a test manager to that of a pilot, so come fly with me.

Thinking of myself as a pilot is an interesting thought and gives a much better visual than when I tried to explain the role of testers (many years ago) to a group of developers as 'life guards' to which they replied "in speedo's? LOL". But anyway, my comparison starts with the pilot receiving instructions of the flight that he/she is to take charge of today. I imagine there are several key details that the instruction contains: the aircraft, destination, departure time, flight crew, number of passengers and expected weather conditions on route. In my experience as a test manager I've received similar briefs about the project I'm going to be managing; intended release date, commencement date, testing resources allocated, scope of testing and warnings about the political landscape!

After receiving the flight instructions even the most experienced of pilots will research the route, weather and all manner of other details (assuming the flight assignment details are allocated several days in advance of the flight) steadily building the plan of the flight. On the day of the flight, the flight plan will be finalised with details of the flight path, check points, , altitude, number of passengers, expected take off weight, boarding gate allocation, expected boarding time, flight deck and cabin crew assignments.

In a test management context, this period equates to the period after you've received the initial project brief through until your troops begin arriving on the ground (covering the development and documentation of a master test plan). You will have discovered all manner of details about the project, including who the key stakeholders are, key stage gates (i.e. change control windows), allocated manual, automated and performance testing resources, environment availability, development and requirements status. All of these (and many more) elements are all considered in the development of a test plan, and each influences how the plan is implemented.

Now with the plan matured it's time to implement. In the case of the pilot, a pre-flight briefing might be the vehicle to bring all the aircraft crew up to speed. Then it's off to the boarding gate to commence the pre-flight checks. The process of getting the plane off the ground involves numerous activities, in my comparison I've focused on the following; the boarding of the passengers (Test Cases), load the catering (Test Data), Storing the luggage below and most importantly load enough fuel (defects) to complete the flight! To me, the process and activities surrounding taking flight are somewhat similar pre-requisites for commencing testing. Ideally at the commencement of a testing program the parameters would be as well defined! Can you image knowing at the commencement of testing outer limits of scope and duration? With a plane its absolutely defined, as a plane can only fit X passengers, has a maximum take off weight and maximum distance it can travel, therefore limiting overall capacity.

The nature of testing often prevents such well defined boundaries, but that's not to say we shouldn't have a detailed understanding of the testing we are about to undertake. In my comparison I've compared the passengers to test cases, and the reason being is the success of a flight can be measured in terms of did it arrive on time which is a quantitative measure, and perhaps more importantly were the passengers satisfied with the quality of the flight in terms of service, comfort and timeliness. Test cases also have quantitative measures such as 'were they all run' which by its itself doesn't provide enough depth to make a definitive statement about success. One needs to look deeper into the results, how many passed, how many failed and what was the magnitude of the failure? More on that later.

Once all the passengers, luggage and catering is loaded its time to depart! Even as a test manager I still get excited when a new build arrives in the test environment, its like the anticipation of taking off once you've boarded a flight you know you ever so close to taking off and you just want to be in the sky! Once clearance to taxi is granted we push back from the gate and taxi to the runway. At this point in the testing process I like to have the test strategy signed off, test cases reviewed and accepted - essentially all my 'Entry Criteria' met so testing can commence. The take off, by far one of the most exciting or scary parts of the trip, as the plane accelerates down the runaway and the G-force pushes you back in your chair and then the point where the plane lifts off the ground - I'll never get tried of that feeling! Often there is a little bit turbulence or bumps as the plane assents into the sky, in the same way as the first delivery of the 'build' often experiences some difficulty being deployed into the test environment. And once your in the sky and the seat belt sign goes off you know you're on the way... This is the equivalent of getting the green light to start testing and so it begins!

Now we're in the sky climbing to the cruising altitude and constantly monitoring all manner of things - weather, wind, cabin pressure and fuel consumption. Fuel consumption I found has some interesting similarities to defects - what tha I here you say. But think about it this way, as the requirements and code are developed defects creep in and steadily build up, then once testing commences (at any level) these defects are discovered and removed. Compare this to the fuel for a plane. The fuel required to make the flight is loaded pre-take off, then as the flight progresses the amount of fuel consumed increases to the point where by the time the plane lands the fuel load is minimal. This is exactly the trend that we want to observe with our testing, that at the beginning of the testing cycle the defect detection rate is high and then as testing progresses the rate decreases, corresponding the number of residual defects decreasing. After all defects are the fuel of testing! While there are defects there will be testing :-) it's not a perfect model, but I like it.

Whilst in the air the passengers consume the catering and the plane steadily makes its way to the destination. The pilot (or co-pilot) monitors the progress along the planned flight path making adjustments as required and all if fine. Then out of no where, that clear trouble maker of the skies strikes, turbulence and has the capacity to effect all on board. The pilot has several options depending on the severity encountered, turn on the seat belt sign, adjust the altitude and/or the flight path (as happened on my flight). In terms of testing, turbulence could be equated to a change in scope mid way through testing, or test environment issues or requirements or code instability. Any of these (and many more) issues often result in the test manager having to make adjustments to the plan! and its just a natural part of the process.

Newtons Law states "What goes up, must come down" and as the destination draws closer its' time to prepare the cabin for landing. All passengers need to return to there seats and place luggage appropriately. Throughout the flight the pilot is in constant communication with Air Traffic control, reporting location and status, and as the pilot prepares to descend, permission to land needs to be sort. The air traffic controller can be compared to either the stakeholders or the project manager whom need to be kept update to constantly about the status of testing -also making decisions in regards to ending testing. In terms of testing this is the point in the cycle where the test manager starts to make sure all is in order, all test cases have been attempted, the defect status is published and understood, requirements coverage is assessed and the test report is in draft. As with the take off there are many factors that influence the 'smoothness' of the landing and the pilot must continually monitor and adjust whilst approaching the runway. In this period even minor issues can have disastrous effects, but more often that now the plane touches down with little or no noticeable issues to the passengers. Once safely on the ground its time to taxi to the terminal and disembark the passengers.

Disembarking the passengers is a key 'milestone' of the flight and its at this point where the quality of the flight can begin to be assessed. Flights like testing cycles have both qualitative and quantitative measures of success. quantitative measures like all statistics are open for interpretation and qualitative measures are subjective so its important to gather both for consideration of 'success'. Its also important to have a definition of what 'success' is at the commencement of the project. Many would argue that simply reached your destination is only part of a satisfactory outcome!

With the plane on the ground, passengers and their luggage disembarked the pilot must complete some paperwork and then its off onto the next assignment. As the test manager wrapping up the testing cycle, the test report is usually the way to close out the testing for this project...

Wednesday, December 30, 2009

Team building on two wheels...

As luck would have it, all of the KJRA consultants who actually live in Canberra own, and ride cruiser style motorcycles! Michael rides a Suziki Boulevard, Neil rides a vintage Yamaha Virago and I ride a Harley Fatboy which as Johnny and Behzad said 'sounds evil'. So in the spirit of team building and hitting the open road those of us (Michael & I) who were on leave headed out for a day's ride.



The rendezvous point was Bungendore which is about 30 mins drive from Canberra on the way to the coast. We had previously decided that rather than battle with the hoards of Canberrians heading to the coast for New Years that we'd head in land and so Tarago was the next stop. The ride is along well sealed roads country roads and accordingly there is minimal traffic. We pasted the newest (and largest) of the districts wind farms and it is just amazing the size of the turbines they are huge. I counted 15 towers, but apparently there is actually 25 in total. Once we reached Tarago 'The Loaded Dog Hotel' was our pub of choice, its steeped in history having opened in 1848 and really there is not much else in Tarago! We had a yarn with the publican, who was most interested in the 'hoops' one had to go through to get a bike license these days. There was a couple of friendly locals and a hand full of other bikers, but the road was calling!



From Tarago we headed for Goulburn and then on to Crookwell, passing by another wind farm which was much smaller than the one we passed earlier. Crookwell was the designated fuel stop for both man and machine, and I thought getting a Pie n sauce might have been an easy option for lunch. However it seems it would have been far easier to go for a pub lunch. The first bakery we stopped at looked open but was not, and then the second bakery we found was open but didn't sell pies or sausage rolls - which was most curious. To my way of thinking this makes it more of a cafe than a bakery. The lady in the no pies bakery said if its pies your after then try 'Barkers Bakery' down the street and so with this vital piece of local knowledge we pushed on in the quest for the great Aussie lunch. Finally we arrived at Barkers and surveyed the offerings of pies on the menu boards, and I asked for a steak and mushroom pie only to be disappointed once again. Since it was getting close to 1:30pm the pie stocks had dwindled and alas no Steak and mushroom pies were left - Doh! So Michael and I changed tack, and asked "well what do you have?" luckily there was two pepper steak pies left so Michael and I didn't have to go toe to toe over who ate and who didn't.


Now that the men were full, it was time to feed the machines and head for home. The road between Crookwell and Gunning is fabulous those of us that enjoy two wheels and loads of twist and turns, with some gentle rises and downhill runs for some great riding. As seemed to be the pattern thus far we passed yet another wind farm and we quite rightly questioned if we'd joined the southern NSW wind farm tour! At the end of this short run is Gunning where three more bikes just out for a run tagged along as we headed too Gundaroo, our final stop. To keep our stops consistent, we pulled into 'Matt's Wine Bar' for a quite ale.



My final comments relate to bugs (well we are a testing company!). Along the some of the 250 odd kilometers we covered we discovered more than a few bugs! There was the bugs that went SPLAT as they hit my sunnies, the ones that went THUD as the collided with my helmet and the rather interesting ones that caused me to go OH Sh!t that hurt as they smacked me in the cheek. Though none of these compared to the legend of the Christmas Beetle that the publican at the Loaded Dog spoke about. He said he'd seen results of an encounter with one of hovering bricks that had crunched into a fellow open face helmet, Harley riding compardray and it wasn't a pretty site! A welt as big as your elbow, squarely in the middle of his head. And he said if you looked close enough you could see the outline of the jolly fatmans' face in the mooshing of blood and skin were skull met beetle! I'll just have to trust him on that one, but needless to say I was on high alert watching for the hovering beasties and even ducked under a couple to avoid a collision. We were within 50kms of home and blasting down the Gundaroo Rd when out of no where BANG! a bug of some sort smashed into the lower rim of my sunnies, and as if that wasn't scary enough then I felt the wriggling and squirming of the bug behind my glasses on my eye lid - seems this bug was not done yet! It would have been so much simple had he just sucked it up and died when he got split in two by my glasses, but no he wanted to fight some more. Luckily for me he was under the right hand side of my glasses and easily squished with my left hand (the now on the accelerator!).


All in all it was a great day for bike riding and a fitting start to a new KJRA Canberra tradition of two wheeled team building. Until new time stay rubber side down and shinny side up!


Saturday, December 26, 2009

It's all about testing!

As I look back on the year that has been and ponder the year ahead I have a smug little smile on my face, and not just from the Christmas cheer that I've over dosed on!

The project that I am currently working on is like any other, multiple systems, tight deadlines, thorw in the mad rush to get all the loose ends tied up before the Christmas break and its pretty much situation normal.

So for several months now I've been telling anyone that will listen, that "the sooner they realise this project is all about testing and that my needs are the most important the better off they will be!" Self centred I know, however I see a large part of my role is keeping testing at the front of every ones minds. Some days I feel like a little kid saying "don't forget about me" but if that's what it takes to ensure that testing is not left out of the information/communication loop then I'm ok with that.

Well in the lead up to Christmas I received a couple of most welcome, and unexpected Christmas presents partly due to my nagging. Both the project manager, and the software engineering manager commented separately to me "This project is all about testing" and without too much of a "I told you so" look on my face I replied with "indeed it is". It's amazing how the acknowledgement of the importance of testing to the projects success lifted the spirits in the run to up Christmas break.

I've been able to establish several reasons for the realisation of the importance of testing to the project. Firstly, the sign off of several key testing areas (performance & SOE) are absolutely critical and not negotiable if we want to release our software into the production network. Secondly, acceptance of the software by the stakeholders is largley based on the results of the final acceptance testing; and sign off of the Acceptance Test Report is therefore a key milestone. Finally, because I said so, and will continue too remind them.