The future of testing as seen through the eyes of an Iqnite attendee

 caught up with several of the delegates before dinner and I asked them "Is the cloud going to change the way we test?" the first response I received was "It already has" obviously I asked how? The response was simple, he said "It allows me to operate my business from anywhere, anytime; I am now able to run large scale load tests for my customers cheaply, using load generators around the world". 

I was lucky enough to attend the VIP breakfast where Dennis Bass (Keynote speaker and Microsoft – Test Architect, Visual Studio TFS) and Michael Palotas (Ebays head of quality engineering) shared their thoughts on were testers, and testing in the clouds is headed. Both of these fine fellows where in violent agreement that the key area of influence on testing going into the future will be the speed. Dennis called it "Cloud Cadence", the speed at which your customers and clients will demand innovation and expansion will increase significantly. 

Michael highlighted that a ebay they are moving into a development cycle that has no real beginning and no end, it's constant. Developers will be able to develop a feature in the morning and have it live in production by the afternoon. This sort of pace and rapid deployment has to involve a change in testing methods and approaches, as quality is paramount as is timeliness. Michael's view is that the line between testers and developers will blur, and to some extend it already had at ebay. He said his best testers are also the best developers, they just choose to be testers. Testers will have to be capable developers, just with a testing bent! Michael spoke of "Quality Engineering" rather than quality assurance or quality control.

Dennis shared a similar view of the future, and in fact its reality at Microsoft already were the developers are testers and the testers are developers, all working together to deliver a quality product to the clients. Dennis highlighted that testers will into the future be required to triage and investigate bugs beyond simply proving it's repeatable. Both Dennis and Michael advocated using methodologies as and when appropriate to the organisation. For example when Dennis is conducting a load test on TFS he has the developers sitting with him to triage bugs before they occur – for Dennis the worse thing that can happen when he encounters a problem is for a bug to be raised, the value of a bug is only realised once it's fixed – collaboration is king. Dennis's final words were something like "Testing cannot be a bottleneck or barrier, otherwise they will go around or without us". The reason for the comment was that if the business wants a product to ship in say 48 hours, the last thing they want to here is it's going to take 2 weeks to test it – this will mean we need to be smarter about the testing, more risk based testing and acceptance of risk. Testing needs to keep pace with development techniques, keep up or be left behind…

Dennis shared his insights and experiences gained from moving Team Foundation Server into the cloud as Software As A Services. I was amazed at some of the lessons, as they seemed to be the inverse of why we would want to move to the cloud. For example, Dennis said that everything takes longer in the cloud, and it costs more! Why you ask, well because it's in the cloud and distributed who knows where moving large amounts of data around is not as efficient as shuffling it around your local networks. The cost was an interesting point – it costs more to test in the cloud  because its an actual cost – as in the cloud provider will be sending you an invoice each month which has to be paid. It's not like having your servers locally and maintained by the server team who costs are hidden away in an operational budget. 

Some 'got-yas' which Dennis imparted included: beware, the cloud is immature,  evolving and largely uncontrolled. As such when your testing in the cloud things are changing servers are being rebooted, patched deployed and your unlikely to be aware of any of it! He stressed "you are not alone" the cloud is the cloud and by definition is it a service used by many and this may impact how your bit of the cloud is operating. Applications will need to be better able to be debugged without access to the servers – as in the cloud you are unlikely to have access to the underlying servers. Operational testing is a must, the applications must be tested for scalability, shrink and growth – applications will also need to be robust enough to operate whilst the underlying systems are patched.

What more would we need to do?

KJRoss & Associates is running the ASTA awards to recognize the outstanding work done by testers within Australia. 

Now testers like most IT professionals are not often self-promoting and will need to be nominated by peers, colleagues or clients. I heard of a situation were a manager was asked, “Do your testers deserve an award?” via email, and the response was “No not really”. This got me to thinking; well what would a testing team have to do to change this response? Find more bugs? Test quicker? And I must admit I’m at a bit of a loss... But I do take heart in that, rather than just ignore the email and its contents the manager actually responded. Is the response, though not worded so, a complement to the testing team indicating that the team is front of mind and is doing a good or sufficient or adequate job just not an outstanding job?

Testers and time poor tourists how much do they have in comment?

The modern tourist rushes from landmark to landmark taking photos to put on Facebook so all their friends can ‘like’ and comment on the magical time they are having; Testers rush around executing test case after test case bouncing from requirement to requirement stopping only long enough to prove it’s been ‘delivered’. Sure defects get in the way, just like the weather or traffic effects the bus tour; but as with a time poor tourist who isn’t able to change their itinerary they must press on to the next stop, after all we are on a schedule! Oh, and if you thought there would be time to double back and visit the thing you missed because of poor weather, well guess again. Today we are going to the next “must see” over there and tomorrow we’re in the next town and the day after that well you’ll be on the plane home and back to work. But never fear, you have the photos!

So lets reflect a moment yeah the tourists have ‘been there’, they have the photos and stories after all but have they really experienced the place? They have eaten the local cuisine, at the hotel but how real is it? From personal experience travelling around Thailand we ate at ‘authentic restaurants’ but there was very little in common with the food being prepared on the side of the road on mini gas burners attached to push bikes. It’s unfortunate that the tours don’t often allow time for the tourist to get of the beaten track to experience the real essence and character of a place. Testing is much the same rush rush rush, test the requirements, find the defects and push the release into production with little if any time for regression testing. Is this effective testing? Are we getting off the beaten track like our users will?

If testers are the time poor tourists then are our users the grey nomads? Just cruising from place to place as they feel like it. They have time to get off the beaten track, or stay another day if the weather doesn’t allow them to go fishing or sailing or play that fabulous golf course. Or are our users akin to the back packers who just lob in to the cool places, party a lot and work if required then just move on when well the mood takes them?

Testing is the cause of poor quality software

I'm going out on a bit of a tangent for this one but I think that's its got some merit.

Recently I was part of a conversation about all of the issues we'd discovered during a testing cycle. The back story is earlier this year a new test environment was built and we tested it. We tested the environment in-line with our current practices which is to run a suite of tests that tests all of the major components and system flows. This test suite can be completed in roughly 4 hours. If we were to run our complete system test it takes about 3 weeks and so taking a risk based approach we opted for the quick test cycle. This decision was predicated on the fact that this environment would be build using tested and approved build instructions which have previously delivered a stable and complete environment.

Fast forward to the conversation mentioned earlier. It went something like they said "Didn't you test it properly?" I said "Yes, we followed our standard process and ran the quick cycle tests" they said "well that's why all the defects, we need you to test it to make sure it done right." I said, well nothing... Pondering the conversation has lead me to believe the testing is the cause of poor quality software. Why, well because the folks who build the software rely on testers for quality control rather than focus on quality assurance themselves; With timelines never being long enough to build a proper solution of course there isn't enough time to test it before it goes into test. But that is ok, that's why we have testers, they test and they'll caught the bugs before they go into production.

So tonight, my thoughts are along the lines of if we get rid of the testers, then the folks who build the software will have to build and test; therefore the quality of software will increase and shift away from the compliancy of the testers will catch it! After all those testers take so long to test anything...

Are we asking the right question?

I'm part way through reading Freakonomics by Steven D Levitt & Stephen J Dubner (Gr8 read just quitely!). In the opening "economics is a science with excellent tools for gaining answers but a serious shortage of interesting questions..." and this starting me wondering are we asking the right questions in software testing? I mean every day I get asked how's our defect count, how close are we to finishing testing, how many tests have you run, how many are left, were is the test report. All of these are valid but really uninteresting questions (I think!) rather blah and binary, and really do the answers to these questions really tell us much? There's a line in one of the Disney movies (Shrek I think) in which one of the characters supporting characters said "blah, blah blah" and then the main character said "yes he's talking, but is he saying anything?" Which is kind of the way I feel about answering the daily questions. I give the answers, but without understanding the meaning behind the numbers or tends am I really saying anything? Shouldn't we be being asked about business processes or risks or impact on revenue? These are much more interesting but I think based on the freakonomics outlook something like ... help help me to find an interesting question!  

more to come...

Are we testing the right things?

Recently on a plane back from Adelaide I read an article in the QANTAS in-flight magazine the Australian Way (as you do!) the article was written through the eyes of a traveller who was visiting somewhere in either Canada or North America. The bits of the article I found most thought provoking was the quotes of the tour guide who said something like 'focus on what you can see, stop and admire all that is around you; don't lament the that you didn't see a bear, be thankful you have seen over 100 different types of plants and animals.' The guide went on to say 'all too often tourists visit these parts and rush from place to place to see this and that, taking only photos of things, not creating a memory by experiencing being there'. This made perfect sense too me the phase "Stop and smell the roses" came to mind. By chance tonight I saw the tagline of another article Look Beyond the Lens on the Sydney Morning Herald's website and after only reading the tagline I pondered.... As testers, in testing do we sometimes focus too much on proving that a requirements have been implemented or that defects have been found as a result of our efforts - to prove we were there? Do we focus on finding the bear in the woods because it's big, scary and easy to see or not see, all the while missing the wondrous smaller wildlife, flowers and trees that are necessary parts of the ecosystem for the bear to survive or equally perish?

My initial thoughts were along the lines of do we focus too much on finding bugs and satisfying requirements (taking pictures - see it is there, I saw it) only to forsake to the things that our users actually need from the system to do there job or improve their efficiencies? And if this is the case, then how do we change our plans to allow us to stop for a while and just "be there"...  Somehow I think the need to take pictures and wiz off to the next location is a result of time constrains and a desire to see as much as possible in a given period of time. Equally in testing, time or lack of time is a huge factor. But it is time that maybe the ultimate measure for a picture only lasts for as long as the media is around and often fades over time (remember when they used to print pictures!) while a memory can live forever being pasted from one generation to another! It's time to choose which we value more, pictures or memories...  
  

The Missing Link, found sort of!

Dude, where's my link? would also have been an appropriate title for this blog entry! I came across this little gem over the weekend - it's a beauty a double fail, firstly the link colour has been set to 'white' when not active and yellow when active. Neither of these two colours work at all well and makes it easy to see why blue link text is the default. Given the location of this particular link (within a tender site) I suspect that its a section of a document that has been copy and pasted in and relies on a style sheet thats not present in the destination website - but that's no excuse for not proof reading!!!

Go where?

Here?

The Law of Gravity

Have you ever sat and thought about something so random as this...

I've had this reoccurring thought recently and I have no idea why, but as we were driving to Sydney over Easter I thought to myself - "How important is gravity to our modern societies concept of law?". So over the past few weeks I have continued to ponder this thought. I seem to recall a car on the side of the road and thinking something like "once we have cars that can fly or vertical cities like the ones in star wars and other such fantasy worlds will we have to change the laws we have now? At the moment if you own a car, you can park it in your driveway or garage and people know its your because its on your property and its our friend gravity that is keeping where you parked it. But imagine with no gravity, you bounce or float out of your car, close the door and make your way inside, only to notice sometime later your car floating by your window, with your neighbours with it - just floating around. I'm sure there won't be too much of an issue in the local area but what about if you visit a friend in another part of town and you have to find your car - you could just claim another one? On reflection cars are not the best example, they have number plates etc so lets think about pot plants or other non-identified house hold items. We have heaps of pot plants around the outside of our house what if they just floated away and we had to fetch them? People know they are ours at the moment, because they are at our house. Not sure what you think, but I am sure that there is a strong link between the law of gravity and the law of common society and possession.

Thoughts provoked by the Information Age (ACS magazine)

There were several articles that tweaked my interested and awoken the sleeping grey matter from its weekend slumber! The first was one titled "Investing in careers with SFIA" which describes how Westpac has incorporated the Skills Framework for the Information Age (SFIA) as part of managing the 'whole employment life-cycle'. SFIA has been at the front of my mind recently as it's featured in many discussions I've had with clients and service providers alike. Initially I was drawn into reading this article because of the relevance to my current activities-discussions which centre around the implementation and benefits of the framework but two-thrids of the way through I found the paragraph circled in the picture below. My first thought was of all the skills in SFIA (there are 89 odd skill areas) the Westpac executive choose to highlight testing, not Development, Not BA's and not Project Managers but Testing. The article didn't explain why testing and not the others, but what I liked was that his person understands that testing is a Professional Skill, with a career path just like any other the skills in the Framework. At the recent Test Managers Forum there was lots of discussion surrounding professionalising testing and I am now seeing evidence we are heading in the right direction! My thought is that the next step will be to have the difference streams of testing (Performance, Automation, Functional) recognised as skills in there own right - more too follow on that one.

The article can be found on page 38-39 of the Information Age.

The next article that captured my attention was  'ICT's biggest money wasters' (pg 46-49). Unfortunately in the software testing game we all too often see evidence of the number 1 cited cause of waste: "Dusty Software Licenses". For the most part I've seen this occur in the niche areas of software testing Automation and Performance Testing and they are usually accompanied by/or caused in part by number 1's closest friend, number 6: "ICT projects gone wild". 

All to often I have seen organisations spend Thousands of dollars on tools to make the testing process more efficient and end up delivering far less than was originally specified. There are many reasons for this result, the least of which is that within IT we often attempt upgrades and process improvement activities internally and we don't apply the same level of rigour and governance that we do as when we are delivering project for an external stakeholders. We often put the tool subject matter expert or super user in charge of the project, on top of there other duties and expert that it'll just happen. The lesson here is that all internal projects (upgrades, automation and performance testing) should be run as projects with milestones, reporting, have established baselines and be constantly measured, with individuals held to account for the time & $$ invested. 

Teleworking is a subject that is close to my heart, as I Telework for one of my jobs :o) It works really well for me (and my employer I hope!). I enjoy the flexibility to do what needs to be done in and around the other things in my life (Family & my other job). The part of the article that made me smile, and then ponder was the part were the author said the 'non-productive time' of the Teleworker can be put towards doing the household chores like cleaning, doing the dishes or preparing dinner - the tasks that usually get done during personal time. This made me go hmmmmm..... Initially I was like, well, that's not cool and as an employer and supervisor I'm not sure I agree. But on reflection, and when considering the alternative proposed in the article (surfing the net or chatting) I'm now thinking that if that is the case, well it'll save us some $$ on bandwidth and download fees and potentially stop the disruption to the other employees and causing them to be unproductive so maybe on balance it's not such a outlandish statement. Though, Boss, if your reading this, this is my only 'unproductive time' ;-p     

A post about something.

It's been a while since I've blogged and it's not because I've not been inspired by many of the things I've heard and read. It's because there never seems to be enough hours in the day! Maybe tonight might be the night...

- Posted using BlogPress from my iPhone

Thoughts from the 2011 Test Managers Forum

What a great few days, as I write this I am sitting in the Gold Coast airport terminal waiting to board the airplane home. It's just a bit nostalgic, as the first blog post I wrote on the way home from the Test Managers Forum in 2009. The theme of that post was testing and aeroplanes (come fly with me!). As was commented by one of my colleagues today - my posts are sometimes a bit random, but always related to testing, somehow. I expect that this post isn't going to be any different ;-)

I've been lucky enough to spent the last two and a bit days with a room full Test Managers from some of Australia's biggest and most reputable organisations. The Test Managers forum brings together like minded testing professionals to discuss, debate and deliberate on a wide range of issues facing us in our everyday activities. The hot themes this year involved resourcing, testing environments and requirements; it seemed no matter the size of the organisation we all suffer in the same ways. The good news, is the collective wisdom of all those present at the forum were able to shed some light as some of the ways we might be able to cope, over come and prosper when faced with these challenges. Of particular note for me were the discussions on tester Certification and tightly coupled issue of Testing as a profession. Equally I stretched my grey matter when thinking about resources, recruiting, retaining and training. 

I've previously blogged about changing the focus and emphasis of the certification process; moving towards a system of professional recognition and on going development; much like many of the other professions (life savers, swimming instructors and engineers). It seems that my views are shared by the wider testing community that were represented by the delegates at TMF. Though we didn't resolve to change the world - we collectively agreed to raising the bar …. or did we? With representatives from ANZTB, CSTP and the ACS on the panel discussion centred around certification being the 'entry' or the start of the learning process, and that there is no substitute for real life experience. However, for those of us that have been testing on the front line for some time now, and who are certified testers the discussion centred around external recognition of our skills. And this is the point for many of us, the certification might not open the door to a new job but may be the differentiator that gets us the job over someone else who is equally capable but is un-certified. There was talk about organisations becoming certified, which looks like it will happen but is at this point a ways off being a reality. The benefit to the organisation is creditability, it may not mean that you are ganuteed to get a better result but as the argument was elegantly put "If one of your mates has a drivers license, and one doesn't who are you going to ask to drive you to the shops?" Well that should be a no brainer! I personally believe the certification has its place in our industry, and will assist us to gain the recognition we aspire too.

The topics of recruiting, resourcing and retaining is tightly coupled to the discussion of Certification, as to get certified one should attend training. However the focus of my thoughts out of this discussion centred around the salaries that testers are demanding/commanding. As a person who's been in charge of hiring and firing for some time now I had some views on it. My general principals when hiring is that salary should be closely aligned to the skill set and role that the person is to perform. Nothing new there. For ease of discussion, I'll start with contract rates; I would generally pay a contractors $1 for every $1000 of the equivalent pay point of a permeant officer. So within the government space, if you were a contractor performing a Test Analyst role equivalent to an APS5, and an APS5 received a salary of $55,000 per annum then $55 per hour was about the mark. I found that this system worked pretty well, and party because we had a good description of what each of the members within our team would be doing, and an outlines of what level they would be expected to operate for a particular APS level. But I was thinking whilst at the forum about the other end of the spectrum which was discussed around our table, that in some organisations an APS5 might be a test lead/team leader and many of the people simply won't apply for a job at that level with those skills because it doesn't pay enough. This is not a new problem, but I thought of a new (I think) way to tackle the organisational 'we don't get paid enough' situation - Warning - This theory has yet to be tested and therefore may have some errors in logic or bugs in it… During the conference we looked at a case study about the cost of defects which slip through all the testing and get into production (on average 10% of defects are detected in production RossReport2010). In the example that was presented the cost to the organisation to 'fix' the defects in production, not accounting for the opportunity cost or loss of revenue was roughly $9000 per defect. $9k is a pretty sizeable number and it got me thinking - if you had to fix 6 defects in production in a year, that equates to $54,000 - do you see were I am going with this? What if, in simple terms you worked out that one more tester in the team would be enough extra resource to find those pesky 6 defects before production? Then that would make for a compelling business case, and in fact might be 'cost neutral' or better. So applying the same logic to a team that is struggling to attract resources because the pay scales are not comparable with the rest of the industry in the area, might try using a similar business case, but instead looking at defects holistically across the SDLC. We all know defects are cheaper to fix earlier, so let's say a defect during test costs $1,000 to fix then for every defect detected and fixed before production there is a saving of $8,000 so if we found 100 that's $800,000 saved. So that value of testing to the organisation could be expressed in terms of (cost of testing - savings delivered by testing) = some amount. You would then have a value conversation with the powers that be, saying that if we were paid just a little better, we'd attract a better class of tester, thereby increasing the quality of our testing in turn delivering greater value to the business and not increasing the actual costs to the business. It's worth a shot, but it has to be a value conversation not just a "we need better rates conversation…"

Arrggg - have I been testing for too long?

I came across this little issue just last night when I was filling in a form. It took me three goes to get it right!

 The thing that brought me undone was that the error message text in red has a space between the time and am/pm. So that's what I did, BUT the validation on the field only accepts the time if there is NO space between the time and am/pm. Not sure who tested this one...

Anyone can be a tester

I came across a situation a few weeks back that I have faced many times before. There is a  project kicking off and it's going to require testing - excellent I should be able to help. Being a passionate tester and test manager I asked if they need any help, after all there are several 0's in the project value. The project leadership team said 'no, we are all good the team from blah office are available'. No problems I say, let me know if the situation changes. Being a tester, and to be frank, a nosey little bugga I did some research, it turns out that the 'Test Manager' has 'Solutions Architect' as her current role in her email signature - hmmm. A little more investigation and the 'Testers' are listed as 'System Engineers' as is the 'Performance Tester'. I was feeling rather deflated. But it got me thinking - why is that so many people in IT organisations seem not to respect testing as a profession? I mean have you heard of a tester being asked to come on board a project to be a SharePoint developer, or DBA?

I recall a presentation delivered by Dr Mark Pedersen, a colleague of mine at K.J. Ross & Associates titled "Selling Testing to the Business". In this presentation there was a slide which presented the findings of a study done by HP in the early 90's (Grady, 1994). I have included it below...

I've translated and calculated some more metric's into the table below. I like to look at things from a few angles.

Test Type	Defects Per Hours	1 Defect every ... Hours	... Defects detected in an 8 hours day
Day to Day	0.21	4.76	1.68
Black box	0.28	3.57	2.24
Glass Box	0.32	3.13	2.56
Inspection	1.05	0.95	8.4

For the purpose of this post, I am focused on the 'Day to Day' vs the 'Black Box' metrics. Based on the above numbers it's reasonable to infer that by using Black box testing technic's (and all other things being equal) we will find slightly more defects than if we just followed the business processes.

My hypothesis as to why so many IT people don't respect testing as a profession, and were the old saying 'Anyone can do testing' comes from, is based on the observations and experiences were folks from all manner of backgrounds and vocations have been involved in testing and found defects just by using the system in the manner to which it was intended... And here in-lies the challenge to the current and next generation of testers to advance the profession by improving our effectiveness - or maybe it's a case of simply communicating the improvements we've already made????

Happy New Year, 2011 bring it on!

On the brink of returning to the office fulltime I'm excited!

2011 is going to a big year for me personally as we embark on the construction of our new house. It's only just dawned on me, even though it took 8 months to develop and get approval of the floor plan, there is soooooooo much more detailed requirements are needed to ensure that the vision in our heads becomes reality. Stay tuned for more on that front as the build progresses.

During the break I received notification that my application to the Australian Computer Society had been accepted. I also received a copy of their magazine, Information Age. I was most pleased to discover the quality of the articles and the topic's covered were of such a high standard that I read it cover to cover, and instead of the novel I'm in the middle of reading! I particularly liked the articles 'Are you ready for 2020' and 'What's a head in 2011'.

I was lucky enough to finish by break by attending the K.J. Ross & Associates 2011 Staff Summer School. The summer school brings together the extended KJRA family from around the country for a 3 day sabbatical. Consultants flew in from Melbourne, Sydney and Canberra to join our colleagues in Brisbane at the University of Queensland.

The courses tort at Summer School cover our own CSTP Foundation; Leadership, Negotiation and Customer Service sessions presented by Rohan Toll. And for those with a technical bent, almost 24 hours of sessions covering various software testing tools (Microsoft VS2010, TOSCA and Silk Performance Tester). The technical stream contained a mixture of technical briefings and hands on training.

Finally I am excited about the program we are putting together for the ACS Testing and KJRA Test Managers Special Interest Groups. I am super excited to be able to announce the following presentation topic to be covered in 2011 (schedule to be confirmed):

• Dr Tafline Murane will deliver her Eurostar 2010 presentation The Carrot or the Whip - What Motivates Testers?; and
• Dr Mark Pedersen will present a Microsoft VS2010 un-boxing and master class; and
• The award winning DIaC Performance Testing team will take us through the journey 'Performance Testing Improvement Through Application Performance Management'.

And this is only the beginning!

QC 11 - Return of the Media....

One of my recent posts I described the trails and tribulations of an on-again, off-again migration to QC11, thawed at the last moment. Well now for the next chapter, all be it rather short.

I was lucky enough to receive a MacBookAir 11 inch for xMas care of my lovely Wife (Yay Babe) and in the process of transferring/consolidating all of my work files from the various Laptops and PC's used to use onto my little shiny MacBook I happened to stumble across a zip file "T7333-15006_1a.zip" in one of the directories. I thought it looked like a HP download and as luck would have it when I opened it it's the install media for QC10 - hooray. Now combining this little fella with the QC10 patch we located earlier and guess the move is back on!

Testing Utopia

In line with some of my recent posts about reporting, requirements, defects and test execution status - I think that I have narrowed down what utopic testing conditions would be. It's a big call but here is what I came up with; All of the...
1) Requirements (functional and non-functional) fully derived and understood,
2) Requirements covered by risk appropriate test cases/scripts/procedures,
3) Documented test cases/scripts/procedures executed,
4) Defects discovered either resolved or well understood and documented,

That's the beauty of utopia, its so great, dynamic and unattainable but well worth striving for...

I've reached the point were I have a vision of what this Utopia looks like and how I might need to report how far away we are from it. It involves a data cube which contains information about requirements, tests and defects. Each of the data objects has records attached requirements have links to tests and defects, a priority and risk assessment. Tests have links to requirements and defects, but also records of each time the test was executed (results), priority and complexity rating. Defects have a status, severity, complexity and business impact. I'm sure there is a few things I have forgotten, but they can be added. I think that I have been asked to provide a report based on all of these attributes at one time or another...

QC11 - The next chapter

This story begins several months ago, when I downloaded the trail version of QualityCentre 11 with only intensions of 'playing'. Yeah right! As with most software trails I played hard for the first few days and then the interest slid south. I liken it to the Garnter hype-cycle... I climbed to the top of the "Peak of Inflated Expectations" and then coasted down to towards the Trough of Disillusionmen - but on the way down I had several little bumps which again have tweaked my interest. I'm not disillusioned at all, rather beaten by circumstance - let me explain.

In my current role, like all that have come before, I am the administrator of our QC instance. As such when it begins to play up, then the server/infrastructure team comes knocking on my door. Recently my little QC has not been playing nicely with the other applications in the sandpit - namely SQL and MOSS. It appears the QC service is hogging all the CPU on the server and as such the other applications, which are also critical, suffer. All this means is that nobody is happy with the testing team!

The solution I thought was to simply capitalise on my investment in the QC 11 trail and install QC 11 on the new (well additional server) and then migrate the old projects from QC 10 on the old congested server to new, fresh and totally dedicated QC 11 server. Oh if it were that simple then I guess we'd have done it my now!!! The first hurdle I encountered was that the trail period for QC11 had expired - not really a big deal I thought I'll just build another virtual server and install it again, besides I could use the practice for when I have to do it for real. First mistake, foolishly I thought that since I'd done it before, blogged about my experiences it would be a smooth process. Seems I was wrong, I encountered several of the errors I had seen previously and a few I hadn't. This time around the QC Service failed do to a java issue - the 'java virtual machine failed to load'. I so tried at least a dozen different ways of setting JAVA_HOME but none seemed to work. I reverted to editing the .bat file with little success, at one point I even installed JBoss standalone. This was the turning point as I discovered that JBoss would run with all the setting as they were. Now having narrowed in to the problem (I thought) I investigated the jboss installed by QC only to follow it down a rabbit whole the likes of which Alice in Wonderland would have enjoyed. Though at the end of this wonderful adventure I discovered I'd been looking the wrong spot - doh. All of this adventure was on my virtual PC at home. We then tried it for real on the new server.

Not surprising we encountered the same issues on the new server as I had had on the virtual, this time we were quicker in debugging the issue. We were able t o get the QC 11 instance up and running, all be it through the .bat file rather than the QCServices. It was at this point that we encountered what would be the mount everest of problems. When we confirmed that QC had started we navigated to the url using the browser which was installed on the server (win2003) which happen to be IE6. A simple detail which I over looked in my excitement to install the trail, and the fact that all my playing was on a win2008 server which has IE7 as default. QC 11 requires IE7 or IE8 BOOM our journey had come to a screaming halt. The network that we work on, is closed to the outside world - entombed to operating systems of days long since gone. The governance of this network doesn't allow for changes to occur without a lengthy consultation process etc. So now we're almost back to square one but not quite and this is the kicker - the simple solution install QC10 on the new server seemed to be the obvious choice, but again the best intensions were thwart, no one can find our QC 10 installation media! Guess this is still a work in progress :-)

Interesting bug?

On the weekend just gone, I was the designated driver, driving my sister and brother in-law plus a few of there friends to a party in Double Bay. When I jumped in the car the fuel light was on, and the trip meter indicated we had a range of 60kms. As the trip progressed the range dropped to 50, 40, 30kms as one would expect. I was a little worried given I was in an area I'm not at all familiar with, but my sister in-law promised me it'd be fine - to quote "with a range of 60kms your in good shape!"

So after dropping them off at the party I started the return journey and the range decreased to the point where it reported the range was 0 kms! I wonder if this is a bug in the trip meter software - as the engine is still running, meaning there is still some fuel and hense some level of range. I'm not sure of the age of this car, it's not brand new and maybe the manufacture has fixed this in the next version? I know that in my car, which is a year old, once the range gets down to about 40kms the trip display simply reports 'Fill with Fuel' rather than a specific value - I guess that's one way of doing it!

Failed Test Cases/Scripts - is there another dimension we should consider?

My thought for the day - just because one test step fails, should the whole test fail?

Today I had cause to ponder the status of failed test cases. I'm sure you've been in the situation were as a test manager you've been asked if the system is 'ready' or 'is testing finished'. I was asked this question today for the umpteenth time. The situation was this, we are currently testing an environment which has been refreshed to the current production baseline (without data!) and we ran our regular 'sanity test' to see if everything was in the correct state. Situation normal. The sanity test set that we run are a high level set of tests that cover all the key functionality and integration points. Often we run this set of tests and there is failures i.e. Exchange isn't working or the search crawl hasn't run. Today when I reported the results back to the Mgt team for some reason the words coming out of my mouth didn't seem to make sense. The stat's were something like 22 passed, 4 failed and we are good to go. short pause, we've raised 3 defects and we are good to go. hmmmm, this is were I started thinking so we've got failed tests and defects but we are still good to go? The fact of the matter is, this is a risk based decision in that the risk of the failures and extant defects in the system causing a loss of functionality or adversely effecting the user experience is low. Still pondering I think that the floor in the presentation of the results (I have) is that all too often we consider the results of the tests somewhat independent of the outstanding defects.

My thoughts today brought me too this conclusion: Each of the test step failures should be given a severity which aligns to the defect which should be raised as a result of the failure. Then at the conclusion of the test (ie when all the test steps are executed) the calculation of Pass-Fail should be an aggregate of the step failures (or not). For each organisation this matrix and algorithm would need to be configured & tweaked but I think it has merit :-)

By reporting on the test status in terms of failure severity I think will bring more meaning to test results (another dimension!). We could further enhance the reporting by assigning each of the test scripts a priority and then reporting on the number of high priority tests with severe failures. Oh the possibilities!

QC 11 - the saga begins!

I learnt during recently that a new version of HP QualityCentre is available, Version 11 - check www.hp.com. Needless to say I downloaded the new version and set about preparing a new virtual server to install it on. Sounds simple enough, just fire one up and away we go.

Oh no no no, First I couldn't remember the administrator password for the virtual server I set up some months ago for some other new toy adventure! This resulted in deletion of the current machine and birth of a new one to take it's place. This was simple enough except I forgot the lesson that I should have learnt last time - note to self: A harddisk with 10 gig of storage is only enough to install the server OS (Win2008), nowhere near enough for HP ALM to be installed... Or SQL Server which I also found out part way through needed to be installed - doh. By this time I am becoming a big fan of virtualised servers and being able to just delete one, and create a new improved one in a matter of minutes.

Now onto server build three (3) and now we are looking like making some progress. Yahoo, we are past the disk space issue, only to smack into another brick wall - the database doesn't want to know about this ALM product: CONNECTION REFUSED! A quick google and I found out that by default the SQL server install doesn't enable 'Named Pipes' or 'TCP/IP', one of which is required for the ALM installer to connect to the database! Solved that problem with a few clicks of the mouse. I thought we must be close now, but no. The final error was that the service failed to start - I believe (not confirmed) that it was JBoss that couldn't find the 'java virtual machine' which stopped the whole process in it's tracks. Some more googling and using some of the faint memories from when I studied Java programming I managed to set the JAVA_HOME and Path variables to my freshly installed jre6 - restarted the services and hey presto Quality Centre 11 roared into life.

QC 11 has a tricked up user interface, which has a Java Swing feel to it, but I had to install a c++ security thingy when I installed the QC client on the workstation which has placed a small doubt in my mind - maybe it's a .Net web interface????

Here's a picture - what do you think?



So now I'm up and running, feeling somewhat proud of myself for managing to get the installation completed given my technical shortcomings. In the same way that my friends who are infrastructure engineers think that testing is repetitive and mind numbing, well after 3 server rebuilds, 2 SQL installs, 3 ALM and countless service restarts I think it might be a case of those in glass houses shouldn't throw stones!