The future of testing as seen through the eyes of an Iqnite attendee

 caught up with several of the delegates before dinner and I asked them "Is the cloud going to change the way we test?" the first response I received was "It already has" obviously I asked how? The response was simple, he said "It allows me to operate my business from anywhere, anytime; I am now able to run large scale load tests for my customers cheaply, using load generators around the world". 

I was lucky enough to attend the VIP breakfast where Dennis Bass (Keynote speaker and Microsoft – Test Architect, Visual Studio TFS) and Michael Palotas (Ebays head of quality engineering) shared their thoughts on were testers, and testing in the clouds is headed. Both of these fine fellows where in violent agreement that the key area of influence on testing going into the future will be the speed. Dennis called it "Cloud Cadence", the speed at which your customers and clients will demand innovation and expansion will increase significantly. 

Michael highlighted that a ebay they are moving into a development cycle that has no real beginning and no end, it's constant. Developers will be able to develop a feature in the morning and have it live in production by the afternoon. This sort of pace and rapid deployment has to involve a change in testing methods and approaches, as quality is paramount as is timeliness. Michael's view is that the line between testers and developers will blur, and to some extend it already had at ebay. He said his best testers are also the best developers, they just choose to be testers. Testers will have to be capable developers, just with a testing bent! Michael spoke of "Quality Engineering" rather than quality assurance or quality control.

Dennis shared a similar view of the future, and in fact its reality at Microsoft already were the developers are testers and the testers are developers, all working together to deliver a quality product to the clients. Dennis highlighted that testers will into the future be required to triage and investigate bugs beyond simply proving it's repeatable. Both Dennis and Michael advocated using methodologies as and when appropriate to the organisation. For example when Dennis is conducting a load test on TFS he has the developers sitting with him to triage bugs before they occur – for Dennis the worse thing that can happen when he encounters a problem is for a bug to be raised, the value of a bug is only realised once it's fixed – collaboration is king. Dennis's final words were something like "Testing cannot be a bottleneck or barrier, otherwise they will go around or without us". The reason for the comment was that if the business wants a product to ship in say 48 hours, the last thing they want to here is it's going to take 2 weeks to test it – this will mean we need to be smarter about the testing, more risk based testing and acceptance of risk. Testing needs to keep pace with development techniques, keep up or be left behind…

Dennis shared his insights and experiences gained from moving Team Foundation Server into the cloud as Software As A Services. I was amazed at some of the lessons, as they seemed to be the inverse of why we would want to move to the cloud. For example, Dennis said that everything takes longer in the cloud, and it costs more! Why you ask, well because it's in the cloud and distributed who knows where moving large amounts of data around is not as efficient as shuffling it around your local networks. The cost was an interesting point – it costs more to test in the cloud  because its an actual cost – as in the cloud provider will be sending you an invoice each month which has to be paid. It's not like having your servers locally and maintained by the server team who costs are hidden away in an operational budget. 

Some 'got-yas' which Dennis imparted included: beware, the cloud is immature,  evolving and largely uncontrolled. As such when your testing in the cloud things are changing servers are being rebooted, patched deployed and your unlikely to be aware of any of it! He stressed "you are not alone" the cloud is the cloud and by definition is it a service used by many and this may impact how your bit of the cloud is operating. Applications will need to be better able to be debugged without access to the servers – as in the cloud you are unlikely to have access to the underlying servers. Operational testing is a must, the applications must be tested for scalability, shrink and growth – applications will also need to be robust enough to operate whilst the underlying systems are patched.

What more would we need to do?

KJRoss & Associates is running the ASTA awards to recognize the outstanding work done by testers within Australia. 

Now testers like most IT professionals are not often self-promoting and will need to be nominated by peers, colleagues or clients. I heard of a situation were a manager was asked, “Do your testers deserve an award?” via email, and the response was “No not really”. This got me to thinking; well what would a testing team have to do to change this response? Find more bugs? Test quicker? And I must admit I’m at a bit of a loss... But I do take heart in that, rather than just ignore the email and its contents the manager actually responded. Is the response, though not worded so, a complement to the testing team indicating that the team is front of mind and is doing a good or sufficient or adequate job just not an outstanding job?

Testers and time poor tourists how much do they have in comment?

The modern tourist rushes from landmark to landmark taking photos to put on Facebook so all their friends can ‘like’ and comment on the magical time they are having; Testers rush around executing test case after test case bouncing from requirement to requirement stopping only long enough to prove it’s been ‘delivered’. Sure defects get in the way, just like the weather or traffic effects the bus tour; but as with a time poor tourist who isn’t able to change their itinerary they must press on to the next stop, after all we are on a schedule! Oh, and if you thought there would be time to double back and visit the thing you missed because of poor weather, well guess again. Today we are going to the next “must see” over there and tomorrow we’re in the next town and the day after that well you’ll be on the plane home and back to work. But never fear, you have the photos!

So lets reflect a moment yeah the tourists have ‘been there’, they have the photos and stories after all but have they really experienced the place? They have eaten the local cuisine, at the hotel but how real is it? From personal experience travelling around Thailand we ate at ‘authentic restaurants’ but there was very little in common with the food being prepared on the side of the road on mini gas burners attached to push bikes. It’s unfortunate that the tours don’t often allow time for the tourist to get of the beaten track to experience the real essence and character of a place. Testing is much the same rush rush rush, test the requirements, find the defects and push the release into production with little if any time for regression testing. Is this effective testing? Are we getting off the beaten track like our users will?

If testers are the time poor tourists then are our users the grey nomads? Just cruising from place to place as they feel like it. They have time to get off the beaten track, or stay another day if the weather doesn’t allow them to go fishing or sailing or play that fabulous golf course. Or are our users akin to the back packers who just lob in to the cool places, party a lot and work if required then just move on when well the mood takes them?

Testing is the cause of poor quality software

I'm going out on a bit of a tangent for this one but I think that's its got some merit.

Recently I was part of a conversation about all of the issues we'd discovered during a testing cycle. The back story is earlier this year a new test environment was built and we tested it. We tested the environment in-line with our current practices which is to run a suite of tests that tests all of the major components and system flows. This test suite can be completed in roughly 4 hours. If we were to run our complete system test it takes about 3 weeks and so taking a risk based approach we opted for the quick test cycle. This decision was predicated on the fact that this environment would be build using tested and approved build instructions which have previously delivered a stable and complete environment.

Fast forward to the conversation mentioned earlier. It went something like they said "Didn't you test it properly?" I said "Yes, we followed our standard process and ran the quick cycle tests" they said "well that's why all the defects, we need you to test it to make sure it done right." I said, well nothing... Pondering the conversation has lead me to believe the testing is the cause of poor quality software. Why, well because the folks who build the software rely on testers for quality control rather than focus on quality assurance themselves; With timelines never being long enough to build a proper solution of course there isn't enough time to test it before it goes into test. But that is ok, that's why we have testers, they test and they'll caught the bugs before they go into production.

So tonight, my thoughts are along the lines of if we get rid of the testers, then the folks who build the software will have to build and test; therefore the quality of software will increase and shift away from the compliancy of the testers will catch it! After all those testers take so long to test anything...

Are we asking the right question?

I'm part way through reading Freakonomics by Steven D Levitt & Stephen J Dubner (Gr8 read just quitely!). In the opening "economics is a science with excellent tools for gaining answers but a serious shortage of interesting questions..." and this starting me wondering are we asking the right questions in software testing? I mean every day I get asked how's our defect count, how close are we to finishing testing, how many tests have you run, how many are left, were is the test report. All of these are valid but really uninteresting questions (I think!) rather blah and binary, and really do the answers to these questions really tell us much? There's a line in one of the Disney movies (Shrek I think) in which one of the characters supporting characters said "blah, blah blah" and then the main character said "yes he's talking, but is he saying anything?" Which is kind of the way I feel about answering the daily questions. I give the answers, but without understanding the meaning behind the numbers or tends am I really saying anything? Shouldn't we be being asked about business processes or risks or impact on revenue? These are much more interesting but I think based on the freakonomics outlook something like ... help help me to find an interesting question!  

more to come...

Are we testing the right things?

Recently on a plane back from Adelaide I read an article in the QANTAS in-flight magazine the Australian Way (as you do!) the article was written through the eyes of a traveller who was visiting somewhere in either Canada or North America. The bits of the article I found most thought provoking was the quotes of the tour guide who said something like 'focus on what you can see, stop and admire all that is around you; don't lament the that you didn't see a bear, be thankful you have seen over 100 different types of plants and animals.' The guide went on to say 'all too often tourists visit these parts and rush from place to place to see this and that, taking only photos of things, not creating a memory by experiencing being there'. This made perfect sense too me the phase "Stop and smell the roses" came to mind. By chance tonight I saw the tagline of another article Look Beyond the Lens on the Sydney Morning Herald's website and after only reading the tagline I pondered.... As testers, in testing do we sometimes focus too much on proving that a requirements have been implemented or that defects have been found as a result of our efforts - to prove we were there? Do we focus on finding the bear in the woods because it's big, scary and easy to see or not see, all the while missing the wondrous smaller wildlife, flowers and trees that are necessary parts of the ecosystem for the bear to survive or equally perish?

My initial thoughts were along the lines of do we focus too much on finding bugs and satisfying requirements (taking pictures - see it is there, I saw it) only to forsake to the things that our users actually need from the system to do there job or improve their efficiencies? And if this is the case, then how do we change our plans to allow us to stop for a while and just "be there"...  Somehow I think the need to take pictures and wiz off to the next location is a result of time constrains and a desire to see as much as possible in a given period of time. Equally in testing, time or lack of time is a huge factor. But it is time that maybe the ultimate measure for a picture only lasts for as long as the media is around and often fades over time (remember when they used to print pictures!) while a memory can live forever being pasted from one generation to another! It's time to choose which we value more, pictures or memories...  
  

The Missing Link, found sort of!

Dude, where's my link? would also have been an appropriate title for this blog entry! I came across this little gem over the weekend - it's a beauty a double fail, firstly the link colour has been set to 'white' when not active and yellow when active. Neither of these two colours work at all well and makes it easy to see why blue link text is the default. Given the location of this particular link (within a tender site) I suspect that its a section of a document that has been copy and pasted in and relies on a style sheet thats not present in the destination website - but that's no excuse for not proof reading!!!

Go where?

Here?