caught up with several of the delegates before dinner and I asked them "Is the cloud going to change the way we test?" the first response I received was "It already has" obviously I asked how? The response was simple, he said "It allows me to operate my business from anywhere, anytime; I am now able to run large scale load tests for my customers cheaply, using load generators around the world".
I was lucky enough to attend the VIP breakfast where Dennis Bass (Keynote speaker and Microsoft – Test Architect, Visual Studio TFS) and Michael Palotas (Ebays head of quality engineering) shared their thoughts on were testers, and testing in the clouds is headed. Both of these fine fellows where in violent agreement that the key area of influence on testing going into the future will be the speed. Dennis called it "Cloud Cadence", the speed at which your customers and clients will demand innovation and expansion will increase significantly.
Michael highlighted that a ebay they are moving into a development cycle that has no real beginning and no end, it's constant. Developers will be able to develop a feature in the morning and have it live in production by the afternoon. This sort of pace and rapid deployment has to involve a change in testing methods and approaches, as quality is paramount as is timeliness. Michael's view is that the line between testers and developers will blur, and to some extend it already had at ebay. He said his best testers are also the best developers, they just choose to be testers. Testers will have to be capable developers, just with a testing bent! Michael spoke of "Quality Engineering" rather than quality assurance or quality control.
Dennis shared a similar view of the future, and in fact its reality at Microsoft already were the developers are testers and the testers are developers, all working together to deliver a quality product to the clients. Dennis highlighted that testers will into the future be required to triage and investigate bugs beyond simply proving it's repeatable. Both Dennis and Michael advocated using methodologies as and when appropriate to the organisation. For example when Dennis is conducting a load test on TFS he has the developers sitting with him to triage bugs before they occur – for Dennis the worse thing that can happen when he encounters a problem is for a bug to be raised, the value of a bug is only realised once it's fixed – collaboration is king. Dennis's final words were something like "Testing cannot be a bottleneck or barrier, otherwise they will go around or without us". The reason for the comment was that if the business wants a product to ship in say 48 hours, the last thing they want to here is it's going to take 2 weeks to test it – this will mean we need to be smarter about the testing, more risk based testing and acceptance of risk. Testing needs to keep pace with development techniques, keep up or be left behind…
Dennis shared his insights and experiences gained from moving Team Foundation Server into the cloud as Software As A Services. I was amazed at some of the lessons, as they seemed to be the inverse of why we would want to move to the cloud. For example, Dennis said that everything takes longer in the cloud, and it costs more! Why you ask, well because it's in the cloud and distributed who knows where moving large amounts of data around is not as efficient as shuffling it around your local networks. The cost was an interesting point – it costs more to test in the cloud because its an actual cost – as in the cloud provider will be sending you an invoice each month which has to be paid. It's not like having your servers locally and maintained by the server team who costs are hidden away in an operational budget.
Some 'got-yas' which Dennis imparted included: beware, the cloud is immature, evolving and largely uncontrolled. As such when your testing in the cloud things are changing servers are being rebooted, patched deployed and your unlikely to be aware of any of it! He stressed "you are not alone" the cloud is the cloud and by definition is it a service used by many and this may impact how your bit of the cloud is operating. Applications will need to be better able to be debugged without access to the servers – as in the cloud you are unlikely to have access to the underlying servers. Operational testing is a must, the applications must be tested for scalability, shrink and growth – applications will also need to be robust enough to operate whilst the underlying systems are patched.
