Calculating the Return on Investment (ROI) of Test Automation

Many of our existing customers and some potential new customers come to Fusion Alliance wanting to know more about test automation and if it’s right for their business. The answer to that question, the following assessments must be done to determine the argument for or against using test automation.

1. Assess the maturity level of the customer’s software testing department and their processes.
2. Assess the nature of the engagement Fusion Alliance is being asked to assist the customer with.

These assessments are essential as many organizations make the mistake of thinking that test automation will immediately reduce the costs of testing, increase test coverage, and shorten test cycles. These are sometimes the longer-term benefits of test automation, but are not immediately evident. To determine if test automation will indeed reduce costs for the organization in the long term, the ROI must be calculated. The ROI for test automation is simply the benefit of test automation divided by the cost of test automation.

The cost of test automation is higher than manual testing and should include any costs related to hardware, software and licenses, time for resources to produce scripts, and cost of the resources themselves.

The benefits of test automation must be calculated over a period of time for the software under test and take into account the reduced time to execute tests and the ability to test as frequently as the organization wishes. These figures should be compared to the costs of manually testing the same software.

Think of the cost of test automation vs. the cost of manual testing as follows:

Cost of test automation = price of hardware needed + price of tool required + time to develop test scripts + (time to maintain test scripts x number of times test scripts are executed) + (time to execute test scripts x number of times)

Cost of manual testing = time to develop test scripts/charters + (time to maintain test scripts/charters x number of times tests are executed) + (time to execute test scripts/charters x number of times)

After figuring the costs of test automation and manual testing, calculate ROI as follows:

ROI = (cost of manual testing – cost of test automation) / cost of test automation

By now, you might be saying to yourself “that seems too easy”. If you are saying this, you would be correct. Other factors must be taken into account such as the benefits of manual testing over test automation. The most complex parts of the software should be left to a manual tester to analyze to ensure maximum quality. Therefore, a mix of test automation and manual testing might be the best solution. Context must also be taken into consideration. What is the complexity of the software? What is the size of the software and what is its role in the business? Smaller, less complex applications that won’t take a considerable amount of time to test won’t require test automation because the upfront costs will outweigh the benefits gained. Also keep in mind the types of testing you are looking to automate. Each of these different types will have a different ROI that needs to be taken into consideration.

Thinking about all of these things upfront will not only help you determine whether to automate your testing or not, but also help you plan your overall testing effort and give you a jump start on your test plan(s).


Why Software Testers Can’t Avoid Being Technical

The time of manual only testers is nearing its end. If you are a software tester you are either standing at the crossroads or already down the path of success. If you’re confused as to what I mean by being “at the crossroads”, I am talking about whether to become technical or not.  New technologies are making software more and more complex which means that there is more and more to test.  Software testers must start thinking about how test automation can aid them in covering all the necessary test scenarios to ensure that a quality product is delivered.  Take for example the mobile application revolution.  The GUI might be very simple to adapt to the limitations of the device, but it doesn’t make the code behind the application any less complex. Companies need their product(s) to perform. Some only get one chance to get it right in today’s economy.  It only takes one bad experience by the users to cost the company millions of dollars in maintenance (or redesign) costs and a new marketing strategy (if the company doesn’t collapse first).

Well you might say “Well I want to become a test manager or a project manager.  I don’t need to worry about that stuff.” Wrong! At the least a test manager/project manager should be technically conscious.  How much better can you relate with the testers you manage on a project that requires some degree of test automation if you understand the time that’s required to build test scripts and interpret results? How do design changes affect your automated test scripts?  Even before the scripting begins, the test manager/project manager will be involved in test planning meetings and must be able to explain the benefits around automating or not automating tests.  If this isn’t enough, how much credit is gained by a test manager that can walk into a room full of executives and explain the ROI of test automation?

Now you might be asking yourself “Are you saying that manual testing is dead?” Of course not! I believe that manual testing will always be needed and must be reserved for the most complex pieces of the system that require a human mind to analyze.  Leave test automation to the most tedious, mind-numbing activities (e.g. entering login credentials).

So choose your path…one leads to a technically conscious test manager and technical tester and the other path leads to something other than software testing. Neither is bad or wrong but which path will you take?