Strategically Placing Testing Resources for Maximum Test Effectiveness

No, I am not talking about test resource management (i.e. placing testing consultants at certain clients to maximize profits) or anything like that.  I’m strictly speaking about test leads/test managers assigning their testing resources to specific areas of an application in order to provide maximum test effectiveness during test planning, test creation, test execution, and result analysis.

At my current client we discovered gaps in our testing by assigning two testing resources to different areas of the system that overlap. For example, my area of the system dealt with defining data in the system that will determine what happens (e.g. when a flag is triggered, what prompts appear to the user, etc.) in another part of the system. The other part of the system was assigned to a different tester. Therefore, the other tester used my test scripts to write his and whala! Gaps discovered!


Calculating the Return on Investment (ROI) of Test Automation

Many of our existing customers and some potential new customers come to Fusion Alliance wanting to know more about test automation and if it’s right for their business. The answer to that question, the following assessments must be done to determine the argument for or against using test automation.

1. Assess the maturity level of the customer’s software testing department and their processes.
2. Assess the nature of the engagement Fusion Alliance is being asked to assist the customer with.

These assessments are essential as many organizations make the mistake of thinking that test automation will immediately reduce the costs of testing, increase test coverage, and shorten test cycles. These are sometimes the longer-term benefits of test automation, but are not immediately evident. To determine if test automation will indeed reduce costs for the organization in the long term, the ROI must be calculated. The ROI for test automation is simply the benefit of test automation divided by the cost of test automation.

The cost of test automation is higher than manual testing and should include any costs related to hardware, software and licenses, time for resources to produce scripts, and cost of the resources themselves.

The benefits of test automation must be calculated over a period of time for the software under test and take into account the reduced time to execute tests and the ability to test as frequently as the organization wishes. These figures should be compared to the costs of manually testing the same software.

Think of the cost of test automation vs. the cost of manual testing as follows:

Cost of test automation = price of hardware needed + price of tool required + time to develop test scripts + (time to maintain test scripts x number of times test scripts are executed) + (time to execute test scripts x number of times)

Cost of manual testing = time to develop test scripts/charters + (time to maintain test scripts/charters x number of times tests are executed) + (time to execute test scripts/charters x number of times)

After figuring the costs of test automation and manual testing, calculate ROI as follows:

ROI = (cost of manual testing – cost of test automation) / cost of test automation

By now, you might be saying to yourself “that seems too easy”. If you are saying this, you would be correct. Other factors must be taken into account such as the benefits of manual testing over test automation. The most complex parts of the software should be left to a manual tester to analyze to ensure maximum quality. Therefore, a mix of test automation and manual testing might be the best solution. Context must also be taken into consideration. What is the complexity of the software? What is the size of the software and what is its role in the business? Smaller, less complex applications that won’t take a considerable amount of time to test won’t require test automation because the upfront costs will outweigh the benefits gained. Also keep in mind the types of testing you are looking to automate. Each of these different types will have a different ROI that needs to be taken into consideration.

Thinking about all of these things upfront will not only help you determine whether to automate your testing or not, but also help you plan your overall testing effort and give you a jump start on your test plan(s).

STPCon 2011 – Nashville, TN

The Software Test Professionals Conference is coming to Nashville, Tennessee and the topic is Test Automation. Don’t miss out!

Why Software Testers Can’t Avoid Being Technical

The time of manual only testers is nearing its end. If you are a software tester you are either standing at the crossroads or already down the path of success. If you’re confused as to what I mean by being “at the crossroads”, I am talking about whether to become technical or not.  New technologies are making software more and more complex which means that there is more and more to test.  Software testers must start thinking about how test automation can aid them in covering all the necessary test scenarios to ensure that a quality product is delivered.  Take for example the mobile application revolution.  The GUI might be very simple to adapt to the limitations of the device, but it doesn’t make the code behind the application any less complex. Companies need their product(s) to perform. Some only get one chance to get it right in today’s economy.  It only takes one bad experience by the users to cost the company millions of dollars in maintenance (or redesign) costs and a new marketing strategy (if the company doesn’t collapse first).

Well you might say “Well I want to become a test manager or a project manager.  I don’t need to worry about that stuff.” Wrong! At the least a test manager/project manager should be technically conscious.  How much better can you relate with the testers you manage on a project that requires some degree of test automation if you understand the time that’s required to build test scripts and interpret results? How do design changes affect your automated test scripts?  Even before the scripting begins, the test manager/project manager will be involved in test planning meetings and must be able to explain the benefits around automating or not automating tests.  If this isn’t enough, how much credit is gained by a test manager that can walk into a room full of executives and explain the ROI of test automation?

Now you might be asking yourself “Are you saying that manual testing is dead?” Of course not! I believe that manual testing will always be needed and must be reserved for the most complex pieces of the system that require a human mind to analyze.  Leave test automation to the most tedious, mind-numbing activities (e.g. entering login credentials).

So choose your path…one leads to a technically conscious test manager and technical tester and the other path leads to something other than software testing. Neither is bad or wrong but which path will you take?

The Time Is Now

I will be the first to admit that I have procrastinated on bettering myself as a software tester over the past year. However, I am a consultant and as a consultant you sometimes have to accept the fact that you can’t possibly juggle work (40+ hours a week), the honey-do list, spending time with family, and taking courses/reading books to advance your career.

Having said this, I recently asked myself “When are you going to take the next step? What better time than now?” So I say the time is now. There will always be the constant juggle of life, work, and someday (for me anyway) children and all of the activities they will be involved in. If you are asking yourself these same questions and wonder when you will find the time I would tell you “The time is now.” Next, you’ll probably be asking the question “So, what’s next?” I have come up with a few steps to guide you in the right direction.

The first step is to set a goal. Is your goal to get a certification? Is your goal to learn a new programming language? Is your goal to become a manager? However big or small, set the goal. If the goal is bigger and will take a longer period of time, like becoming a manager, set smaller goals within the goal. For example, set goals to read some books on project management, find courses you can take, list people that could be your mentors.

The second step is to take action. You’ve thought all this through now act on it. If you want to learn a new programming language go buy a book, ask for training, Google for online tutorials, ask a developer that knows the language for some resources or simple examples to help you. If you want to become a manager start reading the books you picked out, ask your manager to assign tasks to you that they perform on a daily basis (especially ones that put you into a leadership role in some shape or form). I would strongly advise you to talk to the mentors you listed on a daily basis if possible. This will encourage you along the way because at times you might feel overwhelmed.

Lastly, share the knowledge you gain with others by having conversation with colleagues or giving a presentation at a conference. Encourage them to take action. This step is necessary in order to become a mentor to others so that they might someday become mentors and so on and so forth.

In Response to James Whittaker’s GTAC Post

I found James Whittaker’s latest post on Google’s Test Blog very interesting. To think that we could use a Chrome extension to automate tests that could identify bugs in the UI and take us directly to them is exciting to think about not only for testers but also for developers. Not only would this extension identify the bugs in the UI but it would also record and store all the tests that identified them. It doesn’t get much easier than that! However, like many other tools I’ve seen, this will still probably require the skills and trained eye of an experienced tester to determine what truely is a real bug and what is not. However, having a tool like this would certainly aid the tester in identifying places of concern to make those decisions.

To summarize…great advancements are happening in software testing and I believe the combination of a Chrome extension like this with well designed manual test scripts or exploratory charters would greatly enhance the quality of any web application. I will definitely be keeping my eye on this and see what demonstrations, presentations, or papers are released after GTAC.

Test Automation Resources for Beginners

If anyone has any test automation resources for beginners that they’d recommend or wouldn’t mind me borrowing (if you are local), please send me an e-mail (