Closing one chapter and starting another…

Yesterday was my last day as a consultant at Fusion Alliance. I am starting my new job as a Senior Business Analyst/Test Coordinator at a small company in Richmond, IN on Monday.  I’m looking forward to starting this new position and learning about a market I never had the opportunity to serve in during my time at Fusion.

The things I’ve learned over the past 5 1/2 years will be priceless as I tackle the task of implementing/tweaking processes to ensure quality throughout the Software Development Lifecycle and helping to document requirements and design for applications/functionality being implemented to drive and reinforce their purpose as well as reduce headaches when fixing production issues. Stay tuned for more posts that will come from my experiences in this new role!

A couple of good posts…

A good post on uTest that gives 8 Tips for Becoming a Dedicated Tester.

Another good post on uTest that lists 5 Myths about Software Testing.

How to Build an Effective Training Program for Software Testers

Recently, members of the software testing discipline at Fusion Alliance put together some software testing training material for new hires and existing employees in the discipline. This training is rather flexible and allows each individiual to go through the training at their own pace or while they are on the bench in between projects. This keeps the skills of our test consultants sharp and fresh. However, I personally think this training is most valuable for the new hires who might not be familiar with software testing techniques or the testing concepts that we believe have been proven to be successful. Most of all, this program allows us to grow our Sofware Testing discipline without waiting for the “perfect” candidate to walk through the door or to be found by our recruiters.

What should this training program look like?

1. Content should be easy to understand.

This allows those new to the testing discipline to easily understand the basic concepts. However, the content should build upon itself and become more advanced and complex as the trainee advances through the program. 

2. Make the program accessible (preferrably via the Internet or company’s intranet).

This allows the trainee to access the program from anywhere and work at their own pace.

3. Make sure the content is diverse. 
 

The content should not only include technical material (e.g., testing techniques) but also soft skill material (e.g., consulting skills, including interviewing skills, communication skills, and etiquette).

4. Ensure that the content is relevant.

Including “tried-and-true” techniques that are still useful is OK, but make sure you include some material that applies to cutting-edge technologies as well.

5. Make sure the program forces the trainee to apply the knowledge they are gaining in some way (e.g., quizzes and exercises).

Reading about a particular topic is never enough. Give the trainee an opportunity to apply the concepts they are learning. You must walk before you can run and falling is envitable. Just remember: Failure in training is a lot less damaging than failure at the client site.

Context-Driven Testing: iPhone 4S

According to an article by CNET News, users of the iPhone 4S have been rating the battery life of the phone as “subpar”.  Not a good thing, since the phone was just released less than a month ago. Apparently the culprit is a location-based feature that determines when the user has moved to a different time zone and sets the phone’s clock accordingly.

This seems to be a feature that was overlooked during testing of the iPhone 4S. Perhaps it was a feature that was rated as “low risk” and, therefore, due to time constraints, left out of the executed test scenarios all together.  Maybe the feature wasn’t tested at the right point in the development lifecycle.  Whatever the case, this leads us back to questions that have been around for ages.  What do we test?  What don’t we test?  These broad questions lead to more detailed questions. What features have the most visibility to the user and the highest risk of containing defects? What features won’t we test?  Will any features go untested? The answers to these questions are different for each project, client, and product. I believe context must always be taken into consideration.

As a tester, I always ask myself three basic questions on every project I’m on.

  1. What is the estimated timeline for testing on this project?
  2. What features/functionality is part of the critical path of this product?
  3. What tests can I perform to get the most coverage within the given timeline with the tools provided?

Once again, the answers to these questions is always different, but the questions are always good ones to ask.

Three Ways to Test Mobile Applications

There are three ways to test mobile applications.  Not all three ways may, or should, be required when testing an application.  However, choosing not to use one could carry potential risks.  If the application and context are not properly assessed as to what types of testing should take place, critical defects could go undiscovered during the development and testing phases. Keep in mind that all three ways can be performed using manual or automated testing techniques.

The first way to test a mobile application is by using an emulator.  An emulator is a computer program or device that imitates the behavior of another program or device.  For mobile applications, most emulators are computer programs that imitate another program under development or test.

Pros: The emulator accurately imitates the behavior of the program under test including all functions and disruptions that occur on the selected device (e.g. incoming call, switching applications, behavior when buttons available on the device are pressed, etc.). The emulator can also imitate the device under numerous different types of configuration and user settings.

Cons: The emulator cannot imitate the network that the device will be on (including different physical locations).  Depending of context, this way of testing a mobile application may not be appropriate, especially if the application uses GPS in any way. The emulator also doesn’t provide the user with any hardware settings. Emulators vary in how accurately they imitate the device depending on the operation system or the device.

The second way to test a mobile application is by using a real device tethered to a network.  During this type of testing the device is controlled by the user at the workstation and typically involves third party testing tools to invoke different tests on-demand.

Pros: On-demand execution of test scripts.  Actual device responses can be observed instead of emulator responses. The device accurately imitates the behavior of the program under test including all functions and disruptions that occur on the selected device. The device can be configured to accommodate for specific testing scenarios and user settings.

Cons: Configuration of the device has to be done manually and can become costly and tedious.

The third way to test a mobile application is by testing it on a real device on a real network in real locations.  During this type of testing the tester gets to experience the application as the user would and can test anywhere the network allows.

Pros: Depending on the application, GPS-related functionality can be tested since the tester can change locations. The tester gets to experience real device responses instead of emulator responses.

Cons: Each network varies in performance and network outages becomes a risk.  The tester has to travel to different locations if the application uses GPS, which could be a pro or a con depending on where you live or where you travel.  This type of testing is very tedious, costly, and requires manually changing every hardware setting and every user setting.  Using a third party testing tool also becomes tedious if not impossible. Therefore, capturing test results could become very challenging.  This type of testing is and should be combined with one of the other types of testing mentioned above.

As mentioned earlier, a combination of these would probably most appropriate for most mobile applications.  Context should always be considered on any project, mobile or not.  Successful projects are a result of good planning, so take the time you need up front to determine what types of testing that are best suited for your application to cut project costs.

HTML5: The Future of Mobile Applications

HTML5The mobile applications of the future will be web-based.  That is the report from MIT’s Technology Review and I agree.  Pretty soon, mobile device users will be able to download applications via the web to their mobile device and access them offline and eliminate the dependency on external plug-ins such as Flash. For an example of this, download the free version of Angry Birds when you download Google Chrome and then disconnect from your local network. Voila!How is this possible you might ask? HTML5. This language is a revision of the original HTML language and is used to structure and present content over the web.  It presents better error handling and is device independent. It’s a developers dream because it is truely a write-once, run anywhere code that eliminates the need for native code. This means the only applications that need to be native will be those that work with device APIs that aren’t part of existing web standards (in other words, rare cases).
 
HTML5 is also becoming popular in the mobile space because it makes advanced web application features available in all mobile web broswers while standardizing syntax and behavior (the development process is more visible to the public). The biggest advantage to HTML5 is the cache and database make it possible for mobile developers to store things locally on the device and avoid interruptions in connectivity so they can get their work done as well as making development easier by using more markup instead of scripting for things like drawings, video, audio, offline storage, content specific elements, and form controls (calendar, date, time, e-mail, etc.). For example, videos and audio use simple tags (such as <video></video> and <audio></audio>), a handful of support formats (e.g. .ogg, .wav, .mp4), and their own attributes (e.g. height width, src (source), controls).
 
HTML5 is quickly changing the landscape of mobile web applications and is just as quickly gaining support from popular web browsers such as Internet Explorer, FireFox, Safari, Chrome, and Opera. This presents new and exciting challenges on the fronts of mobile web development and mobile application testing.  However, I believe we can leverage our past experiences and use similiar technologies as oracles to help us along the way.
 
For more information on HTML5, you can visit W3Schools.com or Dive Into HTML5.org.

Reasons to Become a Mobile Application Tester

I saw a post recently on www.mobileapptesting.com that listed the Top 10 Reasons to Become a Mobile Application Tester.  I agree with all 10 reasons, however, I’m going to give a couple more reasons. 

1. Mobile application testers are in high demand!

Here’s another post on www.mobileapptesting.com that talks about the growing demand of mobile application developers and testers. To see how true this is, I did a search for “mobile developer” on www.dice.com and got back 1,890 results. Of course, this doesn’t include other searches like “mobile engineer” (2,228 results) and “mobile programmer” (215 results).

Next, I performed a search for “mobile tester” and got back 104 results and “mobile QA engineer” (322 results).  These numbers are small in comparison, however, this shows that the demand is there.  I believe this number will grow even more after companies realize they need a software tester with expertise in mobile testing because their developers cannot double task or do not deliver the quality mobile applications they desire.

2. There are more defects to be found.

The lack of mobile developers will force companies to transition existing developers (or hire new ones) to mobile development.  Some might put these developers through training, but it’s highly unlikely that they will have the money to do that. What’s this mean? That’s right…more defects.

Strategically Placing Testing Resources for Maximum Test Effectiveness

No, I am not talking about test resource management (i.e. placing testing consultants at certain clients to maximize profits) or anything like that.  I’m strictly speaking about test leads/test managers assigning their testing resources to specific areas of an application in order to provide maximum test effectiveness during test planning, test creation, test execution, and result analysis.

At my current client we discovered gaps in our testing by assigning two testing resources to different areas of the system that overlap. For example, my area of the system dealt with defining data in the system that will determine what happens (e.g. when a flag is triggered, what prompts appear to the user, etc.) in another part of the system. The other part of the system was assigned to a different tester. Therefore, the other tester used my test scripts to write his and whala! Gaps discovered!

Calculating the Return on Investment (ROI) of Test Automation

Many of our existing customers and some potential new customers come to Fusion Alliance wanting to know more about test automation and if it’s right for their business. The answer to that question, the following assessments must be done to determine the argument for or against using test automation.

1. Assess the maturity level of the customer’s software testing department and their processes.
2. Assess the nature of the engagement Fusion Alliance is being asked to assist the customer with.

These assessments are essential as many organizations make the mistake of thinking that test automation will immediately reduce the costs of testing, increase test coverage, and shorten test cycles. These are sometimes the longer-term benefits of test automation, but are not immediately evident. To determine if test automation will indeed reduce costs for the organization in the long term, the ROI must be calculated. The ROI for test automation is simply the benefit of test automation divided by the cost of test automation.

The cost of test automation is higher than manual testing and should include any costs related to hardware, software and licenses, time for resources to produce scripts, and cost of the resources themselves.

The benefits of test automation must be calculated over a period of time for the software under test and take into account the reduced time to execute tests and the ability to test as frequently as the organization wishes. These figures should be compared to the costs of manually testing the same software.

Think of the cost of test automation vs. the cost of manual testing as follows:

Cost of test automation = price of hardware needed + price of tool required + time to develop test scripts + (time to maintain test scripts x number of times test scripts are executed) + (time to execute test scripts x number of times)

Cost of manual testing = time to develop test scripts/charters + (time to maintain test scripts/charters x number of times tests are executed) + (time to execute test scripts/charters x number of times)

After figuring the costs of test automation and manual testing, calculate ROI as follows:

ROI = (cost of manual testing – cost of test automation) / cost of test automation

By now, you might be saying to yourself “that seems too easy”. If you are saying this, you would be correct. Other factors must be taken into account such as the benefits of manual testing over test automation. The most complex parts of the software should be left to a manual tester to analyze to ensure maximum quality. Therefore, a mix of test automation and manual testing might be the best solution. Context must also be taken into consideration. What is the complexity of the software? What is the size of the software and what is its role in the business? Smaller, less complex applications that won’t take a considerable amount of time to test won’t require test automation because the upfront costs will outweigh the benefits gained. Also keep in mind the types of testing you are looking to automate. Each of these different types will have a different ROI that needs to be taken into consideration.

Thinking about all of these things upfront will not only help you determine whether to automate your testing or not, but also help you plan your overall testing effort and give you a jump start on your test plan(s).

STPCon 2011 – Nashville, TN

The Software Test Professionals Conference is coming to Nashville, Tennessee and the topic is Test Automation. Don’t miss out!

http://www.stpcon.com/