Test automation is a methodology that has long existed in the software quality assurance and testing discipline to promote overall test efficiency and coverage. Test automation needs to be a very informed decision, bearing in mind what, when, how, and how much to automate. Best practices (such as modular, framework-based test automation) have been advocated around each of these topics to ensure that the test and product teams reap maximum benefits from the test automation effort. While most of these best practices apply regardless of whether the lifecycle is waterfall, agile, or something else, there are some core practices to keep in mind while taking on automation in the agile world. Here are seven of them:
1. Coverage-based Test Automation
Time is of the essence in agile. Code-coverage-based test automation runs really help in understanding the test traceability and areas where the coverage is still falling short. Test teams need to use instrumented builds and focus on code coverage runs as part of test automation execution.
2. Automation at the System Level
We all know that the UI undergoes a lot of change in agile, as the design may be very dynamic, user feedback may continue to flow in, or, simply put, UI may be one of the last areas to be solidified and frozen. Test automation maintenance at the UI level can be quite tedious. To maximize coverage and minimize maintenance, automation needs to be done largely at the system and services levels, especially in agile environments.
3. Test-driven Development
On some teams, agile may be extended to a form of extreme programming, where the testers come into stage first by creating automation tests that become the basis for source code development. In such cases, the test automation effort requires all the more collaboration with the product development team in order to pick the right scenarios and implement in a sustainable manner.
4. Test Automation to Precede Manual Testing
In the past, one of the practices advocated in test automation�especially when it was in sustenance areas such as regression testing�was to finish one round of manual testing to ensure stability before automating tests. This situation has definitely changed in the agile world where, given the time constraints, the team may have to focus on test automation right away. Even in such scenarios, although a formal test pass may not be possible, the team should take on at least one round of exploratory testing to verify product and module stability and eliminate any false issues.
5. Tool Choice
Automation tool choice is very important, especially in the agile workflow, since there is not much time to go over an incorrect choice or migrate from one tool to another. There are a lot of tool options, ranging from commercial to freeware to open source solutions. While alignment of the tool with the automation problem at hand is very important, there are a number of other factors that need to be considered in your final tool decision, including tool cost, maintenance, integration capabilities, and compatibility with existing solutions. (See the Sticky Notes for more.)
6. Testing Your Test Automation Code
Test automation needs to be developed like a regular product-development effort as part of a software development lifecycle. One of the important steps of this is to test your test automation code, which verifies the code and eliminates any test issues. Given the time constraints in agile, there is all the more pressure for the test automation code to be of superior quality, bringing down the overall maintenance effort and enhancing the automation suite's reliability.
One interesting point to note is that a test automation project is almost always undertaken in a waterfall or traditional development model, regardless of the overall product development lifecycle used. The reason for this is that the steps in test automation�requirements gathering, framework design and tool choice, test design and data creation, test bed setup, test code review, test execution, and maintenance�are almost always done in a sequential manner. Unlike regular product development, where you have a design team, a development team, a test team, etc., in test automation the same tester handles all the steps. She might be able to do a few things in parallel, but largely the main process flows through sequentially, one step at a time. Also, you do not need as much rigor (that involves Scrum meetings with multiple people involved) in a test automation effort. In fact, that would be overkill.
7. Code Sharing to Promote Cross-team Code Usage
There are multiple advantages in sharing test automation code with the product development team, especially the development, build, and operations teams. These advantages include increased focus on quality at varied levels, reduced development and test cycles, and knowledge sharing. Sharing code ensures that people other than just the test team are using it, which makes the test automation code more robust and reliable and helps to flush out any test issues early on.
These points touch upon the areas of what, when, and how to automate. An equally important thing to keep in mind is what not to automate. Modules that are more susceptible to change based on user feedback, such as UI (unless you have obtained user feedback on the UI and do not expect it to undergo a lot of changes), and test types such as usability and accessibility, where the human tester element is important, are best tested manually. Remember that I am only touching on areas specific to agile automation and not on areas that are unlikely to benefit from automation regardless of the product development lifecycle in use�e.g., modules not heavily used, administrative modules with a limited user base, etc.
Given the market forces at play, agile development cycles are here to stay for a long time. The test team has to carefully build an automation methodology that combines the best practices from traditional lifecycles with modern lifecycles and current needs in order to create a customized solution that helps enhance the quality of the product under test. The points discussed above will help you achieve this exact goal of delivering a robust product of exceptional quality within the short development cycles at hand.
