Test automation is a means of testing software, by using software other than the software being tested, to automate repetitive tasks in a formalized process.

Test automation can be employed for many reasons, including: exposing new bugs, detecting regressions (old bugs, performance, etc.), ensuring a level of quality, as an aid to refactoring, and so forth. While these can be achieved through other methods including manual testing or monitoring, there are many benefits to software test automation.

1. When to Write Test Automation

There are various opinions on the right timing within the software development life cycle to write the test automation. Test-driven development engineers (TDDers) are passionate about writing tests before starting to code. Others write tests during or toward the tail end of development. And then there are use cases where tests need to be written well after a legacy system was created. Like many things in life, a good solution is usually not found near the extremes. TDD was a big thing a decade or so ago but the zealotry has thankfully faded. There's even a sentiment of late that TDD is not a terribly good idea.

Retrofitting tests onto a legacy system can be miserable work but it can still be quite valuable if that system needs to be changed.

Generally, where possible, it's preferable to write tests before the changes ship. The benefits of writing test automation at this point are:

• Helping keep the bugs at bay
• Ensuring the code is more testable
• Not adding more untested code, allowing future changes to be easier

2. Code Coverage

Code coverage, the percentage of code covered by automated tests can either be low, medium or high. It’s helpful to note where coverage of the software stands. If it’s low or medium, it’s probably worth some effort to bump it up. However, this metric follows a sort of diminishing returns type pattern. Once the coverage nears a high level the value of increasing it is generally not worthwhile. Furthermore, even a coverage of 100% does not guarantee a lack of bugs. All it means is that the tests have managed to traverse all code paths - likely at great cost.

3. How or What to Test With Test Automation

How or what to test is one of the more interesting aspects test automation. There are generally three broad levels of testing, which identifies the scope of the thing(s) being tested:

• Unit - unit tests aim to test something in isolation.
• Integration - Integration tests are about testing multiple pieces in concert.
• System - System tests, or end-to-end tests are used to make sure an entire system functions as expected.

The proportions of each testing level to use in any given situation are context-dependant. And each level of testing has its pros and cons which is a subject all on its own. Some people advocate for more unit, less integration, and even less system tests - the classic test pyramid. Others believe feel that integration tests hit a sweet spot so focus there. And yet others feel that end-to-end testing isn't worthwhile. Regardless, it’s a good idea to consider how much testing at each level is worthwhile and to be deliberate in testing.

Software test automation is a fairly large topic. This article only touched on a few high level aspects. There's a wealth of information out there for those that are interested in delving deeper into the area. Happy testing!