If you don’t care about quality, you can meet any other requirement.

—Gerald M. Weinberg

Agile Testing

Agile Testing applies the principles of agile development to the practice of testing. Although traditional development has used a big-bang, deferred testing approach, agile testing develops and tests systems in small increments, often developing tests before writing the code, Story, or Feature. This way, tests help to elaborate and better define the intended system behavior, even before the system is implemented. Quality is built-in from the beginning. Like agile development, agile testing is a collaborative, team-oriented process. All team members share responsibility for testing the system.

This ‘test-first’ approach applies to all types of agile work. That includes capabilities, features, stories, NFRs, as well as code. It can even be applied to the hardware components of a system. The same way tests are written during coding, acceptance tests for capabilities, features, and stories are written during their elaboration. The just-in-time practice of elaborating the proposed system behavior also mitigates the need for overly detailed requirement specifications and sign-offs. Even better, unlike conventionally-written requirements, these tests are automated wherever possible. But even when they are not, they still provide a definitive statement of what the system does, rather than a statement of early thoughts about what it was supposed to do.

Details

Agile testing is a continuous process that’s integral to Lean and Built-In Quality. Agile Teams and Agile Release Trains (ARTs) can’t go fast without high quality and they surely can’t implement a Continuous Delivery Pipeline and achieve Release on Demand without continuous, and mostly automated testing. Wherever possible, that means testing—and even test automation—come first.

In agile testing, everyone on the team is a tester. Using Behavior-Driven Development (BDD), Product Managers and Product Owners collaborate with their teams to create tests for features and stories. Developers create tests for code changes using Test-Driven Development (TDD).  With these general tests created by their team members, testers can focus their attention on exercising edge cases and the less intuitive interactions with the system.

Agile testing strives to be test-first. Some tests—such as those for features, stories, and code— can be written first through team collaboration. This approach has many benefits:

• Multiple perspectives provide a broad view of the required system behavior and the best approach to testing it.
• Collaboration creates alignment across the team and a shared understanding of how to implement the behavior.
• It forces developers to think broadly about a change before diving into the implementation.

Other tests—such as performance, security, and user acceptance—are typically less collaborative and more difficult to write first. Nonetheless, developers should still perform these tests early and continually to minimize surprises later in development.

The Agile Testing Matrix

At scale, many types of tests are necessary to ensure quality: tests for code, interfaces, security, stories, larger workflows, etc. By describing a matrix that guides the reasoning behind these tests, Extreme Programming (XP) proponent and Agile Manifesto [1] co-author Brian Marick helped pioneer agile testing. The agile testing matrix provides a taxonomy for organizing these types of tests. This approach was further developed in Agile Testing [2] and extended to scaling the agile paradigm in Agile Software Requirements [2, 3].

Figure 1 describes the latest matrix [4] with guidance on what to test and when.

The horizontal axis of the matrix contains business-or technology-facing tests. Written in business terminology, business-facing tests are designed to be understandable by the user. Written in the language of the developer, technology-facing tests are used to evaluate whether the system delivers the behaviors the developer intended.

The vertical axis on the left side guides development by having the development team think about how they will test a story or section of code before they write it. To find defects or missing features, the right side contains tests critiquing the solution by evaluating the system against the user requirements.

Creating tests across the four quadrants (Q1 – Q4) fosters a comprehensive strategy that helps ensure quality.  (Note: The quadrant number is arbitrary and does not imply any priority to creating tests.)

• Q1 – Contains unit and component tests.  To confirm that the system works as intended, tests are written to run before and after code changes. In software, this is largely the home of Test-Driven Development (TDD).
• Q2 – Contains functional tests (user acceptance tests) for stories, features, and capabilities, to validate that they work the way the Product Owner (or Customer/user) intended. Feature-level and capability-level acceptance tests confirm the aggregate behavior of many user stories. Teams automate these tests using BDD techniques whenever possible and use manual ones only when there is no other choice.
• Q3 – Contains system-level acceptance tests to validate that the behavior of the whole system meets usability and functionality requirements, including scenarios that are often encountered during system use. These may include exploratory tests, user acceptance tests, scenario-based tests, and final usability tests. Because they involve users and testers engaged in real or simulated deployment scenarios, these tests are often manual. They’re frequently the final system validation before delivery of the system to the end-user.
• Q4 – Contains system qualities testing to verify the system meets its Nonfunctional Requirements (NFRs), as exhibited in part by Enabler tests. Typically, they’re supported by a suite of automated testing tools (such as load and performance) designed specifically for this purpose. Since any system changes can violate conformance with NFRs, they must be run continuously, or at least whenever it’s practical.

SAFe’s TDD and BDD articles detail practices for quadrants one and two respectively. Continuous Integration provides more information on quadrant three. NFRs, Continuous Integration, and Release on Demand provide more information on quadrant four.

Learn More

Last update: 10 February 2021