Every Web usability study has shown the same thing: users beg us to speed up page downloads. In the beginning, my reaction was along the lines of "let's just give them better design and they will be happy to wait for it". I have since become a reformed sinner since even my skull is not thick enough to withstand consistent user pleas year after year.
Performance Test and Load / Stress Test determine the ability of the application to perform while under load. During Stress/Load testing the tester attempts to stress or load an aspect of the system to the point of failure - the goal being to determine weak points in the system architecture. The tester identifies peak load conditions at which the program will fail to handle required processing loads within required time spans. During Performance testing the tester designs test case scenarios to determine if the system meets the stated performance criteria (i.e. A Login request shall be responded to in 1 second or less under a typical daily load of 1000 requests per minute.). In both cases the tester is trying to determine the capacity of the system under a known set of conditions. The same set of tools and testing techniques can be applied for both types of capacity testing - only the goal of the test changes.
System Test - Capacity Testing
Scheduling Capacity Testing
Performance Test and Stress/Load Test occur during System Test or when enough of the application has been delivered. The earlier capacity testing can be applied, the earlier defects can be detected and mediated. It is critical to detect capacity related defects early because these types of defects often require architectural changes to the system. Because these tests usually depend on functional interfaces, it may be wise to delay until function testing has demonstrated some predefined level of reliability. A balance must be struck between testing early and testing when appropriate.
Designing Capacity Tests
Lack of capacity testing requirements and goals is one of the most common errors made during any capacity testing exercise. Test organizations will often go through the process of capacity testing only to discover the testing results do not present the project with any useful information. The solution is to treat Performance and Stress/Load testing the same as any other testing effort. The test organization needs to perform: Test Planning, Partitioning / Functional Decomposition, Requirements Definition / Verification, Test Case Design, Traceability (Traceability Matrix), Test Case Execution, Defect Management, and Coverage Analysis (for more on this see "Requirements based Function Test").
Implementing Capacity Tests
Both Performance and Stress/Load testing require a toolset that can put the system under a known load and measure the performance of the application while under load. Several shops have developed their own in-shop solutions for capacity testing and there are several capacity testing freeware, shareware, and commercial products available to meet this need. It is easy to fall into the .over-engineering. trap when implementing a capacity testing toolset - to avoid this trap ensure the solution meets the test organizations goals and that the toolset does not become a "goal" in itself.
Implementations
Capacity testing is performed against different aspects of a system. These may exist as standalone applications or as one facet of the system being tested - these include: GUI, Transactional, and Batch. The type of application or aspect of the system undergoing capacity testing has a direct impact on the approach, tooling, and skill set required when performing either Performance or Stress/Load testing. While it is possible to test several aspect of the system at once during capacity testing it is best to focus the testing effort on one aspect of the system at a time - even if the resulting throughput exercises other aspects of the system.
GUI (graphical user interface)
When performing capacity testing against a GUI the focus should be on the ability of the interface to respond to user input. The focus is on the interface not on any transactional aspect of the system (Security, Middleware, Databases, Backend Processing, etc.). To best isolate the testing to the actual interface ensure the remainder of the system is under little (if any load) and then perform the appropriate set of Performance and Stress/Load Tests. Interfaces are often ignored during Performance and Stress/Load testing but applying capacity testing techniques against the interface will often discover a host of defects that might not be detected during Function Test. These can include: Memory Leaks, Memory Management issues, Object Management issues, and functional defects. This type of capacity testing can be performed using the same toolset used for automating Function Test (i.e. Record & Playback).
Transactional
There are several types of transactional-based systems being deployed today: Client/Server, Telephony, Internet-based, and so on. Each implementation or mix of implementations employs an architectural framework composed of hardware, firmware, and software components. The complexity of the architectural solution can often add confusion to the task of capacity testing. Testing the simplest and most common transactions first will often reveal the most critical defects. As an example the tester could start by implementing a test case that simulates users logging unto the system starting with one user and moving up to the number of users expected to be on the system during peak load - for Stress/Load testing simply continue to add Users logging on until the system fails.
When testing organizations first attempt transactional based capacity testing the temptation is to feed a production mix of transactions at the system this often results in so much information being created that the root cause of any failure is difficult to determine. Keeping the transactional mix simple makes the task of defect analysis and remediation easier. Once the system can handle the simpler transactions a more complex set of transactions can be added into the mix. The tactic is to move from simple to complex in a controlled manner to avoid data overload. This type of capacity testing can be performed using an appropriate load-testing tool or toolset.
Batch
Batch processing can be ad-hoc (User Requested) or scheduled. The circumstances around each type of batch processing are often very different and require different testing approaches. Scheduled batch processing often occurs during off-peak hours or when no transactions are permitted. In this situation the tester can focus on the ability of the system to handle large volumes of data with little regard to other aspects of the system (i.e. transactions). Ad-hoc or user requested batch processing must take into account any other business that can occur on the system during the processing of the job including other user requested jobs. Once a set of ad-hoc jobs has been defined the appropriate level of "background noise" must be created. It is often the combination of ad-hoc jobs and background noise that detects defects. Going from a simple set of batch processes to a more complex situation for batch and background noise will help avoid the issue of data overload.
Planning & Staffing
A Performance or Stress/Load testing engagement requires a team that combines an in-depth understanding of the systems: hardware, software, firmware, protocols, transactions, and the business along with load test engineers. Unless capacity testing is an ongoing exercise for the project / system owner this type of talent does not reside in one organization. The Test Manager must work with his peers in operations, development, testing, and any 3rd party IT providers to bring a team together that can address all aspects of the system.
Schedule conflicts and resource allocation are the two most common challenges when planning this type of testing effort. The Test Manager must be both opportunistic (What can be done now?) and Risk sensitive (What must be tested? What should be tested? What can be tested?). Unless capacity testing is a priority to the organization as a whole even the simplest system will not be fully tested - focus on the tests that will have the greatest chance of detecting defects and actually being executed.
