Top challenges of nonfunctional testing

Functional testing is done by using the functional specifications provided by the client or by using the design specifications like use cases provided by the design team. On contrary the non functional testing is quite complex and often subjective as opposed to functional testing. Here the client tries to document what they expect from the system in terms of how fast, how efficiently, how safely, etc

Now in this article we will check some of the common challenges of non functional testing:

1. Requirements are too subjective: In many cases the non functional requirements are too generic and subject to interpretation. Like people may say "application must response quickly when the user will make a search" or "the application must be secure" etc. Now while doing the testing, this kind of requirement is a big challenge and we need to quantify that. For example, the first requirement can be restated in this way: the application must response in 3 seconds while 100 concurrent users make a search into the system. Like this all the other requirements needs to be quantified before we start doing any testing.
2. Infrastructure overhead: Second challenge is simulating a production like environment. If in production you are having 200 machines running then what should be the ideal load of the test environment. Should we have 100 machines configured which would give realistic performance result during performance testing. So in this 100 machine test bed if we get a result of 6 seconds for a user search then can we say in production with 200 machines the response time would be 3 seconds as per the requirement.
3.  How many concurrent users: Again as we mentioned in the point #1 it is not enough to say I want the application to return a user search result within 3 seconds. If 1 user does that search at that point then 3 seconds is fine. But how about 1000 users are going to perform the same operation say login to the system at the start of the day. Of course that time the memory and CPU utilization would be much higher and consequently the response time would be much lower. So while designing non function test cases you always need to consider the number of concurrent users for a particular transaction.
4. Test Data: This is one of the biggest challenges. Two things you need to determine while running performance load and stress this kind of non functional testing. First, the data volume you need to load in you test environment. If the production had 100 terabytes of data, how much data do we need in the test system? More data will have more impact on performance. Because the database insert would take more time. Second, the data variation is also important. If you use a data where the update, insert, delete operations are not happening then definitely it would take less time. Also if you use the same type of data or same data again and again then the system would simple cache it and in next run the data will be fetched from the cache and the response time would be much lower. So you need to have a production like data volume based on your test infrastructure alone with a good data variation.