Security Testing Versus Functional Testing

Functional testing is testing that is performed on behalf of a legitimate user of the product who is attempting to use it in the way it was intended to be used and for its intended purpose. This is who the functional tester is really the advocate for; thus, the majority of functional testing is done from the viewpoint of a customer.

It is important to realize that testing from only this viewpoint will cause you to bypass a large percentage of security tests. Most security vulnerabilities, although they have a chance of being discovered (mostly accidentally) by the intended customers, are unlikely to be exploited by them. Instead, the customer may call technical support to report the bug or maybe just grumble about it to friends or acquaintances. It's unlikely that many of the intended customers will even recognize that bug as more than a nuisance or sign of poor quality, let alone correctly see it as a security risk.

The attention of functional testing is much more focused on how to enable the customers to perform their tasks in the easiest and most convenient way possible while providing enough checks and safety measures so that they can't cause inadvertent harm too easily. It's a sort of "protect them from themselves" mentality. If any security testing is done, it tends to focus on things such as permissions and privileges but, again, only based around the assumption that the customer is using something like the login functionality as intended.

In essence, because you are performing tests on behalf of a customer, you are trusting that all people using the software you are testing are customers and not merely consumers.

All trust is misplaced. Don't trust that everyone who uses all or part of your project will do so as it was designed to be used or for its intended purpose. It is far more realistic to recognize that a subset of people will intentionally attempt to misuse all or part of your product.

All Consumers Are Not Customers
The first step to take in order to move to more of a security mindset is to stop thinking of the users of your project as "customers" and start thinking of them as "consumers." This provides a way to differentiate the set of users your product is designed for from the set of users who may also include people your product was not designed for.

Customers are the people or organizations that your software is intentionally written to solve a problem or problems for. They have been the main focus throughout the entire development cycle, from the gathering of requirements through the implementation, and that then provides the basis for functional testing. For example, the customers for a software program designed to streamline writing a novel are the novelists. Those are the people that the specifications and requirements are written around and from whose point of view functional testing is done.

Consumers, on the other hand, are those people or organizations that might use your software in a way it was or was not intended and who are not included in your customers. Sometimes your product's consumer base grows because your product is able to perform some task as part of its normal repertoire, and that task is all that the consumer wishes to accomplish.

Sometimes it is because your product interfaces with some other software or hardware, and the consumer wants to use that ability to interface to their own advantage or because they think it may be exploitable.

In the prior example of the novel-writing software, the product may have the ability to take a delimited text file and translate its information into the proprietary file format of a major scheduling software system. If consumers merely want to avoid having to use the scheduling software's UI (or even buying a copy of the scheduling software) before being able to share their calendar with others, they can just run their delimited file through the functionality of your software to convert the file, then share the file out.

Other consumers can use that same functionality to generate a specialized file for the scheduling software to consume that takes advantage of a security vulnerability in the scheduling software. Perhaps the scheduling software itself cannot generate a file that would exploit their security bug, but a file that included the exploit would be successfully consumed by the scheduling software and cause the vulnerability to be exploited.

All trust is misplaced. Don't trust all consumers of your software to be actual customers. All customers are consumers (or potential consumers), but not all consumers are customers. Further, there are several classes of non-customer consumers, ranging from convenient consumers to those who are intending to take malicious actions against your product or using your product's functionality to attack something else.

The Intent of Security Testing Versus Functional Testing
The most basic premise behind traditional functional testing is that it is meant to validate that the software you are testing fulfills its requirements and it functions as intended. In other words, it does what it's supposed to. There are usually some basic security tests included in functional testing, typically around items such as passwords and permissions, and whatever login or authentication method is in use. Even these are intended to test within the system's expectations - permissions testing only tends to test items such as insuring that if you are not logged in as an administrator, you cannot carry out administrative functions. It does not, however, focus on items such as how to obtain administrative privileges outside of the user login.

Although "security" is often presented as merely an aspect of functional testing, it really needs to be considered and planned separately. It may take place alongside functional testing, but it has almost an opposite focus.

"Positive" Versus "Negative" Testing
Positive and negative testing are terms that are used periodically in software testing to differentiate between the base intent of the testing being done. Although these terms are used regularly, their exact definitions are often in doubt because there are easily as many interpretations of the terms as there are testers.

My personal opinion is that, especially in the context of security testing, the concept of "positive" versus "negative" testing is so arbitrary as to be pretty much useless. All testers can, of course, think about their testing as they wish, but these terms won't be used in this book.

Although quite a few authors and instructors stress the concept of whether testing is "positive" or "negative," I rarely refer to or think of any of my day-to-day testing as either. Personally, I think that it's much more useful to think of testing in terms of atomic statements of behavior that can then be tested to determine a pass or fail for these very non-ambiguous statements. I don't particularly think this is a bad or wrong way to view testing or to include in your test process, but I don't personally use it or consider it meaningful for security testing. There is a lot more information on this in Chapter 28, which focuses on the process of Test Case Outlining.

Test Overlap and Streamlining
The distinct mindsets of functional and security testing serve different purposes, and each one tends to compensate for some failures or blind spots in the other. Functional bugs are found while running security tests and vice versa. Sometimes tests for both aspects are intermingled and run concurrently, though the results have to be evaluated separately.

That being said, despite the common wisdom of reducing test time and expense by eliminating redundant test cases, I strongly feel that there should be overlap between purely functional test cases and security test cases. This is because the pass or fail criteria for the two types of tests can be so different. I've found that more bugs are missed when assessing the results of a test case if the tester is attempting to assess both functional and security criteria at the same time. The mindset is different as well as the validation and verification, and it's far too easy to not be 100 percent thorough with both.

As an example, if I am running boundary tests on my software's signup function, and specifically on the username field, it would be a typical functional test for me to try employing a username that is the length of the username field +1 long. If I was running a black-box security test on that field, I would try a very long username and construct it with a specific set of characters that I could recognize if it came up in an error dialogue or in a tool I may be using to examine the contents of memory, the registry, etc. Although these are both overly long entry tests, they are not equivalent and should really not be combined into a single test.

Most of the time, the steps required to set up and run a security case are not well suited to a functional case, and combining them isn't attempted, but if the question comes up, think carefully before that type of test case consolidation is attempted.

That said, I have had the experience, while running one type of test, of getting ideas for another type of test, and those flashes of intuition should not be ignored. I advocate always keeping a notepad at hand.

Changing Your Prioritizations

Code Maturity
Traditionally, the guideline for testing existing code is that the more mature the code is, the smaller the percentage of bugs that will be found in that code. This is based on the fact that bugs are often found by users and through prior test passes, so there are fewer and more esoteric functional bugs left to find as time goes on. This "stable" code is often only given a cursory once-over during a functional test pass. When it comes to security testing, this becomes quite a bit more complex and convoluted - almost to the point of it being reversed from the mindset of functional testing. If dedicated security testing has only recently begun on your project, it's actually more likely for the older code to contain a greater percentage of security defects than the newer code for several reasons:

• More security vulnerabilities and attack vectors are found all the time, so code that was written before those came to light may still contain them.
• More ways are found to exploit security vulnerabilities previously deemed to be unexploitable every day, so code that was written prior to those exploits is more likely to still contain those vulnerabilities.
• More tools are being created and distributed that allow exploits to be created and attacks carried out by a greater number of people who have far less technical knowledge than ever before. This increases the odds that any security vulnerabilities already present in existing code will be found and exploited, and having these vulnerabilities in existence and exploitable for longer periods of time gives these tools and attackers longer to discover and craft exploits for them.
• Developers (and testers) have become more security conscious over time and are less likely to make the same mistakes today that they might have made previously. This is partly due to receiving more security training and partly due to self-education, as vulnerability exploits hit the news.

Code Complexity
The rule of thumb for testing when it comes to code complexity remains basically the same. The more complex the code in the project, the more likely it has to have functional bugs. The more complex the code, the more likely it is to have security vulnerabilities as well.

Code Coverage
This speaks more about how much prior testing the code has received but specifically about how much prior security testing. If the code has received extensive security testing previously via code analysis (code reviews or security test tools), it's less likely to contain security vulnerabilities than code that has never been examined before.

You should know that it is very difficult to use automated code coverage tools to provide meaningful data on security testing. The standard of measurement for code coverage metrics is the percentage of code touched during a test pass, but this isn't terribly meaningful for security test efforts. Although it's common (and wise) for white-box testers to walk through all the code in search of security vulnerabilities, the actual percentage of code that is touched by testing is really most meaningful to functional testing.