-
- A most important criterion for a test script is that it must be
repeatable. Provided the environment is set up correctly, the test should
produce the same result each time it is executed. But please note the
dependence on the environment - usually data but potentially other factors
as well such as computer or network configuration.
- There is a risk of "false negatives", i.e. tests said to have been
deemed to have failed but where the failure is due to extraneous errors. For
example, there was an error in the script and the "pass" criteria was not
obtainable, there was a mistake in typing in data (say a wrong selection of
"yes" or "no" - simple but devastating) or required pre-set data did not
exist (the tests had been run out of sequence). It is of course also
possible to make a mistake in the "pass" criteria - hence the script is
faulty not the system.
All "failed" tests should be investigated to establish a reason for
failure. Because of the complexity of testing, in many cases it is a "false
negative", and the root cause needs to be identified so that the test can be
re-run properly. Occasionally investigations will indicate a "false
positive" from another test, which had apparently passed but the system had
failed to carry out some action or other during that test on which this test
depended. Whilst the result might be still one test that failed (albeit a
different one) it may affect the classification of the failure (and hence
overall acceptability). On the other hand it may again simply be that the
test script for that test failed to ensure that the action was carried out -
i.e. a script error not a system error. There are then a string of tests to
be repeated, or if time is not available, discounted.
