The Software Assurance Technology Center (SATC) in the Systems Reliability
and Safety Office at Goddard Space Flight Center (GSFC) is investigating the use
of software error data as an indicator of testing status. Items of interest for
determining the status of testing include projections of the number of errors
remaining in the software and the expected amount of time to find some
percentage of the remaining errors.
To project the number of errors remaining in
software, one needs an estimate of the total number of errors in the software at
the start of testing and a count of the errors found and corrected throughout
testing. There are a number of models that reasonably fit the rate at which
errors are found in software, the most commonly used is referred to in this
paper as the Musa model. This model is not easily applicable at GSFC, however,
due to the availability and the quality of the error data.
At GSFC, useful error data is not easy to obtain for projects not in the
Software Engineering Laboratory. Of the projects studied by the SATC, only a few
had an organized accounting scheme for tracking errors, but they often did not
have a consistent format for recording errors. Some projects record errors that
were found but did not record any information about resources applied to
testing. The error data frequently contained the date of entry of the error data
rather than the actual date of error discovery. In order to use traditional
models such as the Musa model for estimating the cumulative number of errors,
one needs fairly precise data on the time of discovery of errors and the level
of resources applied to testing. Real world software projects are generally not
very accommodating when it comes to either accuracy or completeness of error
data. The models developed by the SATC to perform trending and prediction on
error data attempt to compensate for these shortcomings in the quantity and
availability of project data.
In order to compensate for the quality of the error data, the SATC developed
a software error trending models using two techniques, each based on the basic
Musa model, but with the constant in the exponential term replaced by a function
of time that describes the 'intensity' of the testing effort. The shape and the
parameters for this function can be estimated using measures such as CPU time or
staff hours devoted to testing. The first technique involves fitting cumulative
error data to the modified Musa model using a least squares fit that is based on
gradient methods. This technique requires data on errors found and the number of
staff hours devoted to testing each week of the testing activity. The second
technique uses a Kalman filter to estimate both the total number of errors in
the software and the level of testing being performed. This technique requires
error data and initial estimates of the total number of errors and the initial
amount of effort applied to testing.
