Accounting for realities when estimating the field failure rate of software

A realistic estimate of the field failure rate of software is essential in order to decide when to release the software while maintaining an appropriate balance between reliability, time-to-market and development cost. Typically, software reliability models are applied to system test data with the hope of obtaining an estimate of the software failure rate that will be observed in the field. Unfortunately, test environments are usually quite different from field environments. In this paper, we use a calibration factor to characterize the mismatch between the system test environment and the field environment, and then incorporate the factor into a widely used software reliability model. For projects that have both system test data and field data for one or more previous releases, the calibration factor can be empirically evaluated and used to estimate the field failure rate of a new release based on its system test data. For new projects, the calibration factor can be estimated by matching the software to related projects that have both system test data and field data. In practice, isolating and removing a software fault is a complicated process. As a result, a fault may be encountered more than once before it is ultimately removed. Most software reliability growth models assume instantaneous fault removal. We relax this assumption by relating non-zero fault removal times to imperfect debugging. Finally, we distinguish between two types of faults based on whether their observed occurrence would precipitate a fix in the current or future release, respectively. Type-F faults, which are fixed in the current release, contribute to a growth component of the overall failure rate. Type-D faults, whose fix is deferred to a subsequent release, contribute to a constant component of the overall software failure rate. The aggregate software failure rate is thus the sum of a decreasing failure rate and a constant failure rate.