Software Estimation Models: When is Enough Data Enough?

摘要

In summary, and in answer the question in the title of this paper, COCONUT-based calibration can improve the performance of an effort estimation model. That improvement rises sharply to ten projects, but the improvements continue till 40 projects. Further, after 10 projects, the variance in the estimates keep shrinking to around 30 projects. It would be an error to summarize this study as (say) "here are new a,b parameters for COCOMO-Ⅰ".The large variances in the inter-organizational data (COCOMO-Ⅰ) of Figure 6 show that we can't just offer a single point estimate for COCOMO-Ⅰ tunings (at least, if those tunings are learnt by COCONUT). The conclusions of this paper have to be interpreted within the context of COCONUT. Figure 1 is the preferred format and such plots can be summarized as follows: Based on sequence tuning experiments on N past projects, we predict that we can estimate effort on future projects within such-and-such intervals. (those intervals being read off the min-max curves of Figure 1). Further, a sequence tuning experiment might also conclude that: Based on the history of projects seen to date, we predict that the variance in our effort estimations will reduce by this much if we can collect data from these many more projects. Lest this reports appears too critical of COCOMO, it is important to note that COCONUT is an extension to COCOMO-Ⅰ-1981 and could not work without it. As to COCOMO-Ⅱ-2000, the COCONUT results offer a simpler method of obtaining the same, or even better, results: 1. One of the main motivations for the Bayesian analysis of COCOMO-Ⅱ was the regression results from the 83+78 projects had slopes that contradicted certain expert intuitions. For example regression of the COCOMO data concluded that building reusable components decreased development costs. Most experts believe that the extra effort required to generalize a design actually increases the cost of building such components. This anomaly was explained as follows: the 83+78 projects did not contain enough samples of projects that make heavy use of reuse. To DELPHI panel and the subsequent Bayesian tuning was used to fill in the gaps in the project data with expert knowledge. This combination of DELPHI+Bayesian methods proved successful: COCOMO-Ⅱ-2000 had much higher PRED(N) levels than COCOMO-Ⅰ. 2. Given the COCONUT results, a much simpler method for incorporating expert knowledge is possible. Recall that COCONUT can find reach good PRED(20) after 30 to 40 projects. This number of projects could be artificially generated from, say, 4 experts each asked to describe 10 exemplar projects showing the range of projects typically done at their company. If those descriptions were made in terms of the COCOMO-Ⅰ parameters, then COCONUT could then tune an effort model to that expert option using those 40 expert-generated examples.