A framework for comparing multiple cost estimation methods using an automated visualization toolkit

摘要

Context: The importance of accurate predictions in Software Cost Estimation and the related challenging research problems, led to the introduction of a plethora of methodologies in literature. However, the wide variety of cost estimation methods, the techniques for improving them and the different measures of accuracy have caused new problems such as the inconsistent findings and the conclusion instability. Today, there is a confusion regarding the choice of the most appropriate method for a specific dataset and therefore a need for well-established statistical frameworks as well as for automated tools that will reinforce and lead a comprehensive experimentation and comparison process, based on the thorough study of the cost estimation errors.Objective: The purpose of this paper is to present a framework for visualization and statistical comparison of the errors of several cost estimation methods. It is based on an automated tool which can facilitate strategies for an intelligent decision-making.Method: A systematic procedure comprised of a series of steps corresponding to research questions is proposed. For each of the steps, StatREC, a Graphical User Interface statistical toolkit is utilized. StatREC was designed and developed to take as input a simple data matrix of predictions by multiple models and to provide a variety of graphical tools and statistical hypothesis tests for aiding the users to answer the questions and choose the appropriate model themselves.Results: The study of prediction errors by the proposed framework provides insight of several aspects related to prediction performance of different models. The systematic examination of candidate models by a series of research questions supports the user to make the final decision.Conclusion: Structured procedures based on automated tools like StatREC can efficiently be used for studying the error and comparing cost estimation models.