Methods to evaluate lightweight software process assessment methods based on evaluation theory and engineering design principles

摘要

Achieving a mature software development process has become essential for many software organizations. A mature development process permits software organizations to provide their customers with a high quality software product delivered on time and within budget.ududSoftware organizations have been struggling for decades to improve the quality of their products by improving their software development processes. Designing an improvement program for a software development process is a demanding and complex task. This task consists of two main processes: the assessment process and the improvement process. A successful improvement process requires first a successful assessment; failing to assess the organization's software development process could create unsatisfactory results.ududSoftware processes assessment (SPA) can be used either to determine the capability of another organization, for subcontracting purposes, or to determine and understand the status of the organization's current processes to initiate an improvement process. The increasing number of assessment approaches available; the ISO 15504 standard that sets out the requirements for process assessment; and the popularity of the CMMI model, illustrate the relevance of software process assessment for the software development industry.ududCurrently, several methods are available to assess the maturity and capability of a software development process based on well-known software process assessment and improvement frameworks such as CMMI and ISO-15504. The success of these assessment methods and improvement frameworks is supported by post-development studies on the validity, reliability and effectiveness of these methods. Unfortunately, many researchers consider that such methods are too large to implement in SME organizations. As a result, many researchers have studied process assessment and improvement in SME organizations and proposed assessment methods, usually called lightweight SPA methods, suitable to the organizations' needs.ududThe current research in the SPA field focuses on proposing convenient and easy-to-use assessment methods, without investigating to what extent the design of these methods is related to the engineering design perspective. This unclear alignment with the engineering discipline raises questions about the relevance and representativeness of the results produced by these methods from an engineering viewpoint. Moreover, although numerous SPA methods are currently available which offer help and guidance, unfortunately they only partially address evidences found essential for achieving an SPA success.ududThis thesis presents and discusses the evaluation of lightweight SPA methods. The evaluation is two-fold: evaluating the SPA methods design using a top-down approach and based on engineering viewpoints and evaluating the success of SPA methods using a bottom-up approach. The evaluation theory concepts are used as a framework to formally develop both evaluation methods.ududTo develop the first evaluation method using the top-down approach, an exploratory analytical study of SPA methods from an engineering design viewpoint has been conducted. Vincenti's classification has been used as a tool for this analysis. The aim of this exploratory study is to put the developed SPA methods into an engineering design framework, and use this framework as a guideline to put the new SPA methods to be designed into the same engineering design framework. To develop the second evaluation method using the bottom-up approach, a systematic literature review was conducted to extract the set of evidences for the success of the SPA method based on requirements, observations, lessons learned and recommendations which have been formulated within the industry and published in books, conferences and journals.ududThe development process of the two evaluation methods has then been verified using a set of verification criteria and the proposed evaluation methods were tested by conducting three case studies. The first evaluation method would be useful mainly for the designers of new SPA methods during the design phase, while the second evaluation method would be useful for both designers and practitioners of SPA methods to verify the success of the assessment method in question.ududThis research project forms an entry point to study the alignment of SPA methods design with engineering design principles and sheds light on achieving successful assessment results by studying the successful evidences that should be supported by assessment methods separated from the improvement process. The proposed evaluation methods in this thesis have great benefits for SPA methods designed mainly for SME organizations, because these assessments methods, contrary to well-known methods, lack comprehensive studies on their reliability and effectiveness.