Development and analysis of an off-the-shelf product objectives evaluation model (POEM).

摘要

The purpose of this research was to develop and analyze a methodology for evaluating and selecting (E/S) Off-the-Shelf (OTS) products. The Product Objective Evaluation Model (POEM) was defined in terms of a theoretically based evaluation process, product evaluation objectives, quantitative analysis technique, and supporting decision tool. The study identified how the model integrates with the system design process. The product evaluation objectives set was designed to include design and business factors. The suitability of the Analytic Hierarchy Process (AHP) and Multiattribute Utility Theory (MAUT) as analysis techniques was also investigated.; Evaluation experts were used to develop a product evaluation objectives tree by defining a thorough but generalizable objectives set. Evaluation experts validated POEM prior to the experiment. The experts reviewed a prior evaluation's product information and provided inputs to an existing accepted E/S model and POEM. The models' outputs yielded a statistically significant correlation of product rankings establishing POEM's validity and reliability.; A Posttest-only Counterbalanced design experiment evaluated POEM effectiveness and user satisfaction compared with a current E/S process. The experiment was conducted during an actual OTS product E/S exercise using randomly assigned qualified product evaluators. t-Tests and correlation tests on the empirical data yielded statistically significant results. The results supported the hypothesis that using a robust E/S model can improve an evaluator's effectiveness and satisfaction in evaluating and selecting OTS products. Product rankings created by POEM had statistically significant correlations to the current method's ranking. While the POEM product evaluation form took longer to complete, it was easier to understand and use. Thoroughly considering product attributes that affect product life-cycle utility improved user effectiveness. Quantifiable and defensible product rankings improved user confidence and satisfaction. The correlation of AHP and MAUT rankings indicated equivalent performance and appropriate application.; POEM can improve OTS product evaluation, selection business processes, and risk reduction by providing guidance towards more informed, efficient, and effective resource application. This process improvement could help an organization or program improve the Software Engineering Institute Capability Maturity Model level. Beyond, product evaluation, the framework and concept of utilizing focused decision objectives can be applied across any multi-objective decision-making domain.