A comparative study of students and professionals in syntactical model comprehension experiments

摘要

Empirical evaluations can be conducted with students or professionals as subjects. Students are much more accessible than professionals and they are inexpensive, allowing a greater number of empirical studies to be conducted. Professionals are preferred over students due to concerns regarding the external validity of student-based experiments. Professionals are believed to perform differently, most likely better than students. But with respect to evaluating the cognitive effectiveness of software engineering notations, are professionals really better? The literature has suggested that the presentation of information is just as critical as the content it conveys, hence necessitating this type of empirical studies. If professionals are not much better than students, then such important finding can be a springboard to many much-needed empirical evaluations in this field. In this paper, we report on an experiment that compare the performances of professionals and students with respect to syntactical model comprehension, which is a core factor for evaluating the cognitive effectiveness of notations. The experiment involved two groups of professionals and two student groups, totaling 74 professionals and 75 students. The results of the experiment indicate that students can be used as an adequate replacement to professionals in such type of empirical studies.