An Assessment of Software Sneak Analysis

摘要

In 2000 QinetiQ (then DERA) Boscombe Down were faced with results of an analysis of software for a safety critical helicopter system using a technique known as Sneak. Little was known about this technique as a Software Analysis Tool. The company responsible for the analysis, Independent Design Analyses Inc. (IDA) of Houston, Texas, was reluctant to forward details of the technique due to IPR issues. Faced with results from a tool with unknown integrity, QinetiQ Boscombe Down proposed an evaluation of Software Sneak as a means of obtaining some reassurance of the integrity of Sneak as a software analysis tool. QinetiQ (then DERA) Malvern was asked to conduct a suitable evaluation and this paper summarises the results of the evaluation of Software Sneak Analysis. The purpose of the evaluation was to determine if Software Sneak Analysis is rigorous enough to be conducted on Safety Critical Software, and, if possible determine if it could bear comparison with better-known automated techniques. The evaluation took the form of an experiment. QinetiQ provided suitable test code, which IDA analysed using Software Sneak Analysis. The Sneak Analysis methodology is commented on elsewhere [Brennan2001]. IDA performed Software Sneak Analysis at their offices in Houston, Texas, on a small amount of C, Ada and Assembly code witnessed by a QinetiQ analyst. The 4 test cases, comprised approximately 1400 lines of code in total, and contained various errors, some of which were previously found by analyses performed by QinetiQ, and some of which were introduced for the purposes of the evaluation. The software analysed during this evaluation was known to contain 22 distinct errors, considered by QinetiQ to be within the scope of the analysis (they cause the software to fail to meet the specification provided). Of these 22 errors, 19 were successfully found by Sneak Analysis and 3 were missed. In addition, IDA found 2 additional errors previously unknown to QinetiQ. 4 errors claimed by IDA are not in fact errors. To its credit, Sneak Analysis also found 11 potential errors that are dependent on information not provided to IDA.