A tool for publishing reproducible algorithms & A reproducible, elegant algorithm for sequential experiments

摘要

Tools to ease the burden of reproducibility are important so computer science does not fall into the trap of “cargo cult” science: particularly publishing discussions of algorithms that look like algorithms but which do not work properly when they are copied from the paper.This paper introduces a tool, calledrelit, which makes it very easy to write about and publish correct algorithms, yet without restricting the author's style. In fact,relitcan be used with any material: mathematics, proofs, algorithms or programs. It can be used in papers, in reports and books and, with analogous advantages, in student work — where examiners may wish to automatically check what the student claims to have written is actually correct.To demonstraterelit, this paper presents a new, elegant algorithm for the design of sequential experiments to efficiently control bias, drift, random error, carry-over and other effects. The algorithm is written in C, in a clear style to simplify porting to other languages.We developedrelitbecause it was impossible to find simple reproducible code for this problem, and we wanted to do better. Thanks torelit, the published algorithm is reproducible and worksexactlyas published in the present paper. This paper also includes discussion of the problems and opportunities of reproducibility and the essential contributions ofrelit-style approaches to improving the reliability of computer science publications.