Model adjointization and its cost

摘要

In this article, the least program behavior decomposition method (LPBD) is put forward from a program structure point of view. This method can be extensively used both in algorithms of automatic differentiation (AD) and in tools design, and does not require programs to be evenly separable but the cost in terms of operations count and memory is similar to methods using checkpointing. This article starts by summarizing the rules of adjointization and then presents the implementation of LPBD. Next, thedefinition of the separable program space, based on the fundamental assumptions (FA) of automatic differentiation, is given and the differentiation cost functions are derived. Also, two constants of fundamental importance in AD, a and (i, are derived under FA. Under the assumption of even separability, the adjoint cost of simple and deep decomposition is subsequently discussed quantitatively using checkpointing. Finally, the adjoint costs in terms of operations count and memory through the LPBD method are shown to be uniformly dependent on the depth of structure or decomposition.