Reconciler: A Workflow for Certifying Computational Research Reproducibility
share
Previous work in reproducibility focused on providing frameworks to make research reproducible, however fewer work was done on reproducibility certification. We propose the Reconciler, that certifies the reproducibility of computational research papers written in . Two Docker containers are built and run in separate Virtual Machines (VMs) reproducing the figures, tables and variables (results) of the paper. Then the results are used during the compilation in each VM, to produce corresponding pdf files. Lastly a third VM certifies the research paper as reproducible, if and only if the resulted pdf files are identical. The two builds are needed to enable comparison between different builds of the same source code and any non-deterministic stochasticity would produce different pdf files for different builds, thus making the certifier VM report the paper as non-reproducible. We release an open source implementation of the Reconciler, that certifies the reproducibility of this paper.
READ FULL TEXT