On Bock's Conjecture Regarding the Adam Optimizer

11/16/2021

∙

by   Mohamed Akrout, et al.

∙

11

∙

share

In 2014, Kingma and Ba published their Adam optimizer algorithm, together with a mathematical argument that was meant to help justify it. In 2018, Bock and colleagues reported that a key piece was missing from that argument - an unproven lemma which we will call Bock's conjecture. Here we show that this conjecture is false, but a modified version of it does hold, and fills the gap in Bock's proof of convergence for Adam.