Higher Order Linear Transformer

10/28/2020

∙

by   Jean Mercat, et al.

∙

13

∙

share

Following up on the linear transformer part of the article from Katharopoulos et al., that takes this idea from Shen et al., the trick that produces a linear complexity for the attention mechanism is re-used and extended to a second-order approximation of the softmax normalization.