Increasing transformer token length with a Maximum Entropy Principle Method

Understand

Transformers suffer from the computational overhead of their quadratic dependence on the length of sequences processed.

• We present three methods, all adding an intermediate step between training and inference/generation, which extend the autoregressive length of transformers.
• All rely on a Maximum Entropy Principle (MEP) whereby entropy is maximized in the presence of suitable constraints, accounted for by use of Lagrange Multipliers.
• These constraint methods extend the autoregressive character from T to 2T tokens in a linear-with-T fashion.