Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning

Understand

Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep methods, high-order methods, and multi-particle dynamical systems.

• The precision of the solution to ODEs significantly affects parameter optimization, thereby impacting model performance.
• In this work, we present a series of advanced explorations of Transformer architecture design to minimize the error compared to the true ``solution.'' First, we introduce a predictor-corrector learning framework to minimize truncation errors, which consists of a high-order predictor and a multistep corrector.
• Second, we propose an exponential moving average-based coefficient learning method to strengthen our higher-order predictor.