Understanding Addition in Transformers

Understand

Understanding the inner workings of machine learning models like Transformers is vital for their safe and ethical use.

• This paper provides a comprehensive analysis of a one-layer Transformer model trained to perform n-digit integer addition.
• Our findings suggest that the model dissects the task into parallel streams dedicated to individual digits, employing varied algorithms tailored to different positions within the digits.
• Furthermore, we identify a rare scenario characterized by high loss, which we explain.