The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction

Understand

Transformer-based Large Language Models (LLMs) have become a fixture in modern machine learning.

• Correspondingly, significant resources are allocated towards research that aims to further advance this technology, typically resulting in models of increasing size that are trained on increasing amounts of data.
• This work, however, demonstrates the surprising result that it is often possible to significantly improve the performance of LLMs by selectively removing higher-order components of their weight matrices.
• This simple intervention, which we call LAyer-SElective Rank reduction (LASER), can be done on a model after training has completed, and requires no additional parameters or data.