Built on

Similarity of Neural Network Representations Revisited
Original
Kornblith, S.; Norouzi, M.; Lee, H.; and Hinton, G. 2019 · 1905
Earlier work this paper cites.
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Original
Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019 · 1907
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
Original
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015 · 2015
Earlier work this paper cites.
The LAMBADA dataset: Word prediction requiring a broad discourse context
Original
Paperno, D.; Kruszewski, G.; Lazaridou, A.; Pham, Q. N.; Bernardi, R.; Pezzelle, S.; Baroni, M.; Boleda, G.; and Fernández, R. 2016 · 2016
Earlier work this paper cites.
Attention is All you Need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017 · 2017
Earlier work this paper cites.
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Original
Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018 · 2018
Earlier work this paper cites.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018 · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019 · 2019
Earlier work this paper cites.
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019 · 2019
Earlier work this paper cites.
Understanding Robustness of Transformers for Image Classification
Original
Bhojanapalli, S.; Chakrabarti, A.; Glasner, D.; Li, D.; Unterthiner, T.; and Veit, A. 2021 · 2021
Earlier work this paper cites.

Similar

Training Verifiers to Solve Math Word Problems
Original
Cobbe, K.; Kosaraju, V.; Bavarian, M.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021 · 2021
Cited alongside, same era.
How Many Layers and Why? An Analysis of the Model Depth in Transformers
Simoulin, A.; and Crabbé, B. 2021 · 2021
Cited alongside, same era.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Original
Fedus, W.; Zoph, B.; and Shazeer, N. 2022 · 2022
Cited alongside, same era.
OPT: Open Pre-trained Transformer Language Models
Original
Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022 · 2022
Cited alongside, same era.
Comparing Representational and Functional Similarity in Small Transformer Language Models

Then

VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruning
Original
Dutta, O.; Gupta, R.; and Agarwal, S. 2024 · 2024
Closest in time.
STAT: Shrinking Transformers After Training
Original
Flynn, M.; Wang, A.; Alvarez, D. E.; Sa, C. D.; and Damle, A. 2024 · 2024
Closest in time.
LayerShuffle: Enhancing Robustness in Vision Transformers by Randomizing Layer Execution Order
Original
Freiberger, M.; Kun, P.; Løvlie, A. S.; and Risi, S. 2024 · 2024
Closest in time.
Anisotropy Is Inherent to Self-Attention in Transformers
Original
Godey, N.; Éric de la Clergerie; and Sagot, B. 2024 · 2024
Closest in time.
Shortened LLaMA: A Simple Depth Pruning for Large Language Models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Transformer Layers as Painters

Sun, Qi, Pickett, Marc, Nain, Aakash Kumar et al.

Understand

Despite their nearly universal adoption for large language models, the internal workings of transformers are not well understood.

We aim to better understand the impact of removing or reorganizing information throughout the layers of a pretrained transformer.
Such an understanding could both yield better usage of existing models as well as to make architectural improvements to produce new variants.
We present a series of empirical studies on frozen models that show that the lower and final layers of pretrained transformers differ from middle layers, but that middle layers have a surprising amount of uniformity.

Built on

Similarity of Neural Network Representations Revisited
Original
Kornblith, S.; Norouzi, M.; Lee, H.; and Hinton, G. 2019 · 1905
Earlier work this paper cites.
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Original
Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019 · 1907
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
Original
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015 · 2015
Earlier work this paper cites.
The LAMBADA dataset: Word prediction requiring a broad discourse context
Original
Paperno, D.; Kruszewski, G.; Lazaridou, A.; Pham, Q. N.; Bernardi, R.; Pezzelle, S.; Baroni, M.; Boleda, G.; and Fernández, R. 2016 · 2016
Earlier work this paper cites.
Attention is All you Need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017 · 2017
Earlier work this paper cites.
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Original
Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018 · 2018
Earlier work this paper cites.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018 · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019 · 2019
Earlier work this paper cites.
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019 · 2019
Earlier work this paper cites.
Understanding Robustness of Transformers for Image Classification
Original
Bhojanapalli, S.; Chakrabarti, A.; Glasner, D.; Li, D.; Unterthiner, T.; and Veit, A. 2021 · 2021
Earlier work this paper cites.

Similar

Training Verifiers to Solve Math Word Problems
Original
Cobbe, K.; Kosaraju, V.; Bavarian, M.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021 · 2021
Cited alongside, same era.
How Many Layers and Why? An Analysis of the Model Depth in Transformers
Simoulin, A.; and Crabbé, B. 2021 · 2021
Cited alongside, same era.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Original
Fedus, W.; Zoph, B.; and Shazeer, N. 2022 · 2022
Cited alongside, same era.
OPT: Open Pre-trained Transformer Language Models
Original
Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022 · 2022
Cited alongside, same era.
Comparing Representational and Functional Similarity in Small Transformer Language Models

Then

VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruning
Original
Dutta, O.; Gupta, R.; and Agarwal, S. 2024 · 2024
Closest in time.
STAT: Shrinking Transformers After Training
Original
Flynn, M.; Wang, A.; Alvarez, D. E.; Sa, C. D.; and Damle, A. 2024 · 2024
Closest in time.
LayerShuffle: Enhancing Robustness in Vision Transformers by Randomizing Layer Execution Order
Original
Freiberger, M.; Kun, P.; Løvlie, A. S.; and Risi, S. 2024 · 2024
Closest in time.
Anisotropy Is Inherent to Self-Attention in Transformers
Original
Godey, N.; Éric de la Clergerie; and Sagot, B. 2024 · 2024
Closest in time.
Shortened LLaMA: A Simple Depth Pruning for Large Language Models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Friedman, D.; Lampinen, A. K.; Dixon, L.; Chen, D.; and Ghandeharioun, A. 2023 · 2023

Cited alongside, same era.

Mistral 7B

Original

Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; Lavaud, L. R.; Lachaux, M.-A.; Stock, P.; Scao, T. L.; Lavril, T.; Wang, T.; Lacroix, T.; and Sayed, W. E. 2023 · 2023

Cited alongside, same era.

Llama 2: Open Foundation and Fine-Tuned Chat Models

Original

Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; Bikel, D.; Blecher, L.; Ferrer, C. C.; Chen, M.; Cucurull, G.; Esiobu, D.; Fernandes, J.; Fu, J.; Fu, W.; Fuller, B.; Gao, C.; Goswami, V.; Goyal, N.; Hartshorn, A.; Hosseini, S.; Hou, R.; Inan, H.; Kardas, M.; Kerkez, V.; Khabsa, M.; Kloumann, I.; Korenev, A.; Koura, P. S.; Lachaux, M.-A.; Lavril, T.; Lee, J.; Liskovich, D.; Lu, Y.; Mao, Y.; Martinet, X.; Mihaylov, T.; Mishra, P.; Molybog, I.; Nie, Y.; Poulton, A.; Reizenstein, J.; Rungta, R.; Saladi, K.; Schelten, A.; Silva, R.; Smith, E. M.; Subramanian, R.; Tan, X. E.; Tang, B.; Taylor, R.; Williams, A.; Kuan, J. X.; Xu, P.; Yan, Z.; Zarov, I.; Zhang, Y.; Fan, A.; Kambadur, M.; Narang, S.; Rodriguez, A.; Stojnic, R.; Edunov, S.; and Scialom, T. 2023 · 2023

Cited alongside, same era.

A Study on Transformer Configuration and Training Objective

Original

Xue, F.; Chen, J.; Sun, A.; Ren, X.; Zheng, Z.; He, X.; Chen, Y.; Jiang, X.; and You, Y. 2023 · 2023

Cited alongside, same era.

Evolutionary Optimization of Model Merging Recipes

Original

Akiba, T.; Shing, M.; Tang, Y.; Sun, Q.; and Ha, D. 2024 · 2024

Cited alongside, same era.

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Original

Biderman, S.; Schoelkopf, H.; Anthony, Q.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; Skowron, A.; Sutawika, L.; and van der Wal, O. 2023a

Cited in the paper.

Pythia: A suite for analyzing large language models across training and scaling

Biderman, S.; Schoelkopf, H.; Anthony, Q. G.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; et al. 2023b

Cited in the paper.

Original

Kim, B.-K.; Kim, G.; Kim, T.-H.; Castells, T.; Choi, S.; Shin, J.; and Song, H.-K. 2024 · 2024

Closest in time.

The Remarkable Robustness of LLMs: Stages of Inference?

Original

Lad, V.; Gurnee, W.; and Tegmark, M. 2024 · 2024

Closest in time.

ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

Original

Men, X.; Xu, M.; Zhang, Q.; Wang, B.; Lin, H.; Lu, Y.; Han, X.; and Chen, W. 2024 · 2024

Closest in time.