Fetching the paper…
Reading the bibliography…
In-context learning is a powerful emergent ability in transformer models.
Are sixteen heads really better than one?
Michel, P., Levy, O., and Neubig, G · 1905
Earlier work this paper cites.
Causal mediation analysis for interpreting neural NLP: the case of gender bias
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. M · 2004
Earlier work this paper cites.
Millisecond-timescale, genetically targeted optical control of neural activity
Boyden, E. S., Zhang, F., Bamberg, E., Nagel, G., and Deisseroth, K · 2005
Earlier work this paper cites.
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2012
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge, 2015
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L · 2015
Earlier work this paper cites.
Layer normalization, 2016
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Earlier work this paper cites.
JAX: composable transformations of Python+NumPy programs, 2018
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q · 2018
Earlier work this paper cites.
The lottery ticket hypothesis: Training pruned neural networks
Frankle, J. and Carbin, M · 2018
Earlier work this paper cites.
Constraints on neural redundancy
Hennig, J. A., Golub, M. D., Lund, P. J., Sadtler, P. T., Oby, E. R., Quick, K. M., Ryu, S. I., Tyler-Kabara, E. C., Batista, A. P., Yu, B. M., and Chase, S. M · 2018
Earlier work this paper cites.
Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I · 2019
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020
Earlier work this paper cites.
The DeepMind JAX Ecosystem, 2020
DeepMind, Babuschkin, I., Baumli, K., Bell, A., Bhupatiraju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Dedieu, A., Fantacci, C., Godwin, J., Jones, C., Hemsley, R., Hennigan, T., Hessel, M., Hou, S., Kapturowski, S., Keck, T., Kemaev, I., King, M., Kunesch, M., Martens, L., Merzic, H., Mikulik, V., Norman, T., Papamakarios, G., Quan, J., Ring, R., Ruiz, F., Sanchez, A., Sartran, L., Schneider, R., Sezener, E., Spencer, S., Srinivasan, S., Stanojević, M., Stokowiec, W., Wang, L., Zhou, G., and Viola, F · 2020
Earlier work this paper cites.
interpreting gpt: the logit lens, 2020
nostalgebraist · 2020
Earlier work this paper cites.
Sparse interventions in language models with differentiable masking
Cao, N. D., Schmid, L., Hupkes, D., and Titov, I · 2021
Earlier work this paper cites.
A mathematical framework for transformer circuits
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C · 2021
Earlier work this paper cites.
Causal analysis of syntactic agreement mechanisms in neural language models
Finlayson, M., Mueller, A., Gehrmann, S., Shieber, S. M., Linzen, T., and Belinkov, Y · 2021
Cited alongside, same era.
Causal abstractions of neural networks
Geiger, A., Lu, H., Icard, T., and Potts, C · 2021
Cited alongside, same era.
Equinox: neural networks in JAX via callable PyTrees and filtered transformations
Kidger, P. and Garcia, C · 2021
Cited alongside, same era.
Roformer: Enhanced transformer with rotary position embedding
Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y · 2021
Cited alongside, same era.
An explanation of in-context learning as implicit bayesian inference
Tracr: Compiled transformers as a laboratory for interpretability
Lindner, D., Kramár, J., Rahtz, M., McGrath, T., and Mikulik, V · 2023
Later among the works it cites.
The hydra effect: Emergent self-repair in language model computations, 2023
McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., and Legg, S · 2023
Later among the works it cites.
200 concrete open problems (COP) in MI: Analysing training dynamics, 2023
Nanda, N · 2023
Later among the works it cites.
Progress measures for grokking via mechanistic interpretability, 2023
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J · 2023
Later among the works it cites.
Towards vision-language mechanistic interpretability: A causal tracing tool for blip, 2023
Palit, V., Pandey, R., Arora, A., and Liang, P. P · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T · 2021
Cited alongside, same era.
Data distributional properties drive emergent in-context learning in transformers
Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., and Hill, F · 2022
Cited alongside, same era.
Multi-component learning and s-curves, 2022
Jermyn, A. and Shlegeris, B · 2022
Cited alongside, same era.
Rethinking the role of demonstrations: What makes in-context learning work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L · 2022
Cited alongside, same era.
A comprehensive mechanistic interpretability explainer & glossary, Dec 2022
Nanda, N · 2022
Cited alongside, same era.
Transformerlens
Nanda, N. and Bloom, J · 2022
Cited alongside, same era.
In-context learning and induction heads
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C · 2022
Cited alongside, same era.
We just introduced pytorch 2.0 at the #pytorchconference, introducing torch.compile!, 2022
PyTorch · 2022
Cited alongside, same era.
On the special role of class-selective neurons in early training, 2023
Ranadive, O., Thakurdesai, N., Morcos, A. S., Leavitt, M., and Deny, S · 2023
Later among the works it cites.
Generalization to New Sequential Decision Making Tasks with In-Context Learning, December 2023
Raparthy, S. C., Hambro, E., Kirk, R., Henaff, M., and Raileanu, R · 2023
Later among the works it cites.
The mechanistic basis of data dependence and abrupt learning in an in-context classification task, 2023
Reddy, G · 2023
Later among the works it cites.
The transient nature of emergent in-context learning in transformers, 2023
Singh, A. K., Chan, S. C. Y., Moskovitz, T., Grant, E., Saxe, A. M., and Hill, F · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models, 2023
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T · 2023
Later among the works it cites.
Transformers learn in-context by gradient descent, 2023
von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M · 2023
Later among the works it cites.
Label words are anchors: An information flow perspective for understanding in-context learning, 2023
Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., and Sun, X · 2023
Later among the works it cites.
Efficient In-Context Learning in Vision-Language Models for Egocentric Videos, November 2023
Yu, K. P., Zhang, Z., Hu, F., and Chai, J · 2023
Later among the works it cites.
The clock and the pizza: Two stories in mechanistic explanation of neural networks, 2023
Zhong, Z., Liu, Z., Tegmark, M., and Andreas, J · 2023
Later among the works it cites.
Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms, 2024
Chen, A., Shwartz-Ziv, R., Cho, K., Leavitt, M. L., and Saphra, N · 2024
Closest in time.
Circuit component reuse across tasks in transformer language models, 2024
Merullo, J., Eickhoff, C., and Pavlick, E · 2024
Closest in time.
”attention”, ”transformers”, in neural network ”large language models”, 2024
Shalizi, C. R · 2024
Closest in time.
pyvene: A library for understanding and improving PyTorch models via interventions
Wu, Z., Geiger, A., Arora, A., Huang, J., Wang, Z., Goodman, N. D., Manning, C. D., and Potts, C · 2024
Closest in time.