Built on

M. Smithson, “Conflict aversion: preference for ambiguity vs conflict in sources and evidence,” Organizational behavior and human decision processes , vol. 79, no. 3, pp. 179–198, 1999
1999
Earlier work this paper cites.
K. B. Korb, L. R. Hope, A. E. Nicholson, and K. Axnick, “Varieties of causal intervention,” in PRICAI 2004: Trends in Artificial Intelligence: 8th Pacific Rim International Conference on Artificial Intelligence, Auckland, New Zealand, August 9-13, 2004. Proceedings 8 . Springer, 2004, pp. 322–331
2004
Earlier work this paper cites.
H. Bang and J. M. Robins, “Doubly robust estimation in missing data and causal inference models,” Biometrics , vol. 61, no. 4, pp. 962–973, 2005
2005
Earlier work this paper cites.
J. Pearl, Causality . Cambridge university press, 2009
2009
Earlier work this paper cites.
F. Johansson, U. Shalit, and D. Sontag, “Learning representations for counterfactual inference,” in International conference on machine learning . PMLR, 2016, pp. 3020–3029
2016
Earlier work this paper cites.
A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.
U. Shalit, F. D. Johansson, and D. Sontag, “Estimating individual treatment effect: generalization bounds and algorithms,” in International Conference on Machine Learning . PMLR, 2017, pp. 3076–3085
2017
Earlier work this paper cites.
S. Zhao, Q. Wang, S. Massung, B. Qin, T. Liu, B. Wang, and C. Zhai, “Constructing and embedding abstract event causality networks from text snippets,” in Proceedings of the Tenth ACM International Conference on Web Search and Data Mining , 2017, pp. 335–344
2017
Earlier work this paper cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347 , 2017
Original
2017
Earlier work this paper cites.
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
K. Kuang, P. Cui, S. Athey, R. Xiong, and B. Li, “Stable prediction across unknown environments,” in proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , 2018, pp. 1617–1626
2018
Earlier work this paper cites.
M. Rojas-Carulla, B. Schölkopf, R. Turner, and J. Peters, “Invariant models for causal transfer learning,” Journal of Machine Learning Research , vol. 19, no. 36, pp. 1–34, 2018
2018
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019
2019
Earlier work this paper cites.
R. Zmigrod, S. J. Mielke, H. Wallach, and R. Cotterell, “Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019, pp. 1651–1661
2019
Earlier work this paper cites.
Y. Liu, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019
Original
2019
Earlier work this paper cites.
Z. Lan, “Albert: A lite bert for self-supervised learning of language representations,” arXiv preprint arXiv:1909.11942 , 2019
Original
2019
Earlier work this paper cites.
V. Sanh, “Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108 , 2019
Original
2019
Earlier work this paper cites.
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” arXiv preprint arXiv:1907.02893 , 2019
Original
2019
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research , vol. 21, no. 140, pp. 1–67, 2020
2020
Earlier work this paper cites.
T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , 2020
Original
2020
Earlier work this paper cites.
K. Webster, X. Wang, I. Tenney, A. Beutel, E. Pitler, E. Pavlick, J. Chen, E. Chi, and S. Petrov, “Measuring and reducing gendered correlations in pre-trained models,” arXiv preprint arXiv:2010.06032 , 2020
Original
2020
Earlier work this paper cites.
D. Kaushik, E. Hovy, and Z. Lipton, “Learning the difference that makes a difference with counterfactually-augmented data,” in International Conference on Learning Representations , 2020. [Online]. Available: https://openreview.net/forum?id=Sklgs0NFvr
2020
Earlier work this paper cites.
D. Zhang, H. Zhang, J. Tang, X.-S. Hua, and Q. Sun, “Causal intervention for weakly-supervised semantic segmentation,” Advances in Neural Information Processing Systems , vol. 33, pp. 655–666, 2020
2020
Earlier work this paper cites.
M. Kaneko and D. Bollegala, “Debiasing pre-trained contextualised embeddings,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , 2021, pp. 1256–1266
2021
Earlier work this paper cites.
T. Wang, C. Zhou, Q. Sun, and H. Zhang, “Causal attention for unbiased visual recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 3091–3100
2021
Earlier work this paper cites.
X. Yang, H. Zhang, G. Qi, and J. Cai, “Causal attention for vision-language tasks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 9847–9857
2021
Earlier work this paper cites.
A. Webson and E. Pavlick, “Do prompt-based models really understand the meaning of their prompts?” arXiv preprint arXiv:2109.01247 , 2021
Original
2021
Earlier work this paper cites.
L. Yao, Z. Chu, S. Li, Y. Li, J. Gao, and A. Zhang, “A survey on causal inference,” ACM Transactions on Knowledge Discovery from Data (TKDD) , vol. 15, no. 5, pp. 1–46, 2021
2021
Earlier work this paper cites.
Y. Lu, H. Lu, G. Fu, and Q. Liu, “Kelm: knowledge enhanced pre-trained language representations with message passing on hierarchical relational graphs,” arXiv preprint arXiv:2109.04223 , 2021
Original
2021
Earlier work this paper cites.
Y. Chen, L. Xu, C. Gulcehre, T. L. Paine, A. Gretton, N. de Freitas, and A. Doucet, “On instrumental variable regression for deep offline policy evaluation,” arXiv preprint arXiv:2105.10148 , 2021
Original
2021
Earlier work this paper cites.
L. Liao, Z. Fu, Z. Yang, Y. Wang, M. Kolar, and Z. Wang, “Instrumental variable value iteration for causal offline reinforcement learning,” stat , vol. 1050, p. 13, 2021
2021
Earlier work this paper cites.
Y. Guo, Y. Yang, and A. Abbasi, “Auto-debias: Debiasing masked language models with automated biased prompts,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 1012–1023
2022
Earlier work this paper cites.
J. He, M. Xia, C. Fellbaum, and D. Chen, “Mabel: Attenuating gender bias using textual entailment data,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. 9681–9702
2022
Earlier work this paper cites.
J. Zhang, H. Zhang, W. Su, and D. Roth, “Rock: Causal inference principles for reasoning about commonsense causality,” in International Conference on Machine Learning . PMLR, 2022, pp. 26 750–26 771
2022
Earlier work this paper cites.
J. Frohberg and F. Binder, “Crass: A novel data set and benchmark to test counterfactual reasoning of large language models,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference , 2022, pp. 2126–2140
2022
Earlier work this paper cites.
T. Lin, Y. Wang, X. Liu, and X. Qiu, “A survey of transformers,” AI open , vol. 3, pp. 111–132, 2022
2022
Earlier work this paper cites.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9
2022
Earlier work this paper cites.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al. , “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862 , 2022
Original
2022
Earlier work this paper cites.
A. Wu, J. Yuan, K. Kuang, B. Li, R. Wu, Q. Zhu, Y. Zhuang, and F. Wu, “Learning decomposed representations for treatment effect estimation,” IEEE Transactions on Knowledge and Data Engineering , vol. 35, no. 5, pp. 4989–5001, 2022
2022
Earlier work this paper cites.
N. Meade, E. Poole-Dayan, and S. Reddy, “An empirical survey of the effectiveness of debiasing techniques for pre-trained language models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 1878–1898
2022
Earlier work this paper cites.

Similar

T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé et al. , “Bloom: A 176b-parameter open-access multilingual language model,” 2023
2023
Cited alongside, same era.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023
Original
2023
Cited alongside, same era.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023
Original
2023
Cited alongside, same era.
W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223 , 2023
Original
2023
Cited alongside, same era.

Then

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=HPuSIXJaa9
2023
Later among the works it cites.
A. E. Orhan, “Recognition, recall, and retention of few-shot memories in large language models,” arXiv preprint arXiv:2303.17557 , 2023
Original
2023
Later among the works it cites.
Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al. , “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology , vol. 15, no. 3, pp. 1–45, 2024
2024
Closest in time.
S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” arXiv preprint arXiv:2402.06196 , 2024
Original
2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Causality for Large Language Models

Wu, Anpeng, Kuang, Kun, Zhu, Minqin et al.

Understand

Recent breakthroughs in artificial intelligence have driven a paradigm shift, where large language models (LLMs) with billions or trillions of parameters are trained on vast datasets, achieving unprecedented success across a series of language tasks.

However, despite these successes, LLMs still rely on probabilistic modeling, which often captures spurious correlations rooted in linguistic patterns and social stereotypes, rather than the true causal relationships between entities and events.
This limitation renders LLMs vulnerable to issues such as demographic biases, social stereotypes, and LLM hallucinations.
These challenges highlight the urgent need to integrate causality into LLMs, moving beyond correlation-driven paradigms to build more reliable and ethically aligned AI systems.