2023

A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT

White, Jules, Fu, Quchen, Hays, Sam et al.

Understand

Prompt engineering is an increasingly important skill set needed to converse effectively with large language models (LLMs), such as ChatGPT.

Prompts are instructions given to an LLM to enforce rules, automate processes, and ensure specific qualities (and quantities) of generated output.
Prompts are also a form of programming that can customize the outputs and interactions with an LLM.
This paper describes a catalog of prompt engineering techniques presented in pattern form that have been applied to solve common problems when conversing with LLMs.

Built on

E. Gamma, R. Johnson, R. Helm, R. E. Johnson, and J. Vlissides, Design patterns: elements of reusable object-oriented software . Pearson Deutschland GmbH, 1995
1995
Earlier work this paper cites.
J. Ellson, E. R. Gansner, E. Koutsofios, S. C. North, and G. Woodhull, “Graphviz and dynagraph—static and dynamic graph drawing tools,” Graph drawing software , pp. 127–148, 2004
2004
Earlier work this paper cites.
T. Shin, Y. Razeghi, R. L. L. IV, E. Wallace, and S. Singh, “Autoprompt: Eliciting knowledge from language models with automatically generated prompts,” CoRR , vol. abs/2010.15980, 2020. [Online]. Available: https://arxiv.org/abs/2010.15980
Original
2010
Earlier work this paper cites.
D. C. Schmidt, M. Stal, H. Rohnert, and F. Buschmann, Pattern-oriented software architecture, patterns for concurrent and networked objects . John Wiley & Sons, 2013
2013
Earlier work this paper cites.
J. Krochmalski, IntelliJ IDEA Essentials . Packt Publishing Ltd, 2014
2014
Earlier work this paper cites.
P. Zhang, J. White, D. C. Schmidt, and G. Lenz, “Applying software patterns to address interoperability in blockchain-based healthcare apps,” CoRR , vol. abs/1706.03700, 2017. [Online]. Available: http://arxiv.org/abs/1706.03700
Original
2017
Earlier work this paper cites.
X. Xu, C. Pautasso, L. Zhu, Q. Lu, and I. Weber, “A pattern collection for blockchain-based applications,” in Proceedings of the 23rd European Conference on Pattern Languages of Programs , 2018, pp. 1–20
2018
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
Earlier work this paper cites.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258 , 2021
Original
2021
Earlier work this paper cites.
L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” CoRR , vol. abs/2102.07350, 2021. [Online]. Available: https://arxiv.org/abs/2102.07350
Original
2021
Earlier work this paper cites.
A. Gilson, C. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, and D. Chartash, “How well does chatgpt do when taking the medical licensing exams?” medRxiv , pp. 2022–12, 2022
2022
Earlier work this paper cites.
A. Carleton, M. H. Klein, J. E. Robert, E. Harper, R. K. Cunningham, D. de Niz, J. T. Foreman, J. B. Goodenough, J. D. Herbsleb, I. Ozkaya, and D. C. Schmidt, “Architecting the future of software engineering,” Computer , vol. 55, no. 9, pp. 89–93, 2022
2022
Earlier work this paper cites.

Similar

O. Asare, M. Nagappan, and N. Asokan, “Is github’s copilot as bad as humans at introducing vulnerabilities in code?” arXiv preprint arXiv:2204.04741 , 2022
Original
2022
Cited alongside, same era.
H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 754–768
2022
Cited alongside, same era.
D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi, “Least-to-most prompting enables complex reasoning in large language models,” arXiv preprint arXiv:2205.10625 , 2022
Original
2022
Cited alongside, same era.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” CoRR , vol. abs/2201.11903, 2022. [Online]. Available: https://arxiv.org/abs/2201.11903
Original
2022
Cited alongside, same era.
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent abilities of large language models,” 2022. [Online]. Available: https://arxiv.org/abs/2206.07682
Original
2022
Cited alongside, same era.
Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” 2022. [Online]. Available: https://arxiv.org/abs/2211.01910
Original
2022
Cited alongside, same era.
D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. Le, and E. Chi, “Least-to-most prompting enables complex reasoning in large language models,” 2022. [Online]. Available: https://arxiv.org/abs/2205.10625
Original
2022
Cited alongside, same era.
J. Jung, L. Qin, S. Welleck, F. Brahman, C. Bhagavatula, R. L. Bras, and Y. Choi, “Maieutic prompting: Logically consistent reasoning with recursive explanations,” 2022. [Online]. Available: https://arxiv.org/abs/2205.11822
Original
2022
Cited alongside, same era.
V. Liu and L. B. Chilton, “Design guidelines for prompt engineering text-to-image generative models,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , 2022, pp. 1–23
2022
Cited alongside, same era.
X. Han, W. Zhao, N. Ding, Z. Liu, and M. Sun, “Ptr: Prompt tuning with rules for text classification,” AI Open , vol. 3, pp. 182–192, 2022
2022
Cited alongside, same era.
Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al. , “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023 , 2023
Original
2023
Cited alongside, same era.
“Github copilot · your ai pair programmer.” [Online]. Available: https://github.com/features/copilot
Cited in the paper.

Then

P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys , vol. 55, no. 9, pp. 1–35, 2023
2023
Closest in time.
OpenAI, “ChatGPT: Large-Scale Generative Language Models for Automated Content Creation,” https://openai.com/blog/chatgpt/, 2023, [Online; accessed 19-Feb-2023]
2023
Closest in time.
——, “DALL·E 2: Creating Images from Text,” https://openai.com/dall-e-2/, 2023, [Online; accessed 19-Feb-2023]
2023
Closest in time.
S. Owen, “Building a virtual machine inside a javascript library,” https://www.engraved.blog/building-a-virtual-machine-inside/, 2022, accessed: 2023-02-20
2023
Closest in time.
E. A. van Dis, J. Bollen, W. Zuidema, R. van Rooij, and C. L. Bockting, “Chatgpt: five priorities for research,” Nature , vol. 614, no. 7947, pp. 224–226, 2023
2023
Closest in time.
S. Arora, A. Narayan, M. F. Chen, L. Orr, N. Guha, K. Bhatia, I. Chami, and C. Re, “Ask me anything: A simple strategy for prompting language models,” in International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=bhUPJnS2g0X
2023
Closest in time.
P. Maddigan and T. Susnjak, “Chat2vis: Generating data visualisations via natural language using chatgpt, codex and gpt-3 large language models,” arXiv preprint arXiv:2302.02094 , 2023
Original
2023
Closest in time.
S. Wang, H. Scells, B. Koopman, and G. Zuccon, “Can chatgpt write a good boolean query for systematic review literature search?” arXiv preprint arXiv:2302.03495 , 2023
Original
2023
Closest in time.
C. S. Xia and L. Zhang, “Conversational automated program repair,” arXiv preprint arXiv:2301.13246 , 2023
Original
2023
Closest in time.
J. H. Choi, K. E. Hickman, A. Monahan, and D. Schwarcz, “Chatgpt goes to law school,” Available at SSRN , 2023
2023
Closest in time.
S. Frieder, L. Pinchetti, R.-R. Griffiths, T. Salvatori, T. Lukasiewicz, P. C. Petersen, A. Chevalier, and J. Berner, “Mathematical capabilities of chatgpt,” arXiv preprint arXiv:2301.13867 , 2023
Original
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…