Built on

C. J. Clopper and E. S. Pearson, “The use of confidence or fiducial limits illustrated in the case of the binomial,” Biometrika , no. 4, 1934
1934
Earlier work this paper cites.
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” ArXiv preprint , vol. abs/1609.08144, 2016
Original
2016
Earlier work this paper cites.
A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. J. Crandall, and D. Batra, “Diverse beam search: Decoding diverse solutions from neural sequence,” ArXiv preprint , vol. abs/1610.02424, 2016
Original
2016
Earlier work this paper cites.
C. Hokamp and Q. Liu, “Lexically constrained decoding for sequence generation using grid beam search,” in Proc. of ACL , 2017
2017
Earlier work this paper cites.
W. Ling, D. Yogatama, C. Dyer, and P. Blunsom, “Program induction by rationale generation: Learning to solve and explain algebraic word problems,” in Proc. of ACL , 2017
2017
Earlier work this paper cites.
M. Post and D. Vilar, “Fast lexically constrained decoding with dynamic beam allocation for neural machine translation,” in Proc. of NAACL-HLT , 2018
2018
Earlier work this paper cites.
J. E. Hu, H. Khayrallah, R. Culkin, P. Xia, T. Chen, M. Post, and B. Van Durme, “Improved lexically constrained decoding for translation and monolingual rewriting,” in Proc. of NAACL-HLT , 2019
2019
Earlier work this paper cites.
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in Proc. of ICLR , 2020
2020
Earlier work this paper cites.
C. Meister, R. Cotterell, and T. Vieira, “If beam search is the answer, what was the question?” in Proc. of EMNLP , 2020
2020
Earlier work this paper cites.
Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot performance of language models,” in Proc. of ICML , 2021
2021
Earlier work this paper cites.
M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant, “Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,” Transactions of the Association for Computational Linguistics , 2021
2021
Earlier work this paper cites.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al. , “Training verifiers to solve math word problems,” ArXiv preprint , vol. abs/2110.14168, 2021
Original
2021
Earlier work this paper cites.

Similar

L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts , 2021
2021
Cited alongside, same era.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models are Zero-Shot Reasoners,” ArXiv preprint , vol. abs/2205.11916, 2022
Original
2022
Cited alongside, same era.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Proc. of NeurIPS , 2022
2022
Cited alongside, same era.
G. Poesia, A. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, and S. Gulwani, “Synchromesh: Reliable code generation from pre-trained language models,” in Proc. of ICLR , 2022
2022
Cited alongside, same era.

Then

B. Ozturkler, N. Malkin, Z. Wang, and N. Jojic, “Thinksum: Probabilistic reasoning over sets using large language models,” ArXiv preprint , vol. abs/2210.01293, 2022
Original
2022
Later among the works it cites.
Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” ArXiv preprint , vol. abs/2211.01910, 2022
Original
2022
Later among the works it cites.
T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai, “Promptchainer: Chaining large language model prompts through visual programming,” in CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, Extended Abstracts , 2022
2022
Later among the works it cites.
S. Bach, V. Sanh, Z. X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Fevry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-david, C. Xu, G. Chhablani, H. Wang, J. Fries, M. Al-shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, D. Radev, M. T.-j. Jiang, and A. Rush, “PromptSource: An integrated development environment and repository for natural language prompts,” in Proc. of ACL , 2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Prompt Sketching for Large Language Models

Beurer-Kellner, Luca, Müller, Mark Niklas, Fischer, Marc et al.

Understand

Many recent prompting strategies for large language models (LLMs) query the model multiple times sequentially -- first to produce intermediate results and then the final answer.

However, using these methods, both decoder and model are unaware of potential follow-up prompts, leading to disconnected and undesirably wordy intermediate responses.
In this work, we address this issue by proposing prompt sketching, a new prompting paradigm in which an LLM does not only respond by completing a prompt, but by predicting values for multiple variables in a template.
This way, sketching grants users more control over the generation process, e.g., by providing a reasoning framework via intermediate instructions, leading to better overall results.

Built on

C. J. Clopper and E. S. Pearson, “The use of confidence or fiducial limits illustrated in the case of the binomial,” Biometrika , no. 4, 1934
1934
Earlier work this paper cites.
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” ArXiv preprint , vol. abs/1609.08144, 2016
Original
2016
Earlier work this paper cites.
A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. J. Crandall, and D. Batra, “Diverse beam search: Decoding diverse solutions from neural sequence,” ArXiv preprint , vol. abs/1610.02424, 2016
Original
2016
Earlier work this paper cites.
C. Hokamp and Q. Liu, “Lexically constrained decoding for sequence generation using grid beam search,” in Proc. of ACL , 2017
2017
Earlier work this paper cites.
W. Ling, D. Yogatama, C. Dyer, and P. Blunsom, “Program induction by rationale generation: Learning to solve and explain algebraic word problems,” in Proc. of ACL , 2017
2017
Earlier work this paper cites.
M. Post and D. Vilar, “Fast lexically constrained decoding with dynamic beam allocation for neural machine translation,” in Proc. of NAACL-HLT , 2018
2018
Earlier work this paper cites.
J. E. Hu, H. Khayrallah, R. Culkin, P. Xia, T. Chen, M. Post, and B. Van Durme, “Improved lexically constrained decoding for translation and monolingual rewriting,” in Proc. of NAACL-HLT , 2019
2019
Earlier work this paper cites.
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in Proc. of ICLR , 2020
2020
Earlier work this paper cites.
C. Meister, R. Cotterell, and T. Vieira, “If beam search is the answer, what was the question?” in Proc. of EMNLP , 2020
2020
Earlier work this paper cites.
Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot performance of language models,” in Proc. of ICML , 2021
2021
Earlier work this paper cites.
M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant, “Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies,” Transactions of the Association for Computational Linguistics , 2021
2021
Earlier work this paper cites.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al. , “Training verifiers to solve math word problems,” ArXiv preprint , vol. abs/2110.14168, 2021
Original
2021
Earlier work this paper cites.

Similar

L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts , 2021
2021
Cited alongside, same era.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models are Zero-Shot Reasoners,” ArXiv preprint , vol. abs/2205.11916, 2022
Original
2022
Cited alongside, same era.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Proc. of NeurIPS , 2022
2022
Cited alongside, same era.
G. Poesia, A. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, and S. Gulwani, “Synchromesh: Reliable code generation from pre-trained language models,” in Proc. of ICLR , 2022
2022
Cited alongside, same era.

Then

B. Ozturkler, N. Malkin, Z. Wang, and N. Jojic, “Thinksum: Probabilistic reasoning over sets using large language models,” ArXiv preprint , vol. abs/2210.01293, 2022
Original
2022
Later among the works it cites.
Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” ArXiv preprint , vol. abs/2211.01910, 2022
Original
2022
Later among the works it cites.
T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai, “Promptchainer: Chaining large language model prompts through visual programming,” in CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, Extended Abstracts , 2022
2022
Later among the works it cites.
S. Bach, V. Sanh, Z. X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Fevry, Z. Alyafeai, M. Dey, A. Santilli, Z. Sun, S. Ben-david, C. Xu, G. Chhablani, H. Wang, J. Fries, M. Al-shaibani, S. Sharma, U. Thakker, K. Almubarak, X. Tang, D. Radev, M. T.-j. Jiang, and A. Rush, “PromptSource: An integrated development environment and repository for natural language prompts,” in Proc. of ACL , 2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” ArXiv preprint , vol. abs/2210.03629, 2022

Original

2022

Cited alongside, same era.

D. Dohan, W. Xu, A. Lewkowycz, J. Austin, D. Bieber, R. G. Lopes, Y. Wu, H. Michalewski, R. A. Saurous, J. Sohl-dickstein, K. Murphy, and C. Sutton, “Language Model Cascades,” ArXiv preprint , vol. abs/2207.10342, 2022

Original

2022

Cited alongside, same era.

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Rahane, A. S. Iyer, A. Andreassen, A. Santilli, A. Stuhlmüller, A. M. Dai, A. La, A. K. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, A. Gottardi, A. Norelli, A. Venkatesh, A. Gholamidavoodi, A. Tabassum, A. Menezes, A. Kirubarajan, A. Mullokandov, A. Sabharwal, A. Herrick, A. Efrat, A. Erdem, A. Karakas, and et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities,” ArXiv preprint , vol. abs/2206.04615, 2022

Original

2022

Cited alongside, same era.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” ArXiv preprint , vol. abs/2201.11903, 2022

Original

2022

Cited alongside, same era.

J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” in Proc. of ICLR , 2022

2022

Cited alongside, same era.

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” ArXiv preprint , vol. abs/2210.03629, 2022

Original

2022

Cited alongside, same era.

X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” ArXiv preprint , vol. abs/2203.11171, 2022

Original

2022

Cited alongside, same era.

S. Lundberg and M. T. C. Ribeiro, “Guidance-ai/guidance: A guidance language for controlling large language models.”

Cited in the paper.

2022

Later among the works it cites.

C. Meister, T. Pimentel, G. Wiher, and R. Cotterell, “Locally typical sampling,” ArXiv preprint , vol. abs/2202.00666, 2022

Original

2022

Later among the works it cites.

J. Xu, C. Xiong, S. Savarese, and Y. Zhou, “Best-k search algorithm for neural text generation,” ArXiv preprint , vol. abs/2211.11924, 2022

Original

2022

Later among the works it cites.

J. Xu, S. Jonnalagadda, and G. Durrett, “Massive-scale decoding for text generation using lattices,” in Proc. of NAACL-HLT , 2022

2022

Later among the works it cites.

L. Beurer-Kellner, M. Fischer, and M. Vechev, “Prompting is programming: A query language for large language models,” Proceedings of the ACM on Programming Languages , no. PLDI, 2023

2023

Closest in time.

S. Arora, A. Narayan, M. F. Chen, L. J. Orr, N. Guha, K. Bhatia, I. Chami, and C. Ré, “Ask me anything: A simple strategy for prompting language models,” in Proc. of ICLR , 2023

2023

Closest in time.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” ArXiv preprint , vol. abs/2307.09288, 2023

Original

2023

Closest in time.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al. , “Palm-e: An embodied multimodal language model,” ArXiv preprint , vol. abs/2303.03378, 2023

Original

2023

Closest in time.