Large Language Models as Optimizers

Understand

Optimization is ubiquitous.

• While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications.
• In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task is described in natural language.
• In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step.