Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems

Understand

Reinforcement Learning (RL) algorithms sample multiple n>1 solution attempts for each problem and reward them independently.

• This optimizes for pass@1 performance and prioritizes the strength of isolated samples at the expense of the diversity and collective utility of sets of samples.
• This under-utilizes the sampling capacity, limiting exploration and eventual improvement on harder examples.
• As a fix, we propose Pass-at-k Policy Optimization (PKPO), a transformation on the final rewards which leads to direct optimization of pass@k performance, thus optimizing for sets of samples that maximize reward when considered jointly.