PonderLM: Pretraining Language Models to Ponder in Continuous Space

Understand

Humans ponder before articulating complex sentence elements, enabling deeper cognitive processing through focused effort.

• In this work, we introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step.
• During pondering, instead of generating an actual token sampled from the prediction distribution, the model ponders by yielding a weighted sum of all token embeddings according to the predicted token distribution.
• The generated embedding is then fed back as input for another forward pass.