AnyRefill: A Unified, Data-Efficient Framework for Left-Prompt-Guided Vision Tasks

Understand

In this paper, we present a novel Left-Prompt-Guided (LPG) paradigm to address a diverse range of reference-based vision tasks.

• Inspired by the human creative process, we reformulate these tasks using a left-right stitching formulation to construct contextual input.
• Building upon this foundation, we propose AnyRefill, an extension of LeftRefill, that effectively adapts Text-to-Image (T2I) models to various vision tasks.
• AnyRefill leverages the inpainting priors of advanced T2I model based on the Diffusion Transformer (DiT) architecture, and incorporates flexible components to enhance its capabilities.