Collaborative Control for Geometry-Conditioned PBR Image Generation

Understand

Graphics pipelines require physically-based rendering (PBR) materials, yet current 3D content generation approaches are built on RGB models.

• We propose to model the PBR image distribution directly, avoiding photometric inaccuracies in RGB generation and the inherent ambiguity in extracting PBR from RGB.
• As existing paradigms for cross-modal fine-tuning are not suited for PBR generation due to both a lack of data and the high dimensionality of the output modalities, we propose to train a new PBR model that is tightly linked to a frozen RGB model using a novel cross-network communication paradigm.
• As the base RGB model is fully frozen, the proposed method retains its general performance and remains compatible with e.g.