BRAT: Bonus oRthogonAl Token for Architecture Agnostic Textual Inversion

Understand

Textual Inversion remains a popular method for personalizing diffusion models, in order to teach models new subjects and styles.

• We note that textual inversion has been underexplored using alternatives to the UNet, and experiment with textual inversion with a vision transformer.
• We also seek to optimize textual inversion using a strategy that does not require explicit use of the UNet and its idiosyncratic layers, so we add bonus tokens and enforce orthogonality.
• We find the use of the bonus token improves adherence to the source images and the use of the vision transformer improves adherence to the prompt.