DM-Align: Leveraging the Power of Natural Language Instructions to Make Changes to Images

Understand

Text-based semantic image editing assumes the manipulation of an image using a natural language instruction.

• Although recent works are capable of generating creative and qualitative images, the problem is still mostly approached as a black box sensitive to generating unexpected outputs.
• Therefore, we propose a novel model to enhance the text-based control of an image editor by explicitly reasoning about which parts of the image to alter or preserve.
• It relies on word alignments between a description of the original source image and the instruction that reflects the needed updates, and the input image.