IP-Composer: Semantic Composition of Visual Concepts

IP-Composer enables compositional generation from a set of visual concepts. These are portrayed through a set of input images, along with a prompt describing the desired concept to be extracted from each.

Abstract

Content creators often draw inspiration from multiple visual sources, combining distinct elements to craft new compositions. Modern computational approaches now aim to emulate this fundamental creative process. Although recent diffusion models excel at text-guided compositional synthesis, text as a medium often lacks precise control over visual details. Image-based composition approaches can capture more nuanced features, but existing methods are typically limited in the range of concepts they can capture, and require expensive training procedures or specialized data. We present IP-Composer, a novel training-free approach for compositional image generation that leverages multiple image references simultaneously, while using natural language to describe the concept to be extracted from each image. Our method builds on IP-Adapter, which synthesizes novel images conditioned on an input image's CLIP embedding. We extend this approach to multiple visual inputs by crafting composite embeddings, stitched from the projections of multiple input images onto concept-specific CLIP-subspaces identified through text. Through comprehensive evaluation, we show that our approach enables more precise control over a larger range of visual concept compositions.

Image Composition Examples
with IP-Composer

How Does it Work?

  • 🔗 Concept Extraction: We use an LLM to generate texts describing possible variations of a concept we want to extract from the concept-image. We encode the responses using CLIP and find the embedding subspace that they span.
  • 🔗 Embedding Composition: We generate a composite CLIP-embedding by replacing the projection of the reference image on this embedding subspace with the matching projection of the concept image.
  • 🔗 Image Generation: The composite embedding can be used by an off-the-shelf IP-Adapter to generate images combining the reference and the visual concept. The same approach can be applied with additional concept images.

Some More Examples

Text Prompt Integration

In addition to image prompts, our method is also able to integrate text prompts, leveraging IP-Adapter’s built-in support for text conditioning.