01Text-Guided Transfer
The target is given as text. The agents read the metaphor of the reference, find a carrier that fits the new subject, and re-materialize the idea. The same reference can be re-instantiated for several different targets. Click any card to view it large.
02Image-Guided Transfer
The target is given as an image — typically a product shot. The reference contributes only its abstract logic; the target contributes its identity. The result is a new metaphor for the target, not a blend of the two pictures — and the same product can borrow a different metaphor from each reference.
03How it works
We ground the task in Conceptual Blending Theory, write a visual metaphor down as a 7-tuple Schema Grammar, and let four agents extract, transfer, render and criticize it in a closed loop.
Theoretical basis
Conceptual Blending Theory
Fauconnier & Turner's account of how the mind creates new meaning: a metaphor is not a linear mapping from one thing to another, but the integration of four mental spaces.
- Input Space 1 & 2Two specific contexts — for a visual metaphor, the subject and the carrier. Their elements are linked as counterparts through cross-space mapping.
- Generic SpaceThe abstract structure both inputs share — roles, frames, relations — captured by structure mapping. It is what makes the two inputs comparable at all.
- Blended SpaceReceives a selective projection from both inputs and, through composition, completion and elaboration, develops emergent structure: meaning that exists in neither input alone.
Our Schema Grammar operationalizes these spaces one-to-one: S, C instantiate the input spaces; G is the generic space; V, I realize the blend — the violation is the selective projection, the emergent meaning is its emergent structure.
G is preserved; C, V, I are re-instantiated for the new subject. Hover each element.
Perception Agent
Distills the reference image into SG_ref, separating surface entities from the abstract relational logic that makes the metaphor work.
Transfer Agent
Keeps the Generic Space G invariant while profiling the new subject, discovering an apt carrier and redesigning the violation — yielding SG_tgt.
Generation Agent
Compiles SG_tgt into a stylistically rigorous text-to-image master prompt and synthesizes the image.
Diagnostic Agent
Critiques subject salience, violation realization, relational coherence and meaning alignment — then triggers hierarchical backtracking.
Closed-loop hierarchical backtracking (up to τ = 5 iterations)
04Numbers
VLM-as-judge scores on 126 curated visual metaphors (Gemini-3-pro judge, 10-point scale — similar margins under GPT-5.2 and Claude-4.5)
Metaphor Consistency
Analogy Appropriateness
Conceptual Integration
Preferred by human raters in > 60% of pairwise comparisons against every baseline (100 participants).
Qualitative comparison with BAGEL, Midjourney, GPT-Image and Banana-Pro
05Abstract
A visual metaphor constitutes a high-order form of human creativity, employing cross-domain semantic fusion to transform abstract concepts into impactful visual rhetoric. Despite the remarkable progress of generative AI, existing models remain largely confined to pixel-level instruction alignment and surface-level appearance preservation, failing to capture the underlying abstract logic necessary for genuine metaphorical generation. To bridge this gap, we introduce the task of Visual Metaphor Transfer (VMT), which challenges models to autonomously decouple the “creative essence” from a reference image and re-materialize that abstract logic onto a user-specified target subject. We propose a cognitive-inspired, multi-agent framework that operationalizes Conceptual Blending Theory (CBT) through a novel Schema Grammar (𝒮𝒢). This structured representation decouples relational invariants from specific visual entities, providing a rigorous foundation for cross-domain logic re-instantiation. Our pipeline executes VMT through a collaborative system of specialized agents: a perception agent that distills the reference into a schema, a transfer agent that maintains generic space invariance to discover apt carriers, a generation agent for high-fidelity synthesis, and a hierarchical diagnostic agent that mimics a professional critic, performing closed-loop backtracking to identify and rectify errors across abstract logic, component selection, and prompt encoding. Extensive experiments and human evaluations demonstrate that our method significantly outperforms state-of-the-art baselines in metaphor consistency, analogy appropriateness, and visual creativity.
06Try it as an Agent Skill
The full method ships as a self-contained agent skill — the four paper prompts orchestrated by a closed-loop workflow that Codex, Claude Code or Cursor can execute directly. No training, no deployment.
- Clone the repo and copy
visual-metaphor-transfer/into your agent's skills directory (~/.codex/skills/,~/.claude/skills/, or~/.cursor/skills/). - Start a new conversation and upload a reference image (and optionally a target image).
- Ask for a transfer — the agent runs Perception → Transfer → Generation → Diagnosis and iterates until the metaphor lands.
❯
◆ Phase 1 · Perception — extracting SG_ref … done
◆ Phase 2 · Transfer — G invariant, new carrier found … done
◆ Phase 3 · Generation — master prompt → image … done
◆ Phase 4 · Diagnosis — all four constraints … PASS ✓
07BibTeX
@inproceedings{xu2026beyondpixels,
title = {Beyond Pixels: Visual Metaphor Transfer via Schema-Driven Agentic Reasoning},
author = {Xu, Yu and Zhang, Yuxin and Gao, Lin and Deussen, Oliver and Lee, Tong-Yee and Tang, Fan},
booktitle = {SIGGRAPH Asia 2026 Conference Papers},
year = {2026}
}