Abstract
Personalized generation paradigms empower designers to customize visual intellectual property with the help of textual descriptions by adapting pre-trained text-to-image models on a few images. Recent studies focus on simultaneously customizing content and detailed visual style in images but often struggle with entangling the two. In this study, we reconsider the customization of content and style concepts from the perspective of parameter space construction. Unlike existing methods that utilize a shared parameter space for content and style learning, we propose a novel framework that separates the parameter space to facilitate individual learning of content and style by introducing "partly learnable projection" (PLP) matrices to separate the original adapters into divided sub-parameter spaces. A "break-for-make" customization learning pipeline based on PLP is proposed: we first break the original adapters into "up projection" and "down projection" for content and style concept under orthogonal prior and then make the entity parameter space by reconstructing the content and style PLP matrices by using Riemannian preconditioning to adaptively balance content and style learning. Experiments on various styles, including textures, materials, and artistic style, show that our method outperforms state-of-the-art single/multiple concept learning pipelines regarding content-style-prompt alignment.
Method
Frameworks of existing approaches and ours for customized content-style image generation. Joint training LoRA will mix the parameter space of content and style, leading to the entanglement of both. ZipLoRA effectively merges independently trained content and style LoRAs. However, the conflicting parameters between the content LoRA and style LoRA can lead to unfaithful reproduction of content and/or style after fusion. Orthogonal LoRA focuses on multi-subject customization by learning orthogonal LoRAs for each subject. These adapters are composed in a ‘continual learning' manner, in which different concepts do not influence each other. However, this approach leads to a failure in content-style fusion. Our method trains content and style in separated parameter subspaces of LoRA, resulting in a disentangled and faithful fusion of content and style.
Experiments
We present qualitative and quantitative comparisons between our method and baseline approaches.
Qualitative evaluation and comparison of DB+LoRA, TI, ProSpect, CD, ZipLoRA, and our method in diverse styles. We present the results of customized generation of the same content and different styles. Results indicate that our method generates harmonious fusion images of the content and style while preserving the disentanglement of content and style, as well as maintaining high-level fidelity.
Qualitative evaluation and comparison of DB+LoRA, TI, ProSpect, CD, ZipLoRA, and our method in diverse contents. The results indicate that our method generates harmonious content-style fusion images with diverse contents while preserving the disentanglement of content and style as well as maintaining high-level fidelity.
Results of generating diverse customized content-style images. This indicates that our method exhibits excellent editing capabilities as well as generalization capabilities to both content and style.
Applications
Content-style customization of various textures, including knit, burlap, denim, and fabric texture.
Content-style customization of portraits.
Content-style customization of hypnotic line art and portrait map art.
BibTeX
@article{xu2025b4m,
title={B4M: Breaking Low-Rank Adapter for Making Content-Style Customization},
author={Xu, Yu and Tang, Fan and Cao, Juan and Zhang, Yuxin and Deussen, Oliver and Dong, Weiming and Li, Jintao and Lee, Tong-Yee},
journal={ACM Transactions on Graphics},
volume={44},
number={2},
pages={1--17},
year={2025},
publisher={ACM New York, NY},
doi={10.1145/3728461}
}