DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN
Wanchao Su1
Hui Ye1
Shu-Yu Chen2
Lin Gao2
Hongbo Fu1,*
1 School of Creative Media, City University of Hong Kong
2 Institute of Computing Technology, Chinese Academy of Sciences
* Corresponding Author
[Paper]
[Code(Coming soon)]
[Supplemental Material]
Our DrawingInStyles system helps users with limited drawing skills to produce high-quality portrait images with diversified geometry and appearance (best viewed with zoom in) from scratch. Our data-driven suggestive interface assists users in interactive refinement of sketches and semantic maps (Bottom), which provide precise conditions for subsequent image synthesis. Our method also supports high-quality portrait image editing by editing the sketch and/or semantic map. The minor changes to the input are highlighted in red boxes and zoom-in box.

Abstract

The research topic of sketch-to-portrait generation has witnessed a boost of progress with deep learning techniques. The recently proposed StyleGAN architectures achieve state-of-the-art generation ability but the original StyleGAN is not friendly for sketch-based creation due to its unconditional generation nature. To address this issue, we propose a direct conditioning strategy to better preserve the spatial information under the StyleGAN framework. Specifically, we introduce Spatially Conditioned StyleGAN (SC-StyleGAN for short), which explicitly injects spatial constraints to the original StyleGAN generation process. We explore two input modalities, sketches and semantic maps, which together allow users to express desired generation results more precisely and easily. Based on SC-StyleGAN, we present DrawingInStyles, a novel drawing interface for non-professional users to easily produce high-quality, photo-realistic face images with precise control, either from scratch or editing existing ones. Qualitative and quantitative evaluations show the superior generation ability of our method to existing and alternative solutions. The usability and expressiveness of our system are confirmed by a user study.

Video


Bibtex

If you find our work useful for your research, please cite our work as:
@article{su2022drawinginstyles,
    author = {Su, Wanchao and Ye, Hui and Chen, Shu-Yu and Gao, Lin and Fu, Hongbo},
    title = {DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN},
    journal = {IEEE Transactions on Visualization and Computer Graphics},
    year = {2023},
    volume={29},
    number={10},
    pages={4074-4088},
    doi={10.1109/TVCG.2022.3178734},
    publisher={IEEE}
}