Facial Sketch Synthesis using Attention guided CycleGAN with Surface Normal Loss

Abstract

Sketches play an important role in encoding representative facial features through simple strokes. The state-of-the-art approaches for sketch synthesis using generative and unsupervised learning methods have been able to generate perceptual facial sketches. However, such methods are prone to render low-quality facial sketches due to the effect of the geometry of faces and the portrait background. In this paper, to tackle this problem, we formulate and incorporate new loss functions based on facial surface normals and edge maps into a generative adversarial training framework based on CycleGAN. Facial surface normals are estimated by a convolutional neural network based on Dense Inductive Biases for Surface Normal Estimation (DSINE) and encoded into colored frames indicating normal directions of pictures, better capturing the geometric information of the face. Edge maps are estimated by Holistically-Nested Edge Detection (HED), better capturing the lines and contours of face pictures. Furthermore, we extend the attentionguided generator that separates foreground and background during training, thereby reducing the impact of background elements on the face sketch. As such, our approach aims to learn generator architectures to translate pictures to sketches and vice versa with utmost consistency and geometric accuracy. Our computational experiments using the FS2K (containing annotated facial 2,104 images) on an Nvidia A6000 show the improved performance in terms of Learned Perceptual Image Patch Similarity (LPIPS, 0.412), Structural Similarity (SSIM, 0.456), Multiscale Structural Similarity (MS-SSIM, 0.515), Feature Similarity (FSIM, 0.609), Structure Co-Occurrence Texture (SCOOT, 0.404). Our results have the potential to study improved cycle-consistent architectures to generate face sketches with highquality and rich details.