{"title": "Learning a High Fidelity Pose Invariant Model for High-resolution Face Frontalization", "book": "Advances in Neural Information Processing Systems", "page_first": 2867, "page_last": 2877, "abstract": "Face frontalization refers to the process of synthesizing the frontal view of a face from a given profile.  Due to self-occlusion and appearance distortion in the wild, it is extremely challenging to recover faithful results and preserve texture details in a high-resolution. This paper proposes a High Fidelity Pose Invariant Model (HF-PIM) to produce photographic and identity-preserving results. HF-PIM frontalizes the profiles through a novel texture warping procedure and leverages a dense correspondence field to bind the 2D and 3D surface spaces. We decompose the prerequisite of warping into dense correspondence field estimation and facial texture map recovering, which are both well addressed by deep networks. Different from those reconstruction methods relying on 3D data, we also propose Adversarial Residual Dictionary Learning (ARDL) to supervise facial texture map recovering with only monocular images. Exhaustive experiments on both controlled and uncontrolled environments demonstrate that the proposed method not only boosts the performance of pose-invariant face recognition but also dramatically improves high-resolution frontalization appearances.", "full_text": "Learning a High Fidelity Pose Invariant Model for\n\nHigh-resolution Face Frontalization\n\nJie Cao, Yibo Hu, Hongwen Zhang, Ran He\u2217, Zhenan Sun\n\nNational Laboratory of Pattern Recognition, CASIA\n\nCenter for Research on Intelligent Perception and Computing, CASIA\n\nCenter for Excellence in Brain Science and Intelligence Technology, CASIA\n\nUniversity of Chinese Academy of Sciences, Beijing, 100049, China\n\n{jie.cao,yibo.hu,hongwen.zhang}@cripac.ia.ac.cn {rhe,znsun}@nlpr.ia.ac.cn\n\nAbstract\n\nFace frontalization refers to the process of synthesizing the frontal view of a\nface from a given pro\ufb01le. Due to self-occlusion and appearance distortion in the\nwild, it is extremely challenging to recover faithful results and preserve texture\ndetails in a high-resolution. This paper proposes a High Fidelity Pose Invariant\nModel (HF-PIM) to produce photographic and identity-preserving results. HF-PIM\nfrontalizes the pro\ufb01les through a novel texture warping procedure and leverages a\ndense correspondence \ufb01eld to bind the 2D and 3D surface spaces. We decompose\nthe prerequisite of warping into dense correspondence \ufb01eld estimation and facial\ntexture map recovering, which are both well addressed by deep networks. Different\nfrom those reconstruction methods relying on 3D data, we also propose Adversarial\nResidual Dictionary Learning (ARDL) to supervise facial texture map recovering\nwith only monocular images. Exhaustive experiments on both controlled and\nuncontrolled environments demonstrate that the proposed method not only boosts\nthe performance of pose-invariant face recognition but also dramatically improves\nhigh-resolution frontalization appearances.\n\n1\n\nIntroduction\n\nFace frontalization refers to predicting the frontal view image from a given pro\ufb01le. It is an effective\npreprocessing method for pose-invariant face recognition. Frontalized pro\ufb01le faces can be directly\nused by general face recognition methods without retraining the recognition models. Recent studies\nhave shown that frontalization is a promising approach to address long-standing problems caused\nby pose variation in face recognition system. Additionally, generating photographic frontal faces\nare bene\ufb01cial for a series of face-related tasks, including face reconstruction, face attribute analysis,\nfacial animation, etc.\nDue to the appealing prospect in theories and applications, research interest has been lasting for years.\nIn the early stage, most traditional face frontalization methods [8, 14, 15, 9, 40] are 3D-based. These\nmethods mainly leverage theories in monocular face reconstruction to recover 3D faces, and then\nrender frontal view images. The well-known 3D Morphable Model (3DMM) [2] has been widely\nemployed to express facial shape and appearance information. Recently, great breakthroughs have\nbeen made by the methods based on generative adversarial networks (GAN) [10]. Those methods\nfrontalize faces from the perspective of 2D image-to-image translation and build deep networks\nwith novel architectures. The visual realism has been improved signi\ufb01cantly, for instance, in Multi-\nPIE [11], some synthesized results [20, 37] from small pose pro\ufb01les are so photographic that it\nis dif\ufb01cult for human observers to distinguish them from the real ones. Furthermore, frontalized\n\n\u2217Ran He is the corresponding author.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fresults have been proved to be effective to tackle the pose discrepancy in face recognition. Through\nthe \u201crecognition via generation\u201d framework, i.e., rotating the pro\ufb01les to the frontal views, which\ncan be directly used by general face recognition methods, frontalization methods [36, 37] achieve\nstate-of-the-art pose-invariant face recognition performance on multiple datasets, including Multi-PIE\nand IJB-A [23].\nEven though much progress has been made, there are still some ongoing issues for in-the-wild face\nfrontalization. For traditional 3D-based approaches, due to the shortage of 3D data and the limited\nrepresentation power of backbone 3D model, their performances are commonly less competitive\ncompared with GAN-based methods albeit some improvements [5, 32] have been made. However,\nGAN-based methods heavily rely on minimizing pixel-wise losses to deal with the noisy data for in\nthe wild settings. As discussed in many other image restoration tasks [19, 21], the consequence is that\nthe outputs lack variations and tend to keep close to the statistical meaning of the training data. The\nresults will be over-smoothed with little high-level texture information. Hence, current frontalization\nresults are less appealing in a high-resolution and the output size is often no larger than 128 \u00d7 128.\nTo address the above issues, this paper proposes a High Fidelity Pose Invariant Model (HF-PIM)\nthat combines the advantages of 3D and GAN based methods. In HF-PIM, we frontalize the pro\ufb01les\nvia a novel texture warping procedure. Inspired by recent progress in 3D face analysis [12, 13], we\nintroduce a dense correspondence \ufb01eld to bind the 2D and 3D surface spaces. Thus, the prerequisite\nof our warping procedure is decomposed into two well-constrained problems: dense correspondence\n\ufb01eld estimation and facial texture map recovering. We build a deep network to address the two\nproblems and bene\ufb01t from its greater representation power than traditional 3D-based methods.\nFurthermore, we propose Adversarial Residual Dictionary Learning (ARDL) to get rid of the heavy\nreliance on 3D data. Thanks to the 3D-based deep framework and the capacity of ARDL for \ufb01ne-\ngrained texture representation [6], high-resolution results with faithful texture details can be obtained.\nWe make extensive comparisons with state-of-the-art methods on the IJB-A, LFW [18] and Multi-PIE\ndatasets. We also frontalize 256 \u00d7 256 images from CelebA-HQ [22] to push forward the advance\nin high-resolution face frontalization. Quantitative and qualitative results demonstrate our HF-PIM\ndramatically improves pose-invariant face recognition and produces photographic high-resolution\nresults potentially bene\ufb01tting many real-world applications.\nTo summarize, our main contributions are listed as follows:\n\n\u2022 A novel High Fidelity Pose Invariant Model (HF-PIM) is proposed to produce more realistic\n\nand identity-preserving frontalized face images with a higher resolution.\n\n\u2022 Through dense correspondence \ufb01eld estimation and facial texture map recovering, our\nwarping procedure can frontalize pro\ufb01le images with large poses and preserves abundant\nlatent 3D shape information.\n\n\u2022 Without the need of 3D data, we propose ARDL to supervise the process of facial texture\nmap recovering, effectively compensating the texture representation capacity for 3D-based\nframework.\n\n\u2022 A uni\ufb01ed end-to-end deep network is built to integrate all algorithmic components, which\n\nmakes the training process elegant and \ufb02exible.\n\n\u2022 Extensive experiments on four face frontalization databases demonstrate that HF-PIM not\nonly boosts pose-invariant face recognition in the wild, but also dramatically improves the\nvisual quality of high-resolution images.\n\n2 Related Works\n\nIn recent years, GAN, proposed by Goodfellow et al. [10], has been successfully introduced into the\n\ufb01eld of computer vision. GAN can be regarded as a two-player non-cooperative game model. The\nmain components, generator and discriminator, are rivals of each other. The generator tries to map a\ngiven input distribution to a target data distribution. Whereas the discriminator tries to distinguish the\ndata produced by the generator from the real one. Recently, deep convolutional generative adversarial\nnetwork (DCGAN) [29] has demonstrated the superior performance of image generation. Info-GAN\n[4] applies information regularization to optimization. Furthermore, Wasserstein GAN [1] improves\nthe learning stability of GAN and provides solutions for debugging and hyperparameter searching\n\n2\n\n\ffor GAN. These successful theoretical analyses of GAN show the effectiveness and possibility of\nphotorealistic face image generation and synthesis.\nGAN has dominated the \ufb01eld of face frontalization since it is \ufb01rstly used by DR-GAN [33]. Later,\nTP-GAN [20] is proposed with a two-pathway structure and perceptual supervision. CAPG-GAN [17]\nintroduce pose guidance through inserting conditional information carried by \ufb01ve-point heatmaps.\nPIM [37] aims to generate high-quality results through adding regularization items to learn face\nrepresentations more robust to hard examples. CR-GAN [31] introduces a generation sideway to\nmaintain the completeness of the learned embedding space and utilizes both labeled and unlabeled\ndata to further enrich the embedding space for realistic generations. All those methods treat face\nfrontalization as a 2D image-to-image translation problem without considering the intrinsic 3D\nproperties of human face. They indeed perform well in the situation where training data is suf\ufb01cient\nand captured well controlled. However, in-the-wild setting often leads to inferior performance, as we\ndiscussed in Sec. 1.\nThe attempt to combine prior knowledge of 3D face has been made by FF-GAN [35], 3D-PIM\n[38] and UV-GAN [7]. Their and our methods are all 3D-based but there are many differences. In\nFF-GAN, a CNN is trained to regress the 3DMM coef\ufb01cients of the input. Those coef\ufb01cients are\nintegrated as a supplement of low-frequency information. 3D-PIM incorporates a simulator with\nthe aid of a 3DMM to obtain prior information to accelerate the training process and reduce the\namount of required training data. In contrast, we do not employ 3DMM to present shape or texture\ninformation. We introduce a novel dense correspondence \ufb01eld and frontalize the pro\ufb01les through\nwarping. UV-GAN leverages an out-of-the-box method to project a 2D face to a 3D surface space.\nTheir network can be regarded as a 2D image-to-image translation model in the facial texture space.\nIn contrast, once the training procedure is \ufb01nished, our model can estimate the latent 3D information\nfrom the pro\ufb01les without the need for any additional out-of-the-box methods.\nIt is also notable that some face frontalization methods tend to improve the performance of face\nrecognition by data augmentation. Inspired by [30], DA-GAN [36], which acts as a 2D face image\nre\ufb01ner, can be employed for pose-invariant face recognition. In brief, the re\ufb01ner improves the quality\nof data augmented by ordinary methods. The training processes for face recognition methods bene\ufb01t\nfrom these re\ufb01ned data and the performances are boosted. Thus, DA-GAN is a method for augmenting\ntraining data. Note that UV-GAN mentioned above can be used to bene\ufb01t face recognition in the\nsame manner with DA-GAN, so it is also a data augmentation method. In contrast, our HF-PIM\nis trained to directly rotate the given pro\ufb01le to the frontal face, which can be directly used for face\nrecognition.\n\n3 High Fidelity Pose Invariant Model\n\nGiven a pro\ufb01le face image X, our goal is to produce the frontal face image as close to the ground\ntruth Y as possible (X, Y \u2208 RN\u00d7N\u00d73). As a reminder, we use Iij to denote the value of the pixel\nwith coordinate (i, j) in an image I. To learn the mapping, image pairs (X, Y ) are employed for\nmodel training. Inspired by recent progress in 3D face analysis [12, 13], we propose a brand-new\nframework which frontalizes given pro\ufb01le face through recovering geometry and texture information\nof the 3D face without explicitly building it. Concretely, the facial texture map and a novel dense\ncorrespondence \ufb01eld are leveraged to produce Y through warping. The facial texture map T lies in\nUV space - a space in which the manifold of the face is \ufb02attened into a contiguous 2D atlas. Thus, T\nrepresents the surface of the 3D face. The dense correspondence \ufb01eld F = (u; v)(u, v \u2208 RN\u00d7N )\nis proposed to establish the connections between 2D and 3D surface spaces. F is speci\ufb01ed by the\nfollowing statement: assuming that the coordinate of a point in T is (uij, vij), the corresponding\ncoordinate in Y is (i, j) after the warping operation. The right side on the top of Fig. 1 provides\nan intuitionistic illustration. Formally, given T and F with respect to Y , T can be warped into Y\nthrough the following formulation:\n\nYij = warp(i, j; F , T ) = Tuij ,vij , (i, j) \u2208 F,\n\n(1)\nwhere F is the coordinate set of those pixels standing for the facial part of Y . Our proposed warping\nprocedure inherits the virtue of morphable model construction: geometry and texture are well\ndisentangled whereas bound by dense correspondence. However, there are also some limitations, e.g.,\nneglect of image background. Thus, additional process is necessary to produce the non-facial regions\n\n3\n\n\fFigure 1: Left side on top: the framework of our HF-PIM to frontalize face images. The procedure\nconsists of correspondence \ufb01eld estimation (A), facial texture feature map recovering (B) and frontal\nview warping (C). The right side on top: an illustration about the warping procedure discussed in\nEq. 1. Those red dots and purple lines indicate the relationships between the facial texture map,\ncorrespondence \ufb01eld, and the RGB color image. Bottom side: the discriminators employed for ARDL\n(on the left) and ordinary adversarial learning (on the right).\n\nfor Eq. 1. To overcome these constraints, we employ a CNN-based wrapper Rec to take T as well as\nF as the input and map them to Y . Our Rec can produce the non-\ufb01cal parts simultaneously with\nthe facial parts and keep the overall visual realism consistent with Y . Concretely, Rec is trained via\noptimizing the reconstruction loss:\n\nLrec = (cid:107)Rec(F , T ) \u2212 Y (cid:107)1,\n\n(2)\n\nwhere (cid:107)\u00b7(cid:107)1 denotes calculating the mean of the element-wise absolute value summation of a matrix.\nNote that for our Rec, T is not limited to the RGB color space. In our experiment, we increase the\nnumber of feature channel to 32 and \ufb01nd that better performance is obtained.\nIn the following, we describe how to estimate the dense correspondence \ufb01eld F of the frontal view in\nSec. 3.1. Then, the recovering procedure of the facial texture map T via ARDL is illustrated in Sec.\n3.2. Regularization items and the overall loss function are introduced in Sec. 3.3.\n\n3.1 Dense Correspondence Field Estimation\n\nTo obtain the ground truth dense correspondence \ufb01eld F of monocular frontal face images for training,\nwe employ face reconstruction method for 3D shape estimation. Concretely, we employ BFM [28]\nas the 3D face model. Through the model \ufb01tting method proposed by [41], we get estimated shape\nparameters containing coordinates of vertices. To build F , we map those vertices to UV space via the\ncylindrical unwrapping described in [3]. Those non-visible vertices are culled via z-buffering.\nTo infer the dense correspondence \ufb01eld of the frontal view from the pro\ufb01le image, we build a\ntransformative autoencoder, C, with U-Net architecture. Given the input, C \ufb01rst encodes it into\npose-invariant shape representations and then recover dense correspondence \ufb01eld of frontal view.\nFurther, those shortcuts in U-Net guarantee the preservation of spatial information in the output. To\n\n4\n\n\fsupervise C during training, we minimize the pixel-wise error between the estimated map and the\nground truth F , namely:\n\nLcorr = (cid:107)C(X) \u2212 F(cid:107)1.\n\n(3)\n\n3.2 Facial Texture Map Recovery\n\nWe employ a transformative autoencoder consisting of the encoder Et and the decoder Dt for facial\ntexture map recovering. However, the ground truth facial texture map T of monocular face image\ncaptured in the wild is absent. To sidestep the demand for T , we introduce Adversarial Residual\nDictionary Learning (ARDL) which provides supervision for learning. During the training procedure,\nonly Y is required instead of T .\nThe learning dictionary is set as: given a set of texture feature embeddings B = {b1,\u00b7\u00b7\u00b7 , bn}\nand a learnable codebook C = {c1,\u00b7\u00b7\u00b7 , cm} containing m codewords with d dimension, the\ncorresponding residual vector is denoted as rik = bi \u2212 ck, for i = 1,\u00b7\u00b7\u00b7 , n and k = 1,\u00b7\u00b7\u00b7 , m.\nThrough dictionary encoding, a \ufb01xed length representation E = {e1,\u00b7\u00b7\u00b7 , em} can be calculated as\nfollows:\n\nn(cid:88)\n\nn(cid:88)\n\nek =\n\neik =\n\nwikrik,\n\n(4)\n\nwhere wik is the corresponding weight for rik. Inspired by [34] that assigns a descriptor to each\ncodeword, we make those weights learnable. Concretely, the assigning weight is given by:\n\ni=1\n\ni=1\n\nwik =\n\nm(cid:80)\n\nexp(\u2212sk(cid:107)rik(cid:107)2)\nexp(\u2212sj(cid:107)rij(cid:107)2)\n\n,\n\nj=1\n\n(5)\n\nwhere s = (s1,\u00b7\u00b7\u00b7 , sm) is the smoothing factor, which is also learnable. We denote the mapping\nfrom feature embeddings to dictionary representation as Ddic. The encoder Et is also employed to\nextract features for the dictionary learning.\nWe combine dictionary representation with adversarial learning, i.e., propose ARDL based on such\nan observation: when the identity label is \ufb01xed, for X across different poses, the recovered texture\nmap T should be invariant. To this end, Et should eliminate those discrepancies caused by different\nviews and encode the input into pose-invariant facial texture representation. We introduce adversarial\nlearning mechanism to supervise Et by making Ddic as its rival. Formally, the adversarial loss\nintroduced by ARDL is formulated as:\n\nLadv = EX\u223cpdata [log Ddic(Et(X))].\n\nAccordingly, Ddic is optimized to minimize:\n\nLdic = EX,Y\u223cpdata [log Ddic(Et(Y)) + log(1 \u2212 Ddic(Et(X))],\n\n(6)\n\n(7)\n\nwhere we add a fully connected (FC) layer upon Ddic to make binary predictions standing for real and\nfake. Through optimizing Eq. 6 and 7 alternatively, Et manages to make the encodings of the pro\ufb01le\nand the frontal view as similar as possible. In the meantime, Ddic tries to \ufb01nd the clues standing for\npose information, which provides the adversarial supervision information for Et.\n\n3.3 Overall Training Method\n\nFollowing previous work [21, 20], we also add the perceptual loss to integrate domain knowledge of\nidentities. An identity preserving network, e.g., VGG-Face [26] or Light CNN [16], can be employed\n\n5\n\n\fto supervise the frontalized results to be as close to the ground truth as possible in feature-level.\nFormally, the perceptual loss is formulated as:\n\nLp = (cid:107)\u03c6(Rec(F , T )) \u2212 \u03c6(Y )(cid:107)2\n2 ,\n\n(8)\nwhere \u03c6(\u00b7) denotes the extracted identity representation obtained by the second last fully connected\nlayer within the identity preserving network and (cid:107)\u00b7(cid:107)2 denotes the vector 2-norm.\nWe also introduce the adversarial loss in the RGB color image space following those GAN-based\nmethods [33, 20, 17, 37, 35]. A CNN named Drgb is employed to give adversarial supervision in\ncolor space. Note that our method can be easily extended to those advanced versions [1, 25] of GAN.\nBut in this paper, we simply use the original form of adversarial loss function [10] to prove that the\neffectiveness comes from our own contributions.\nIn summary, all the involved algorithmic components in our network are differentiable. Hence, the\nparameters can be optimized in an end-to-end manner via gradient backpropagation. The whole\ntraining process is described in Algorithm 1.\n\nAlgorithm 1 Training algorithm of HF-PIM\n1: Input: pro\ufb01le X, the ground truth frontal face Y with the ground truth dense correspondence\n\n\ufb01eld F , maximum iteration iter and the identity preserving network [16].\n\n2: Output: the frontalized result \u02c6Y\n3: Initializing C, Et, Dt, Rec, Ddic, Drgb\n4: i \u2190 0\n5: while i < iter do\n6:\n7:\n8:\n9:\n\nand Ld (for the discriminator)\n\nSampling training data\nModel forward propagation\nCalculating Lrec, Lcorr, Ldic, Ladv and Lp\nCalculating the adversarial losses in the RGB color image space, i.e., Lg (for the generator)\nL \u2190 Lrec + Lcorr + Ladv + Lp + Lg\nOptimize C, Rec, Et, Dt by minimizing L\nOptimize Ddic by minimizing Ldic\nOptimize Drgb by minimizing Ld\ni \u2190 i + 1\n\n10:\n11:\n12:\n13:\n14:\n15: end while\n\n4 Experiments\n\n4.1 Experimental Settings\n\nDatasets. To demonstrate the superiority of our method in both controlled and unconstrained\nenvironments and produce high-resolution face frontalization results, we conduct our experiment on\nfour datasets: Multi-PIE [11], LFW [18], IJB-A [23], and CelebA-HQ [22]. Multi-PIE is established\nfor studying on PIE (pose, illumination and expression) invariant face recognition. 20 illumination\nconditions, 13 poses within 90 yaw angles and 6 expressions of 337 subjects were captured in\ncontrolled environments. LFW is a benchmark database for face recognition. Over 13,000 face\nimages are captured in unconstrained environments. IJB-A is the most challenging unconstrained\nface recognition dataset at present. It has 5, 396 images and 20, 412 video frames of 500 subjects with\nlarge pose variations. CelebA [24] is a large-scale face attributes dataset. Contained images cover\nlarge pose variations and background clutter. CelebA-HQ is a high-resolution subset established by\n[22]. Since Multi-PIE, LFW and IJB-A consist of images with relatively low resolutions, we use\nCelebA-HQ for high-resolution (256 \u00d7 256) face frontalization.\nImplementation Details. The training set is drawn from Multi-PIE and CelebA-HQ. We follow the\nprotocol in [33] to split the Multi-PIE dataset. The \ufb01rst 200 subjects are used for training and the\nrest 137 ones for testing. Each testing identity has one gallery image from his/her \ufb01rst appearance.\nHence, there are 72,000 and 137 images in the probe and gallery sets, respectively. For CelebA-HQ,\nwe apply head pose estimation [41] to \ufb01nd those frontal faces and employ them (19, 203 images) for\n\n6\n\n\fTable 1: Comparisons on rank-1 recognition rates (%) across views under Multi-PIE Setting 2.\n\n\u00b115\u25e6 \u00b130\u25e6 \u00b145\u25e6 \u00b160\u25e6 \u00b175\u25e6 \u00b190\u25e6\n94.9\n94.6\n98.6\n98.7\n99.8\n99.3\n99.99\n\n87.2\n89.7\n92.1\n95.4\n97.3\n98.5\n99.88\n\n84.6\n85.2\n62.1\n87.7\n90.3\n98.1\n99.14\n\n91.1\n92.5\n97.4\n98.1\n99.6\n99.0\n99.98\n\n-\n\n77.2\n24.2\n77.4\n83.1\n95.0\n96.40\n\nMethod\n\nDR-GAN [33]\nFF-GAN [35]\nLight CNN [16]\nTP-GAN [20]\n\nCAPG-GAN [17]\n\nPIM [37]\n\nHF-PIM(Ours)\n\n-\n\n61.2\n5.5\n64.6\n66.1\n86.5\n92.32\n\ntraining. We choose those images with large poses (5, 998 ones) for testing. Apparently, there are no\noverlap between our training and testing sets. LFW and IJB-A are only used for testing. Note that\nthe images selected for training in CelebA-HQ are all frontal view, and we employ the face pro\ufb01ling\nmethod in [40] to make corresponding pro\ufb01les. We adapt the model architecture in [39] to build\nour networks. We use Adam optimizer with a learning rate of 1e-4 and \u03b21 = 0.5, \u03b22 = 0.99. Our\nproposed method is implemented based on the deep learning library Pytorch [27]. Two NVIDIA\nTitan X GPUs with 12GB GDDR5X RAM is employed for the training and testing process.\nEvaluation Metrics. To measure the quality of frontalized faces, the most common method is to\nevaluate the face recognition/veri\ufb01cation performances via \u201crecognition via generation\u201d, which means\npro\ufb01les are frontalized \ufb01rst, and then the performance is evaluated on these processed face images.\nThis evaluation manner prefers frontalization results that preserve more identity information and\ndirectly re\ufb02ect the contributions of frontalization methods on face recognition. Thus, \u201crecognition\nvia generation\u201d has been adopted by a series of existing methods [20, 17, 35, 37]. Besides, since\nphotographic results also indicate the performances qualitatively, visual quality is also compared in\nour experiment, as most GAN-based methods do.\n\n4.2 Frontalization Results in Controlled Situations\n\nIn this subsection, we systematically compare our method with DR-GAN, TP-GAN, FF-GAN, CAPG-\nGAN and PIM on the Multi-PIE dataset. Those pro\ufb01les with extreme poses (75\u25e6 and 90\u25e6) are very\nchallenging cases. Our performances are tested following the protocol of the setting 2 provided by\nMulti-PIE. Remind that our performance is evaluated by the \u201crecognition via generation\u201d framework.\nConcretely, when evaluating on Multi-PIE, pro\ufb01les are \ufb01rst frontalized by our model and then used\ndirectly for veri\ufb01cation and recognition. As for evaluating on those in-the-wild datasets (discussed in\nthe next subsection), all the faces are frontalized by our model since their yaw angles are not known in\nadvance. After the frontalization preprocessing, Light CNN [16] is employed as the feature extractor.\nWe compute the cosine distance of extracted feature vectors for veri\ufb01cation and recognition. The\nresults are reported across different poses in Table 1. Note that the manners for evaluating TP-GAN,\nFF-GAN, CAPG-GAN, and PIM are the same with our model. Light CNN is used for these methods\nexcept FF-GAN (their feature extractor is not publicly available). DR-GAN is evaluated in a different\nmanner: the feature vectors are directly extracted by their model. Thus, no extra feature extractor is\nneeded for DR-GAN. Besides frontalization methods, the performance of Light CNN is also included\nas the baseline. The results are reported across different poses in Table 1. For those poses less than\n60\u25e6, the performances of most methods are quite good whereas our method performs better. We infer\nthat the performance has almost saturated in this case. For those extreme poses, our methods can still\nproduce visually convincing results and achieve state-of-the-art recognition performance. In general,\nwhen testing on Multi-PIE, due to its balanced data distribution and highly controlled environment,\nmost methods perform relatively well (except those extreme poses).\n\n4.3 Frontalization Results in the Wild\n\nExtending face frontalization to in-the-wild setting is a very challenging problem with signi\ufb01cant\nimportance. We focus on testing on IJB-A and LFW in this subsection. For LFW, we evaluate face\nveri\ufb01cation performance on the frontalized results of the 6000 face pairs provided by the dataset.\nFor IJB-A, both veri\ufb01cation and identi\ufb01cation are tested in 10-fold cross-validation. The results are\n\n7\n\n\fTable 2: Face recognition performance (%) comparisons for in-the-wild datasets. The left part is\ncompared on LFW and the right side is on IJB-A. The results on IJB-A are averaged over 10 testing\nsplits. \u201c-\u201d means the result is not reported.\n\nMethod\n\nLFW\n\nVeri\ufb01cation\nAUC\nACC\n\nMethod\n\nVeri\ufb01cation\n\nRecognition\n\nIJB-A\n\nTP-GAN [20]\nFF-GAN [35]\nLight CNN [16]\nCAPG-GAN [17]\nHF-PIM(Ours)\n\n96.13\n96.42\n99.39\n99.37\n99.41\n\n99.42\n99.45\n99.87\n99.90\n99.92\n\nDR-GAN [33]\nFF-GAN [35]\nLight CNN [16]\n\nPIM [37]\n\nHF-PIM(Ours)\n\nFAR=0.01\n77.4\u00b12.7\n85.2\u00b11.0\n91.5\u00b11.0\n93.3\u00b11.1\n95.2\u00b10.7\n\nFAR=0.001\n53.9\u00b14.3\n66.3\u00b13.3\n84.3\u00b12.4\n87.5\u00b11.8\n89.7\u00b11.4\n\nRank-1\n85.5\u00b11.5\n90.2\u00b10.6\n93.0\u00b11.0\n94.4\u00b11.1\n96.1\u00b10.5\n\nRank-5\n94.7\u00b11.1\n95.4\u00b10.5\n\n-\n-\n\n97.9\u00b10.2\n\nFigure 2: Visual comparisons of face frontalization results. The samples on the left are drawn from\nLFW and the right side are from IJB-A.\n\nsummarized in Table 2. All the methods are tested with the same setting. Note that the training set of\nIJB-A is not been used by any involved method for comparison.\nWe can see that face frontalization methods only marginally improve the performance on LFW\nbecause most faces in this dataset are (near) frontal view. Besides, the baseline model Light CNN\nhas already achieved a relatively high performance. But our method still outperforms existing\nfrontalization methods in this case. When testing on IJB-A which contains lots of images with\nlarge and even extreme poses, our method shows a signi\ufb01cant improvement for face veri\ufb01cation\nand recognition. The visual comparison2, which is shown in Fig 2, also proves our superiority of\npreserving identity information and texture details. Thanks to the 3D-based framework and powerful\nadversarial residual dictionary learning, our HF-PIM produces results with very high \ufb01delity. For\nother methods, they indeed produce reasonable images but redundant manipulations can be observed.\nFor instance, DR-GAN make the eyes of the subject in the middle in IJB-A open; TP-GAN and\nCAPR-GAN tend to change the skin color and background.\n\n4.4 High-Resolution Face Frontalization\n\nGenerating high-resolution results has great importance on extending the application of face frontal-\nization. However, due to its dif\ufb01culty, few methods consider producing images with size larger than\n128 \u00d7 128. To further demonstrate our superiority, frontalized 256 \u00d7 256 results on CelebA-HQ are\nproposed in this paper. Some samples are shown in Fig 3. We also make comparisons with TP-GAN\nand CAPG-GAN. Note that since results on CelebA-HQ have not been reported by previous methods,\nwe contact the authors to get their model and produce 128 \u00d7 128 results through carefully following\ntheir instructions. The images in CelebA-HQ contain rich textures that are dif\ufb01cult for the generator\nto reproduce faithfully. Even in such a challenging situation, HF-PIM is still able to produce plausible\nresults. The results of [17] and [20] look less appealing.\n\n2Visual results produced by other methods are released by their authors. Different methods usually report\n\nvisual examples of different identities. We try our best to \ufb01nd those identities reported by most methods.\n\n8\n\n\fFigure 3: High-resolution frontalized results on the testing set of CelebA-HQ. The \ufb01rst row is the\ninput pro\ufb01le images. The second row is the frontalized images produce by our HF-PIM. The results\nof CAPG-GAN (on the left for each subject) and TP-GAN (on the right) are shown in the third row.\n\nExisting methods [20, 33, 37, 35, 17] measure the performance of face recognition to re\ufb02ect the\nquality of frontalized results. This measurement cannot be applied to those datasets without identity\nlabels (like CelebA-HQ) and neglects texture information that are not sensitive to identity. However,\nthe neglected textures also play an import role on the visual quality and should be preserved faithfully.\nFor face attribute analysis, data augmentation and many other practical applications, recovering\nhigh-resolution frontal view with detailed texture information has great potential for making progress.\nFinding new applications for face frontalization and putting forward new metrics need further\nresearch.\n\n5 Conclusion\n\nThis paper has proposed High Fidelity Pose Invariant Model (HF-PIM) to produce realistic and\nidentity-preserving frontalization results with a higher resolution. HF-PIM combines the advantages\nof 3D and GAN based methods and frontalizes pro\ufb01le images via a novel texture warping procedure.\nThrough leveraging a novel dense correspondence \ufb01eld, the prerequisite of warping is decomposed\ninto dense correspondence \ufb01eld estimation and facial texture map recovering, which are well addressed\nby a uni\ufb01ed end-to-end deep network. We also have introduced Adversarial Residual Dictionary\nLearning (ARDL) to supervise facial texture map recovering without the need of 3D data. Exhaustive\nexperiments have shown proposed method can preserve more identity information as well as texture\ndetails, which make the high-resolution results far more realistic.\n\n6 Acknowledgments\n\nThis work is funded by the National Key Research and Development Program of China (Grant No.\n2017YFC0821602, 2016YFB1001000) and the National Natural Science Foundation of China (Grant\nNo. 61427811, 61573360).\n\nReferences\n[1] Arjovsky, Martin, Chintala, Soumith, and Bottou, L\u00e9on. Wasserstein GAN. In ICML, 2017.\n\n9\n\n\f[2] Blanz, Volker and Vetter, Thomas. A morphable model for the synthesis of 3D faces. In\n\nSIGGRAPH, 1999.\n\n[3] Booth, James and Zafeiriou, Stefanos. Optimal UV spaces for facial morphable model construc-\n\ntion. In ICIP, 2014.\n\n[4] Chen, Xi, Duan, Yan, Houthooft, Rein, Schulman, John, Sutskever, Ilya, and Abbeel, Pieter. In-\nfoGAN: Interpretable representation learning by information maximizing generative adversarial\nnets. In NeurIPS, 2016.\n\n[5] Cole, Forrester, Belanger, David, Krishnan, Dilip, Sarna, Aaron, Mosseri, Inbar, and Freeman,\n\nWilliam T. Synthesizing normalized faces from facial identity features. In CVPR, 2017.\n\n[6] Dana, Hang Zhang Jia Xue Kristin. Deep TEN: Texture encoding network. In CVPR, 2017.\n\n[7] Deng, Jiankang, Cheng, Shiyang, Xue, Niannan, Zhou, Yuxiang, and Zafeiriou, Stefanos. UV-\nGAN: Adversarial facial UV map completion for pose-invariant face recognition. In CVPR,\n2018.\n\n[8] Dovgard, Roman and Basri, Ronen. Statistical symmetric shape from shading for 3D structure\n\nrecovery of faces. In ECCV, 2004.\n\n[9] Ferrari, Claudio, Lisanti, Giuseppe, Berretti, Stefano, and Del Bimbo, Alberto. Effective 3D\n\nbased frontalization for unconstrained face recognition. In ICPR, 2016.\n\n[10] Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair,\nSherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In NeurIPS, 2014.\n\n[11] Gross, Ralph, Matthews, Iain, Cohn, Jeffrey, Kanade, Takeo, and Baker, Simon. Multi-PIE.\n\nIVC, 2010.\n\n[12] G\u00fcler, R\u0131za Alp, Trigeorgis, George, Antonakos, Epameinondas, Snape, Patrick, Zafeiriou,\nStefanos, and Kokkinos, Iasonas. Densereg: Fully convolutional dense shape regression in-the-\nwild. In CVPR, 2017.\n\n[13] G\u00fcler, R\u0131za Alp, Neverova, Natalia, and Kokkinos, Iasonas. DensePose: Dense human pose\n\nestimation in the wild. In CVPR, 2018.\n\n[14] Hassner, Tal. Viewing real-world faces in 3D. In ICCV, 2013.\n\n[15] Hassner, Tal, Harel, Shai, Paz, Eran, and Enbar, Roee. Effective face frontalization in uncon-\n\nstrained images. In CVPR, 2015.\n\n[16] He, Ran, Wu, Xiang, Sun, Zhenan, and Tan, Tieniu. Learning invariant deep representation for\n\nNIR-VIS face recognition. In AAAI, 2017.\n\n[17] Hu, Yibo, Wu, Xiang, Yu, Bing, He, Ran, and Sun, Zhenan. Pose-guided photorealistic face\n\nrotation. In CVPR, 2018.\n\n[18] Huang, Gary B, Ramesh, Manu, Berg, Tamara, and Learned-Miller, Erik. Labeled faces in the\nwild: A database for studying face recognition in unconstrained environments. Technical report,\nUniversity of Massachusetts, Amherst, 2007.\n\n[19] Huang, Huaibo, He, Ran, Sun, Zhenan, and Tan, Tieniu. Wavelet-SRnet: A wavelet-based CNN\n\nfor multi-scale face super resolution. In ICCV, 2017.\n\n[20] Huang, Rui, Zhang, Shu, Li, Tianyu, and He, Ran. Beyond face rotation: Global and local\nperception GAN for photorealistic and identity preserving frontal view synthesis. In ICCV,\n2017.\n\n[21] Johnson, Justin, Alahi, Alexandre, and Fei-Fei, Li. Perceptual losses for real-time style transfer\n\nand super-resolution. In ECCV, 2016.\n\n[22] Karras, Tero, Aila, Timo, Laine, Samuli, and Lehtinen, Jaakko. Progressive growing of GANs\n\nfor improved quality, stability, and variation. In ICLR, 2018.\n\n10\n\n\f[23] Klare, Brendan F., Jain, Anil K., Klein, Ben, Taborsky, Emma, Blanton, Austin, Cheney,\nJordan, Allen, Kristen, Grother, Patrick, Mah, Alan, and Burge, Mark. Pushing the frontiers of\nunconstrained face detection and recognition: IARPA Janus Benchmark A. 2015.\n\n[24] Liu, Ziwei, Luo, Ping, Wang, Xiaogang, and Tang, Xiaoou. Deep learning face attributes in the\n\nwild. In ICCV, 2015.\n\n[25] Mao, Xudong, Li, Qing, Xie, Haoran, Lau, Raymond YK, Wang, Zhen, and Smolley,\n\nStephen Paul. Least squares generative adversarial networks. In ICCV, 2017.\n\n[26] Parkhi, Omkar M, Vedaldi, Andrea, Zisserman, Andrew, et al. Deep face recognition. In BMVC,\n\n2015.\n\n[27] Paszke, Adam, Gross, Sam, Chintala, Soumith, Chanan, Gregory, Yang, Edward, DeVito,\nZachary, Lin, Zeming, Desmaison, Alban, Antiga, Luca, and Lerer, Adam. Automatic differen-\ntiation in pytorch. In NeurIPS-W, 2017.\n\n[28] Paysan, Pascal, Knothe, Reinhard, Amberg, Brian, Romdhani, Sami, and Vetter, Thomas. A 3D\n\nface model for pose and illumination invariant face recognition. In AVSS, 2009.\n\n[29] Radford, Alec, Metz, Luke, and Chintala, Soumith. Unsupervised representation learning with\n\ndeep convolutional generative adversarial networks. In ICLR, 2016.\n\n[30] Shrivastava, Ashish, P\ufb01ster, Tomas, Tuzel, Oncel, Susskind, Josh, Wang, Wenda, and Webb,\nRuss. Learning from simulated and unsupervised images through adversarial training. In CVPR,\n2017.\n\n[31] Tian, Yu, Peng, Xi, Zhao, Long, Zhang, Shaoting, and Metaxas, Dimitris N. CR-GAN: Learning\n\ncomplete representations for multi-view generation. In IJCAI, 2018.\n\n[32] Tran, Luan and Liu, Xiaoming. Nonlinear 3D face morphable model. In CVPR, 2018.\n\n[33] Tran, Luan, Yin, Xi, and Liu, Xiaoming. Disentangled representation learning GAN for\n\npose-invariant face recognition. In CVPR, 2017.\n\n[34] Van Gemert, Jan C, Geusebroek, Jan-Mark, Veenman, Cor J, and Smeulders, Arnold WM.\n\nKernel codebooks for scene categorization. In ECCV, 2008.\n\n[35] Yin, Xi, Yu, Xiang, Sohn, Kihyuk, Liu, Xiaoming, and Chandraker, Manmohan. Towards\n\nlarge-pose face frontalization in the wild. In ICCV, 2017.\n\n[36] Zhao, Jian, Xiong, Lin, Jayashree, Panasonic Karlekar, Li, Jianshu, Zhao, Fang, Wang, Zhecan,\nPranata, Panasonic Sugiri, Shen, Panasonic Shengmei, Yan, Shuicheng, and Feng, Jiashi. Dual-\nIn NeurIPS,\nagent GANs for photorealistic and identity preserving pro\ufb01le face synthesis.\n2017.\n\n[37] Zhao, Jian, Cheng, Yu, Xu, Yan, Xiong, Lin, Li, Jianshu, Zhao, Fang, Jayashree, Karlekar,\nPranata, Sugiri, Shen, Shengmei, Xing, Junliang, et al. Towards pose invariant face recognition\nin the wild. In CVPR, 2018.\n\n[38] Zhao, Jian, Xiong, Lin, Cheng, Yu, Cheng, Yi, Li, Jianshu, Zhou, Li, Xu, Yan, Karlekar,\nJayashree, Pranata, Sugiri, Shen, Shengmei, et al. 3d-aided deep pose-invariant face recognition.\nIn IJCAI, 2018.\n\n[39] Zhu, Jun-Yan, Park, Taesung, Isola, Phillip, and Efros, Alexei A. Unpaired image-to-image\n\ntranslation using cycle-consistent adversarial networks. In ICCV, 2017.\n\n[40] Zhu, Xiangyu, Lei, Zhen, Yan, Junjie, Yi, Dong, and Li, Stan Z. High-\ufb01delity pose and\n\nexpression normalization for face recognition in the wild. In CVPR, 2015.\n\n[41] Zhu, Xiangyu, Lei, Zhen, Liu, Xiaoming, Shi, Hailin, and Li, Stan Z. Face alignment across\n\nlarge poses: A 3D solution. In CVPR, 2016.\n\n11\n\n\f", "award": [], "sourceid": 1496, "authors": [{"given_name": "Jie", "family_name": "Cao", "institution": "Center for Research on Intelligent Perception and Computing (CRIPAC) at Institute of Automation, Chinese Academy of Sciences."}, {"given_name": "Yibo", "family_name": "Hu", "institution": "Institute of Automation, Chinese Academy of Sciences"}, {"given_name": "Hongwen", "family_name": "Zhang", "institution": "CASIA"}, {"given_name": "Ran", "family_name": "He", "institution": "NLPR, CASIA"}, {"given_name": "Zhenan", "family_name": "Sun", "institution": "Institute of Automation, Chinese Academy of Sciences (CASIA)"}]}