{"title": "Pose Guided Person Image Generation", "book": "Advances in Neural Information Processing Systems", "page_first": 406, "page_last": 416, "abstract": "This paper proposes the novel Pose Guided Person Generation Network (PG$^2$) that allows to synthesize person images in arbitrary poses, based on an image of that person and a novel pose. Our generation framework PG$^2$ utilizes the pose information explicitly and consists of two key stages: pose integration and image refinement. In the first stage the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. The second stage then refines the initial and blurry result by training a U-Net-like generator in an adversarial way. Extensive experimental results on both 128$\\times$64 re-identification images and 256$\\times$256 fashion photos show that our model generates high-quality person images with convincing details.", "full_text": "Pose Guided Person Image Generation\n\nLiqian Ma1 Xu Jia2\u2217 Qianru Sun3\u2217 Bernt Schiele3 Tinne Tuytelaars2 Luc Van Gool1,4\n\n1KU-Leuven/PSI, TRACE (Toyota Res in Europe)\n\n2KU-Leuven/PSI, IMEC\n\n3Max Planck Institute for Informatics, Saarland Informatics Campus\n\n{liqian.ma, xu.jia, tinne.tuytelaars, luc.vangool}@esat.kuleuven.be\n\n{qsun, schiele}@mpi-inf.mpg.de\n\nvangool@vision.ee.ethz.ch\n\n4ETH Zurich\n\nAbstract\n\nThis paper proposes the novel Pose Guided Person Generation Network (PG2)\nthat allows to synthesize person images in arbitrary poses, based on an image of\nthat person and a novel pose. Our generation framework PG2 utilizes the pose\ninformation explicitly and consists of two key stages: pose integration and image\nre\ufb01nement. In the \ufb01rst stage the condition image and the target pose are fed into a\nU-Net-like network to generate an initial but coarse image of the person with the\ntarget pose. The second stage then re\ufb01nes the initial and blurry result by training a\nU-Net-like generator in an adversarial way. Extensive experimental results on both\n128\u00d764 re-identi\ufb01cation images and 256\u00d7256 fashion photos show that our model\ngenerates high-quality person images with convincing details.\n\n1\n\nIntroduction\n\nGenerating realistic-looking images is of great value for many applications such as face editing,\nmovie making and image retrieval based on synthesized images. Consequently, a wide range of\nmethods have been proposed including Variational Autoencoders (VAE) [14], Generative Adversarial\nNetworks (GANs) [6] and Autoregressive models (e.g., PixelRNN [30]). Recently, GAN models have\nbeen particularly popular due to their principle ability to generate sharp images through adversarial\ntraining. For example in [21, 5, 1], GANs are leveraged to generate faces and natural scene images\nand several methods are proposed to stabilize the training process and to improve the quality of\ngeneration.\nFrom an application perspective, users typically have a particular intention in mind such as changing\nthe background, an object\u2019s category, its color or viewpoint. The key idea of our approach is to\nguide the generation process explicitly by an appropriate representation of that intention to enable\ndirect control over the generation process. More speci\ufb01cally, we propose to generate an image by\nconditioning it on both a reference image and a speci\ufb01ed pose. With a reference image as condition,\nthe model has suf\ufb01cient information about the appearance of the desired object in advance. The\nguidance given by the intended pose is both explicit and \ufb02exible. So in principle this approach can\nmanipulate any object to an arbitrary pose. In this work, we focus on transferring a person from a\ngiven pose to an intended pose. There are many interesting applications derived from this task. For\nexample, in movie making, we can directly manipulate a character\u2019s human body to a desired pose or,\nfor human pose estimation, we can generate training data for rare but important poses.\nTransferring a person from one pose to another is a challenging task. A few examples can be seen in\nFigure 1. It is dif\ufb01cult for a complete end-to-end framework to do this because it has to generate both\ncorrect poses and detailed appearance simultaneously. Therefore, we adopt a divide-and-conquer\nstrategy, dividing the problem into two stages which focus on learning global human body structure\n\n\u2217Equal contribution.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fFigure 1: Generated samples on DeepFashion dataset [16] (a)(c) and Market-1501 dataset [37] (b).\n\nand appearance details respectively similar to [35, 9, 3, 19]. At stage-I, we explore different ways to\nmodel pose information. A variant of U-Net is employed to integrate the target pose with the person\nimage. It outputs a coarse generation result that captures the global structure of the human body in\nthe target image. A masked L1 loss is proposed to suppress the in\ufb02uence of background change\nbetween condition image and target image. However, it would generate blurry result due to the use of\nL1. At stage-II, a variant of Deep Convolutional GAN (DCGAN) model is used to further re\ufb01ne the\ninitial generation result. The model learns to \ufb01ll in more appearance details via adversarial training\nand generates sharper images. Different from the common use of GANs which directly learns to\ngenerate an image from scratch, in this work we train a GAN to generate a difference map between\nthe initial generation result and the target person image. The training converges faster since it is an\neasier task. Besides, we add a masked L1 loss to regularize the training of the generator such that it\nwill not generate an image with many artifacts. Experiments on two dataset, a low-resolution person\nre-identi\ufb01cation dataset and a high-resolution fashion photo dataset, demonstrate the effectiveness of\nthe proposed method.\nOur contribution is three-fold. i) We propose a novel task of conditioning image generation on a\nreference image and an intended pose, whose purpose is to manipulate a person in an image to an\narbitrary pose. ii) Several ways are explored to integrate pose information with a person image.\nA novel mask loss is proposed to encourage the model to focus on transferring the human body\nappearance instead of background information. iii) To address the challenging task of pose transfer,\nwe divide the problem into two stages, with the stage-I focusing on global structure of the human\nbody and the stage-II on \ufb01lling in appearance details based on the \ufb01rst stage result.\n\n2 Related works\n\nRecently there have been a lot of works on generative image modeling with deep learning techniques.\nThese works fall into two categories. The \ufb01rst line of works follow an unsupervised setting. One\npopular method under this setting is variational autoencoders proposed by Kingma and Welling [14]\nand Rezende et al. [25], which apply a re-parameterization trick to maximize the lower bound of the\ndata likelihood. Another branch of methods are autogressive models [28, 30, 29] which compute the\nproduct of conditional distributions of pixels in a pixel-by-pixel manner as the joint distribution of\npixels in an image. The most popular methods are generative adversarial networks (GAN) [6], which\nsimultaneously learn a generator to generate samples and a discriminator to discriminate generated\nsamples from real ones. Many works show that GANs can generate sharp images because of using\n\n2\n\nOur generation sequence (refined results)Condition imageTargetposeCoarseresultRefinedresultTargetimage (GT)Condition imageTargetposeCoarseresultRefinedresultTargetimage (GT)Condition imageTarget pose sequence(c) Generating from a sequence of poses(a) DeepFashion(b) Market-1501Refined results\fFigure 2: The overall framework of our Pose Guided Person Generation Network (PG2). It contains\ntwo stages. Stage-I focuses on pose integration and generates an initial result that captures the global\nstructure of the human. Stage-II focuses on re\ufb01ning the initial result via adversarial training and\ngenerates sharper images.\n\nthe adversarial loss instead of L1 loss. In this work, we also use the adversarial loss in our framework\nin order to generate high-frequency details in images.\nThe second group of works generate images conditioned on either category or attribute labels, texts or\nimages. Yan et al. [32] proposed a Conditional Variational Autoencoder (CVAE) to achieve attribute\nconditioned image generation. Mirza and Osindero [18] proposed to condition both generator\nand discriminator of GAN on side information to perform category conditioned image generation.\nLassner et al. [15] generated full-body people in clothing, by conditioning on the \ufb01ne-grained body\npart segments. Reed et al. proposed to generate bird image conditioned on text descriptions by adding\ntextual information to both generator and discriminator [24] and further explored the use of additional\nlocation, keypoints or segmentation information to generate images [22, 23]. With only these visual\ncues as condition and in contrast to our explicit condition on the intended pose, the control exerted\nover the image generation process is still abstract. Several works further conditioned image generation\nnot only on labels and texts but also on images. Researchers [34, 33, 11, 8] addressed the task of face\nimage generation conditioned on a reference image and a speci\ufb01c face viewpoint. Chen et al. [4]\ntackled the unseen view inference as a tensor completion problem, and use latent factors to impute\nthe pose in unseen views. Zhao et al. [36] explored generating multi-view cloth images from only a\nsingle view input, which is most similar to our task. However, a wide range of poses is consistent\nwith any given viewpoint making the conditioning less expressive than in our work. In this work, we\nmake use of pose information in a more explicit and \ufb02exible way, that is, using poses in the format\nof keypoints to model diverse human body appearance. It should be noted that instead of doing\nexpensive pose annotation, we use a state-of-the-art pose estimation approach to obtain the desired\nhuman body keypoints.\n\n3 Method\n\nOur task is to simultaneously transfer the appearance of a person from a given pose to a desired pose\nand keep important appearance details of the identity. As it is challenging to implement this as an\nend-to-end model, we propose a two-stage approach to address this task, with each stage focusing on\none aspect. For the \ufb01rst stage we propose and analyze several model variants and for the second stage\nwe use a variant of a conditional DCGAN to \ufb01ll in more appearance details. The overall framework\nof the proposed Pose Guided Person Generation Network (PG2) is shown in Figure 2.\n\n3.1 Stage-I: Pose integration\n\nAt stage-I, we integrate a conditioning person image IA with a target pose PB to generate a coarse\nresult \u02c6IB that captures the global structure of the human body in the target image IB.\n\n3\n\nTargetposeCondition imagecoarseresultDiscriminator (D) at Stage-IIConditionimageConditionimageTargetimageFake pairGenerator at Stage-I (G1)......Generator at Stage-II (G2)RealorFakedifference maprefined resultCondition imageskip connectionskip connectionconcatconcatReal pairResUAE (G1) at Stage-IFConv-ResUAE (G2) at Stage-II\fPose embedding. To avoid expensive annotation of poses, we apply a state-of-the-art pose estima-\ntor [2] to obtain approximate human body poses. The pose estimator generates the coordinates of\n18 keypoints. Using those directly as input to our model would require the model to learn to map\neach keypoint to a position on the human body. Therefore, we encode pose PB as 18 heatmaps. Each\nheatmap is \ufb01lled with 1 in a radius of 4 pixels around the corresponding keypoints and 0 elsewhere\n(see Figure 3, target pose). We concatenate IA and PB as input to our model. In this way, we can\ndirectly use convolutional layers to integrate the two kinds of information.\nGenerator G1. As generator at stage I, we adopt a U-Net-like architecture [20], i.e., convolutional\nautoencoder with skip connections as is shown in Figure 2. Speci\ufb01cally, we \ufb01rst use several stacked\nconvolutional layers to integrate IA and PB from small local neighborhoods to larger ones so\nthat appearance information can be integrated and transferred to neighboring body parts. Then, a\nfully connected layer is used such that information between distant body parts can also exchange\ninformation. After that, the decoder is composed of a set of stacked convolutional layers which are\nsymmetric to the encoder to generate an image. The result of the \ufb01rst stage is denoted as \u02c6IB1. In the\nU-Net, skip connections between encoder and decoder help propagate image information directly\nfrom input to output. In addition, we \ufb01nd that using residual blocks as basic component improves the\ngeneration performance. In particular we propose to simplify the original residual block [7] and have\nonly two consecutive conv-relu inside a residual block.\nPose mask loss. To compare the generation \u02c6IB1\nwith the target image IB, we adopt L1 distance\nas the generation loss of stage-I. However, since\nwe only have a condition image and a target\npose as input, it is dif\ufb01cult for the model to\ngenerate what the background would look like if\nthe target image has a different background from\nthe condition image. Thus, in order to alleviate\nthe in\ufb02uence of background changes, we add\nanother term that adds a pose mask MB to the\nL1 loss such that the human body is given more\nweight than the background. The formulation of\npose mask loss is given in Eq. 1 with (cid:12) denoting\nthe pixels-wise multiplication:\n\nFigure 3: Process of computing the pose mask.\n\nLG1 = (cid:107)(G1(IA, PB) \u2212 IB) (cid:12) (1 + MB)(cid:107)1.\n\n(1)\nThe pose mask MB is set to 1 for foreground and 0 for background and is computed by connecting\nhuman body parts and applying a set of morphological operations such that it is able to approximately\ncover the whole human body in the target image, see the example in Figure 3.\nThe output of G1 is blurry because the L1 loss encourages the result to be an average of all possible\ncases [10]. However, G1 does capture the global structural information speci\ufb01ed by the target pose,\nas shown in Figure 2, as well as other low-frequency information such as the color of clothes. Details\nof body appearance, i.e. the high-frequency information, will be re\ufb01ned at the second stage through\nadversarial training.\n\n3.2 Stage-II: Image re\ufb01nement\n\nSince the model at the \ufb01rst stage has already synthesized an image which is coarse but close to\nthe target image in pose and basic color, at the second stage, we would like the model to focus on\ngenerating more details by correcting what is wrong or missing in the initial result. We use a variant\nof conditional DCGAN [21] as our base model and condition it on the stage-I generation result.\nGenerator G2. Considering that the initial result and the target image are already structurally similar,\nwe propose that the generator G2 at the second stage aims to generate an appearance difference map\nthat brings the initial result closer to the target image. The difference map is computed using a U-Net\nsimilar to the \ufb01rst stage but with the initial result \u02c6IB1 and condition image IA as input instead. The\ndifference lies in that the fully-connected layer is removed from the U-Net. This helps to preserve\nmore details from the input because a fully-connected layer compresses a lot of information contained\nin the input. The use of difference maps speeds up the convergence of model training since the model\nfocuses on learning the missing appearance details instead of synthesizing the target image from\n\n4\n\npose keypointsestimationpose keypoints connectionTargetimageTargetposePose maskmorphologicaloperationsPose skeleton\fscratch. In particular, the training already starts from a reasonable result. The overall architecture of\nG2 can be seen in Figure 2.\nDiscriminator D. In traditional GANs, the discriminator distinguishes between real groundtruth\nimages and fake generated images (which is generated from random noise). However, in our\nconditional network, G2 takes the condition image IA instead of a random noise as input. Therefore,\nreal images are the ones which not only are natural but also satisfy a speci\ufb01c requirement. Otherwise,\nG2 will be mislead to directly output IA which is natural by itself instead of re\ufb01ning the coarse result\nof the \ufb01rst stage \u02c6IB1. To address this issue, we pair the G2 output with the condition image to make\nthe discriminator D to recognize the pairs\u2019 fakery, i.e., ( \u02c6IB2, IA) vs (IB, IA). This is diagrammed in\nFigure 2. The pairwise input encourages D to learn the distinction between \u02c6IB2 and IB instead of\nonly the distinction between synthesized and natural images.\nAnother difference from traditional GANs is that noise is not necessary anymore since the generator is\nconditioned on an image IA, which is similar to [17]. Therefore, we have the following loss function\nfor the discriminator D and the generator G2 respectively,\n\nLD\nadv = Lbce(D(IA, IB), 1) + Lbce(D(IA, G2(IA, \u02c6IB1)), 0),\nLG\nadv = Lbce(D(IA, G2(IA, \u02c6IB1)), 1),\n\n(2)\n(3)\nwhere Lbce denotes binary cross-entropy loss. Previous work [10, 17] shows that mixing the adver-\nsarial loss with a loss minimizing Lp distance can regularize the image generation process. Here\nwe use the same masked L1 loss as is used at the \ufb01rst stage such that it pays more attention to the\nappearance of targeted human body than background,\n\nLG2 = LG\n\nadv + \u03bb(cid:107)(G2(IA, \u02c6IB1) \u2212 IB) (cid:12) (1 + MB)(cid:107)1,\n\n(4)\nwhere \u03bb is the weight of L1 loss. It controls how close the generation looks like the target image at\nlow frequencies. When \u03bb is small, the adversarial loss dominates the training and it is more likely\nto generate artifacts; when \u03bb is big, the the generator with a basic L1 loss dominates the training,\nmaking the whole model generate blurry results2.\nIn the training process of our DCGAN, we alternatively optimize discriminator D and generator G2.\nAs shown in the left part of Figure 2, generator G2 takes the \ufb01rst stage result and the condition image\nas input and aims to re\ufb01ne the image to confuse the discriminator. The discriminator learns to classify\nthe pair of condition image and the generated image as fake while classifying the pair including the\ntarget image as real.\n\n3.3 Network architecture\n\nWe summarize the network architecture of the proposed model PG2. At stage-I, the encoder of G1\nconsists of N residual blocks and one fully-connected layer , where N depends on the size of input.\nEach residual block consists of two convolution layers with stride=1 followed by one sub-sampling\nconvolution layer with stride=2 except the last block. At stage-II, the encoder of G2 has a fully\nconvolutional architecture including N-2 convolution blocks. Each block consists of two convolution\nlayers with stride=1 and one sub-sampling convolution layer with stride=2. Decoders in both G1\nand G2 are symmetric to corresponding encoders. Besides, there are shortcut connections between\ndecoders and encoders, which can be seen in Figure 2. In G1 and G2, no batch normalization or\ndropout are applied. All convolution layers consist of 3\u00d73 \ufb01lters and the number of \ufb01lters are\nincreased linearly with each block. We apply recti\ufb01ed linear unit (ReLU) to each layer except the\nfully connected layer and the output convolution layer. For the discriminator, we adopt the same\nnetwork architecture as DCGAN [21] except the size of the input convolution layer due to different\nimage resolutions.\n\n4 Experiments\n\nWe evaluate the proposed PG2 network on two person datasets (Market-1501 [37] and DeepFash-\nion [16]), which contain person images with diverse poses. We present quantitative and qualitative\nresults for three main aspects of PG2: different pose embeddings; pose mask loss vs. standard L1\nloss; and two-stage model vs. one-stage model. We also compare with the most related work [36].\n\n2The in\ufb02uence of \u03bb on generation quality is analyzed in supplementary materials.\n\n5\n\n\f4.1 Datasets\n\nThe DeepFashion (In-shop Clothes Retrieval Benchmark) dataset [16] consists of 52,712 in-shop\nclothes images, and 200,000 cross-pose/scale pairs. All images are in high-resolution of 256\u00d7256.\nIn the train set, we have 146,680 pairs each of which is composed of two images of the same person\nbut different poses. We randomly select 12,800 pairs from the test set for testing.\nWe also experiment on a more challenging re-identi\ufb01cation dataset Market-1501 [37] containing\n32,668 images of 1,501 persons captured from six disjoint surveillance cameras. Persons in this\ndataset vary in pose, illumination, viewpoint and background, which makes the person generation\ntask more challenging. All images have size 128\u00d764 and are split into train/test sets of 12,936/19,732\nfollowing [37]. In the train set, we have 439,420 pairs each of which is composed of two images of\nthe same person but different poses. We randomly select 12,800 pairs from the test set for testing.\nImplementation details On both datasets, we use the Adam [13] optimizer with \u03b21 = 0.5 and\n\u03b22 = 0.999. The initial learning rate is set to 2e-5. On DeepFashion, we set the number of\nconvolution blocks N = 6. Models are trained with a minibatch of size 8 for 30k and 20k iterations\nrespectively at stage-I and stage-II. On Market-1501, we set the number of convolution blocks N = 5.\nModels are trained with a minibatch of size 16 for 22k and 14k iterations respectively at stage-I and\nstage-II. For data augmentation, we do left-right \ufb02ip for both datasets3.\n\n4.2 Qualitative results\n\nAs mentioned above, we investigate three aspects of our proposed PG2 network. Different pose\nembeddings and losses are compared within stage-I and then we demonstrate the advantage of our\ntwo-stage model over a one-stage model.\nDifferent pose embeddings. To evaluate our proposed pose embedding method, we implement two\nalternative methods. For the \ufb01rst, coordinate embedding (CE), we pass the keypoint coordinates\nthrough two fully connected layers and concatenate the embedded feature vector with the image\nembedding vector at the bottleneck fully connected layer. For the second, called heatmap embedding\n(HME), we feed the 18 keypoint heatmaps to an independent encoder and extract the fully connected\nlayer feature to concatenate with image embedding vector at the bottleneck fully connected layer.\nColumns 4, 5 and 6 of Figure 4 show qualitative results of the different pose embedding methods\nwhen used in stage-I, that is of G1 with CE (G1-CE-L1), with HME (G1-HME-L1) and our G1\n(G1-L1). All three use standard L1 loss. We can see that G1-L1 is able to synthesize reasonable\nlooking images that capture the global structure of a person, such as pose and color. However, the\nother two embedding methods G1-CE-L1 and G1-HME-L1 are quite blurry and the color is wrong.\nMoreover, results of G1-CE-L1 all get wrong poses. This can be explained by the additional dif\ufb01culty\nto map the keypoint coordinates to appropriate image locations making training more challenging.\nOur proposed pose embedding using 18 channels of pose heatmaps is able to guide the generation\nprocess effectively, leading to correctly generated poses. Interestingly, G1-L1 can even generate\nreasonable face details like eyes and mouth, as shown by the DeepFashion samples.\nPose mask loss vs. L1 loss. Comparing the results of G1 trained with L1 loss (G1-L1) and G1\ntrained with poseMaskLoss (G1-poseMaskLoss) for the Market-1501 dataset, we \ufb01nd that pose mask\nloss indeed brings improvement to the performance (columns 6 and 7 in Figure 4). By focusing the\nimage generation on the human body, the synthesized image gets sharper and the color looks nicer.\nWe can see that for person ID 164, the person\u2019s upper body generated by G1-L1 is more noisy in\ncolor than the one generated by G1-poseMaskLoss. For person ID 23 and 346, the method with pose\nmask loss generates more clear boundaries for shoulder and head. These comparisons validate that\nour pose mask loss effectively alleviates the in\ufb02uence of noisy backgrounds and guides the generator\nto focus on the pose transfer of the human body. The two losses generate similar results for the\nDeepFashion samples because the background is much simpler.\nTwo-stage vs. one-stage. In addition, we demonstrate the advantage of our two-stage model over\na one-stage model. For this we use G1 as generator but train it in an adversarial way to directly\ngenerate a new image given a condition image and a target pose as input. This one-stage model is\ndenoted as G1+D and our full model is denoted as G1+G2+D. From Figure 4, we can see that our full\nmodel is able to generate photo-realistic results, which contain more details than the one-stage model.\n\n3More details about parameters of the network architecture are given in supplementary materials.\n\n6\n\n\fFor example, for DeepFashion samples, more details in the face and the clothes are transferred to the\ngenerated images. For person ID 245, the shorts on the result of G1+D have lighter color and more\nblurry boundary than G1+G2+D. For person ID 346, the two-stage model is able to generate both\nthe right color and textures for the clothes, while the one-stage model is only able to generate the\nright color. On Market-1501 samples, the quality of the images generated by both methods decreases\nbecause of the more challenging setting. However, the two-stage model is still able to generate\nbetter results than the one-stage method. We can see that for person ID 53, the stripes on the T-shirt\nare retained by our full model while the one-stage model can only generate a blue blob as clothes.\nBesides, we can also clearly see the stool in the woman\u2019s hands (person ID 23).\n\nFigure 4: Test results on DeepFashion (upper 3 rows, images are cut for the sake of display) and\nMarket-1501 dataset (lower 3 rows). We test G1 in two aspects: (1) three pose embedding methods,\ni.e., coordinate embedding (CE), heatmap embedding (HME) and our pose heatmap concatenation\nin G1-L1, and (2) two losses, i.e., the proposed poseMaskLoss and the standard L1 loss. Column\n7, 8 and 9 show the differences among our stage-I (G1), one-stage adversarial model (G1+D) and\nour two-stage adversarial model (G1+G2+D). Note that all three use poseMaskLoss. The IDs are\nassigned randomly when splitting the datasets.\n\n4.3 Quantitative results\nWe also give quantitative results on both datasets. Structural Similarity (SSIM) [31] and the Inception\nScore (IS) [26] are adopted to measure the quality of synthesis. Note that in the Market-1501 dataset,\ncondition images and target images may have different background. Since there is no information\nin the input about the background in the target image, our method is not able to imagine what the\n\n7\n\nID. 49ID. 53ID. 164ID. 23Target image(GT)Condition imageG1-poseMaskLoss (our coarse result)G1-L1Target poseG1+DG1+G2+D(our refined result)G1-HME-L1G1-CE-L1ID. 147ID. 127ID. 245ID. 116ID. 146ID. 170123456789ID. 346\fTable 1: Quantitative evaluation. For all measures, higher is better.\n\nModel\nG1-CE-L1\nG1-HME-L1\nG1-L1\nG1-poseMaskLoss\nG1+D\nG1+G2+D\n\nSSIM\n0.694\n0.735\n0.735\n0.779\n0.761\n0.762\n\nDeepFashion\nIS\n\n2.395\n2.427\n2.427\n2.668\n3.091\n3.090\n\nSSIM\n0.219\n0.294\n0.304\n0.340\n0.283\n0.253\n\nIS\n\n2.568\n3.171\n3.006\n3.326\n3.490\n3.460\n\nMarket-1501\n\nmask-SSIM\n\n0.771\n0.802\n0.809\n0.817\n0.803\n0.792\n\nmask-IS\n2.455\n2.508\n2.455\n2.682\n3.310\n3.435\n\nTable 2: User study results from AMT\n\nModel\nG1+D\nG1+G2+D\n\nDeepFashion\nG2R5\n9.3%\n14.9%\n\nR2G4\n7.8%\n9.2%\n\nMarket-1501\nG2R\nR2G\n11.1%\n17.1%\n11.2%\n5.5%\n\nnew background looks like. To reduce the in\ufb02uence of background in our evaluation, we propose\na variant of SSIM, called mask-SSIM. A pose mask is added to both the synthesis and the target\nimage before computing SSIM. In this way we only focus on measuring the synthesis quality of a\nperson\u2019s appearance. Similarly, we employ mask-IS to eliminate the effect of background. However,\nit should be noted that image quality does not always correspond to such image similarity metrics.\nFor example, in Figure 4, our full model generates sharper and more photo-realistic results than\nG1-poseMaskLoss, but the latter one has a higher SSIM. This is also observed in super-resolution\npapers [12, 27].\nThe advantages are also clearly shown in the numerical scores in Table 1. E.g. the proposed pose\nembedding (G1-L1) consistently outperforms G1-CE-L1 across all measures and both datasets. G1-\nHME-L1 obtains similar quantitative numbers probably due to the similarity of the two embeddings.\nChanging the loss from L1 to the proposed poseMaskLoss (G1-poseMaskLoss) consistently improves\nfurther across all measures and for both datasets. Adding the discriminator during training either\nafter the \ufb01rst stage (G1+D) or in our full model (G1+G2+D) leads to comparable numbers, even\nthough we have observed clear differences in the qualitative results as discussed above. This is\nexplained by the fact that blurry images often get good SSIM despite being less convincing and\nphoto-realistic [12, 27].\n\n4.4 User study\n\nWe perform a user study on Amazon Mechanical Turk (AMT) for both datasets. For each one, we\nshow 55 real images and 55 generated images in a random order to 30 users. Following [10, 15],\neach image is shown for 1 second. The \ufb01rst 10 images are used for practice thus are ignored when\ncomputing scores. From the results reported in Table. 2, we can get some observations that (1)\nOn DeepFashion our generated images of G1+D and G1+G2+D manage to confuse users on 9.3%\nand 14.9% trials respectively (see G2R), showing the advantage of G1+G2+D over G1+D; (2) On\nMarket-1501, the average score of G2R is lower, because the background is much more cluttered than\nDeepFashion; (3) On Market-1501, G1+G2+D gets a lower score than G1+D, because G1+G2+D\ntransfers more backgrounds from the condition image, which can be \ufb01gured out in Figure. 4, but\nin the meantime it brings extra artifacts on backgrounds which lead users to rate \u2018Fake\u2019; (4) With\nrespect to R2G, we notice that Market-1501 gets clearly high scores (>10%) because human users\nsometimes get confused when facing low-quality surveillance images.\n\n4R2G means #Real images rated as generated / #Real images\n5G2R means #Generated images rated as Real / #Generated images\n\n8\n\n\fFigure 5: Comparison examples with [36].\n\nFigure 6: Our failure cases on DeepFashion.\n\n4.5 Further analysis\n\nSince our task with pose condition is novel, there is no direct comparison work. We only compare\nwith the most related one6 [36], which did multi-view person image synthesis on the DeepFashion\ndataset. It is noted that [36] used the condition image and an additional word vector of the target\nview e.g. \u201cside\u201d as network input. Comparison examples are shown in Figure 5. It is clear that\nour re\ufb01ned results are much better than those of [36]. Taking the second row as an example, we\ncan generate high-quality whole body images conditioned on an upper body while the whole body\nsynthesis by [36] only has a rough body shape.\nAdditionally, we give two failure DeepFashion examples by our model in Figure 6. In the top row,\nonly the upper body is generated consistently. The \u201cpieces of legs\u201d is caused by the rare training\ndata for such complicated poses. The bottom row shows inaccurate gender which is caused by the\nimbalance of training data for male / female. Besides, the condition person wears a long-sleeve jacket\nof similar color to his inner short-sleeve, making the generated cloth look like a mixture of both.\n\n5 Conclusions\n\nIn this work, we propose the Pose Guided Person Generation Network (PG2) to address a novel task\nof synthesizing person images by conditioning it on a reference image and a target pose. A divide-\nand-conquer strategy is employed to divide the generation process into two stages. Stage-I aims to\ncapture the global structure of a person and generate an initial result. A pose mask loss is further\nproposed to alleviate the in\ufb02uence of the background on person image synthesis. Stage-II \ufb01lls in\nmore appearance details via adversarial training to generate sharper images. Extensive experimental\nresults on two person datasets demonstrate that our method is able to generate images that are both\nphoto-realistic and pose-wise correct. In the future work, we plan to generate more controllable and\ndiverse person images conditioning on both pose and attribute.\n\nAcknowledgments\nWe gratefully acknowledge the support of Toyota Motors Europe, FWO Structure from Semantics\nproject, KU Leuven GOA project CAMETRON, and German Research Foundation (DFG CRC 1223).\nWe would like to thank Bo Zhao for his helpful discussions.\n\nReferences\n[1] Mart\u00edn Arjovsky, Soumith Chintala, and L\u00e9on Bottou. Wasserstein GAN. arXiv, 1701.07875, 2017.\n\n[2] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using\n\npart af\ufb01nity \ufb01elds. arXiv, 1611.08050, 2016.\n\n[3] Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iter-\native error feedback. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\npages 4733\u20134742, 2016.\n\n6Results of VariGAN are provided by the authors.\n\n9\n\nTarget image (GT)Condition imageOurs(refined) ID. 215VariGAN [36](refined)ID. 246few specific pose samples-->failure pose in G1ID. 2662few man samples and complex texture-->failure details in G2 Target image (GT)Condition imageOurs(refined) Target poseOurs(coarse) \f[4] Chao-Yeh Chen and Kristen Grauman. Inferring unseen views of people. In CVPR, pages 2003\u20132010,\n\n2014.\n\n[5] Xi Chen, Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan:\nInterpretable representation learning by information maximizing generative adversarial nets. In NIPS,\npages 2172\u20132180, 2016.\n\n[6] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,\n\nAaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.\n\n[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n\nIn CVPR, pages 770\u2013778, 2016.\n\n[8] Rui Huang, Shu Zhang, Tianyu Li, and Ran He. Beyond face rotation: Global and local perception gan for\n\nphotorealistic and identity preserving frontal view synthesis. arXiv, 1704.04086, 2017.\n\n[9] Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie. Stacked generative\n\nadversarial networks. arXiv preprint arXiv:1612.04357, 2016.\n\n[10] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional\n\nadversarial networks. In CVPR, 2017.\n\n[11] Xu Jia, Amir Ghodrati, Marco Pedersoli, and Tinne Tuytelaars. Towards automatic image editing: Learning\n\nto see another you. In BMVC, 2016.\n\n[12] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and\n\nsuper-resolution. In ECCV, 2016.\n\n[13] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv, 1412.6980, 2014.\n\n[14] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv, 1312.6114, 2013.\n\n[15] Christoph Lassner, Gerard Pons-Moll, and Peter V Gehler. A generative model of people in clothing. arXiv\n\npreprint arXiv:1705.04098, 2017.\n\n[16] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes\n\nrecognition and retrieval with rich annotations. In CVPR, pages 1096\u20131104, 2016.\n\n[17] Micha\u00ebl Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean\n\nsquare error. In ICLR, 2016.\n\n[18] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv, 1411.1784, 2014.\n\n[19] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In\n\nEuropean Conference on Computer Vision, pages 483\u2013499. Springer, 2016.\n\n[20] Tran Minh Quan, David G. C. Hildebrand, and Won-Ki Jeong. Fusionnet: A deep fully residual convolu-\n\ntional neural network for image segmentation in connectomics. arXiv, 1612.05360, 2016.\n\n[21] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep\n\nconvolutional generative adversarial networks. arXiv, 1511.06434, 2015.\n\n[22] Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning\n\nwhat and where to draw. In NIPS, 2016.\n\n[23] Scott Reed, A\u00e4ron van den Oord, Nal Kalchbrenner, Victor Bapst, Matt Botvinick, and Nando de Freitas.\n\nGenerating interpretable images with controllable structure. Technical report, 2016.\n\n[24] Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.\n\nGenerative adversarial text to image synthesis. In ICML, 2016.\n\n[25] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi-\n\nmate inference in deep generative models. In ICML, 2014.\n\n[26] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved\n\ntechniques for training gans. In NIPS, pages 2226\u20132234, 2016.\n\n[27] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel\nRueckert, and Zehan Wang. Real-time single image and video super-resolution using an ef\ufb01cient sub-pixel\nconvolutional neural network.\n\n10\n\n\f[28] Benigno Uria, Marc-Alexandre C\u00f4t\u00e9, Karol Gregor, Iain Murray, and Hugo Larochelle. Neural autoregres-\n\nsive distribution estimation. arXiv, 1605.02226, 2016.\n\n[29] A\u00e4ron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex\n\nGraves. Conditional image generation with pixelcnn decoders. In NIPS, pages 4790\u20134798, 2016.\n\n[30] A\u00e4ron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In\n\nICML, pages 1747\u20131756, 2016.\n\n[31] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from\n\nerror visibility to structural similarity. IEEE Trans. Image Processing, 13(4):600\u2013612, 2004.\n\n[32] Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attribute2image: Conditional image generation\n\nfrom visual attributes. In ECCV, pages 776\u2013791, 2016.\n\n[33] Jimei Yang, Scott Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentangling with\n\nrecurrent transformations for 3d view synthesis. In NIPS, 2015.\n\n[34] Junho Yim, Heechul Jung, ByungIn Yoo, Changkyu Choi, Du-Sik Park, and Junmo Kim. Rotating your\n\nface using multi-task deep neural network. In CVPR, 2015.\n\n[35] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dimitris\nMetaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks.\narXiv:, 1612.03242, 2016.\n\n[36] Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, and Jiashi Feng. Multi-view image generation from a\n\nsingle-view. arXiv, 1704.04886, 2017.\n\n[37] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person\n\nre-identi\ufb01cation: A benchmark. In ICCV, pages 1116\u20131124, 2015.\n\n11\n\n\f", "award": [], "sourceid": 301, "authors": [{"given_name": "Liqian", "family_name": "Ma", "institution": "KU Leuven"}, {"given_name": "Xu", "family_name": "Jia", "institution": "KU Leuven"}, {"given_name": "Qianru", "family_name": "Sun", "institution": "MPI Informatics"}, {"given_name": "Bernt", "family_name": "Schiele", "institution": "Max Planck Institute for Informatics"}, {"given_name": "Tinne", "family_name": "Tuytelaars", "institution": "KU Leuven"}, {"given_name": "Luc", "family_name": "Van Gool", "institution": "KU Leuven"}]}