{"title": "FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification", "book": "Advances in Neural Information Processing Systems", "page_first": 1222, "page_last": 1233, "abstract": "Person re-identification (reID) is an important task that requires to retrieve a person's images from an image dataset, given one image of the person of interest. For learning robust person features, the pose variation of person images is one of the key challenges. Existing works targeting the problem either perform human alignment, or learn human-region-based representations. Extra pose information and computational cost is generally required for inference. To solve this issue, a Feature Distilling Generative Adversarial Network (FD-GAN) is proposed for learning identity-related and pose-unrelated representations. It is a novel framework based on a Siamese structure with multiple novel discriminators on human poses and identities. In addition to the discriminators, a novel same-pose loss is also integrated, which requires appearance of a same person's generated images to be similar. After learning pose-unrelated person features with pose guidance, no auxiliary pose information and additional computational cost is required during testing. Our proposed FD-GAN achieves state-of-the-art performance on three person reID datasets, which demonstrates that the effectiveness and robust feature distilling capability of the proposed FD-GAN.", "full_text": "FD-GAN: Pose-guided Feature Distilling GAN for\n\nRobust Person Re-identi\ufb01cation\n\nYixiao Ge1\u2217\n\nZhuowan Li2,3\u2217\u2020\n\nHaiyu Zhao2\n\nShuai Yi2\n\nXiaogang Wang1\n\nGuojun Yin2,4\u2020\n\nHongsheng Li1 \u2021\n\n1CUHK-SenseTime Joint Laboratory, The Chinese University of Hong Kong\n\n2SenseTime Research\n\n3Johns Hopkins University\n\n4University of Science and Technology of China\n\n{yxge@link, hsli@ee, xgwang@ee}.cuhk.edu.hk\n\n{zhaohaiyu, yishuai}@sensetime.com\n\nzli110@jhu.edu\n\ngjyin@mail.ustc.edu.cn\n\nAbstract\n\nPerson re-identi\ufb01cation (reID) is an important task that requires to retrieve a\nperson\u2019s images from an image dataset, given one image of the person of interest.\nFor learning robust person features, the pose variation of person images is one of\nthe key challenges. Existing works targeting the problem either perform human\nalignment, or learn human-region-based representations. Extra pose information\nand computational cost is generally required for inference. To solve this issue,\na Feature Distilling Generative Adversarial Network (FD-GAN) is proposed for\nlearning identity-related and pose-unrelated representations. It is a novel framework\nbased on a Siamese structure with multiple novel discriminators on human poses\nand identities. In addition to the discriminators, a novel same-pose loss is also\nintegrated, which requires appearance of a same person\u2019s generated images to\nbe similar. After learning pose-unrelated person features with pose guidance, no\nauxiliary pose information and additional computational cost is required during\ntesting. Our proposed FD-GAN achieves state-of-the-art performance on three\nperson reID datasets, which demonstrates that the effectiveness and robust feature\ndistilling capability of the proposed FD-GAN. \u2021\u2021\n\n1\n\nIntroduction\n\nPerson re-identi\ufb01cation (reID) is a challenging task, with the purpose of matching pedestrian images\nwith the same identity across multiple cameras. With the wide usage of deep learning methods,\nreID performances by different algorithms increase rapidly. There are various attempts on learning\nrepresentations with deep neural networks, however, posture variations, blur and occlusion still pose\ngreat challenges for learning discriminative features. Two types of methods were used for addressing\nthe issues, aligning pedestrian images [1] or integrating human pose information by learning body-\nregion features [2]. However, these works also require auxiliary pose information in the inference\nstage, which limits the generalization of the algorithms to new images without pose information.\nMeanwhile, the computational cost increases due to more complicated inference of pose estimation.\n\n\u2217The \ufb01rst two authors contribute equally to this work.\n\u2020This work was done when they were interns at SenseTime Research.\n\u2021Hongsheng Li is the corresponding author.\n\u2021\u2021The code is now available. https://github.com/yxgeee/FD-GAN\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: The image encoder in the FD-GAN is trained to learn robust identity-related and pose-\nunrelated representations with assistance of pose-guided image generator and discriminators. During\ninference, it does not need pose information and additional computational cost.\n\nGenerative adversarial network (GAN) is gaining increasing attention for image generation. Recently,\nsome works exploited GANs\u2019 potential on aiding current person reID algorithms. Zheng et al. [3]\nproposed a semi-supervised structure which learns generative images with label smoothing regulariza-\ntion for outliers (LSRO) regularization. PTGAN [4] was proposed to bridge the domain gap between\nseparate datasets. In addition to image synthesis, GAN can be used for representation learning as\nwell. In this work, we propose a novel identity-related representation learning framework for robust\nperson re-identi\ufb01cation.\nThe proposed Feature Distilling Generative Adversarial Network (FD-GAN) maintains identity\nfeature consistency under pose variation without increasing the complexity of inference (illustrated in\nFigure 1). It adopts a Siamese structure for feature learning. Each of the branch consists of an image\nencoder and an image generator. The image encoder embeds person visual features given the input\nimages. The image generator generates new person images conditioned on the pose information and\nthe input person features by the encoder. Multiple discriminators are integrated in the framework to\ndistinguish inter-branch and intra-branch relations between generated images by the two branches.\nThe proposed identity discriminator, the pose discriminator, and the veri\ufb01cation classi\ufb01er together\nwith a reconstruction loss and a novel same-pose loss jointly regularizes the feature learning process\nfor achieving robust person reID. With the adversarial losses, identity-irrelevant information, such\nas pose and background appearance, in the input image is mitigated from the visual features by\nthe image encoder. More importantly, during inference, additional pose information is no longer\nneeded and saves additional computational cost. Our method outperforms previous works in three\nwidely-used reID datasets, i.e. Market-1501 [5], CUHK03 [6] and DukeMTMC-reID [7] datasets.\nOverall, this paper has the following contributions. 1) We propose a novel framework, FD-GAN,\nto learn identity-related and pose-unrelated representations for person re-identi\ufb01cation with pose-\nvariation. Unlike existing alignment or region-based learning methods, our framework does not\nrequire extra auxiliary pose information or increase the computational complexity during inference.\n2) Although person image generation is an auxiliary task for our framework, the generated person\nimages by our proposed method show better quality than existing speci\ufb01c person-generation methods.\n3) The proposed FD-GAN achieves state-of-the-art re-identi\ufb01cation performance on Market-1501 [5],\nCUHK03 [6], and DukeMTMC-reID [7] datasets.\n\n2 Related Work\n\nGenerative Adversarial Network (GAN). Goodfellow et al. [8] \ufb01rst introduced the adversarial\nprocess to learn generative models. The GAN is generally composed of a generator and a discrimi-\nnator, where the discriminator attempts to distinguish the generated images from real distribution\nand the generator learns to fool the discriminator. A set of constraints are proposed in previous\nworks [9, 10, 11, 12] to improve the training process of GANs, e.g., interpretable representations are\nlearned by using additional latent code in [12]. GAN-based algorithms shows excellent performance\nin image generation [13, 14, 15, 16, 17]. In terms of person image generation, PG2 was proposed\nto synthesize person images in arbitrary poses in [18]. Siarohin et al. [19] designed a single-stage\napproach with deformable skip connections in the generator for better deformable human generation.\nZan\ufb01r et al. [20] transferred the appearance from the source image onto the target image while\npreserving the target shape and clothing segmentation layout. In contrast, our method aims at learn-\ning person features for person reID with assistance of GANs. Apart from person image synthesis,\npose-disentangled representations were learned for face recognition by DR-GAN [21], which has key\ndifferences with our method. Our experimental results show that our proposed FD-GAN performs\n\n2\n\nEncoderDiscriminatorsVerificationGeneratorPerson ImagesPose & NoiseInferenceTrainPerson Features\fFigure 2: The Siamese structure of the proposed FD-GAN. Robust identity-related and pose-unrelated\nfeatures are learned by the image encoder E with a veri\ufb01cation loss and the auxiliary task of\ngenerating fake images to fool identity and pose discriminators. A novel same-pose loss term is\nintroduced to further encourage learning identity-related and pose-unrelated visual features.\n\nbetter than DR-GAN on person reID. Our main goal is to decompose the pose information from the\nimage features via adversarial training for learning identity-related and pose-unrelated representation.\nPerson Re-identi\ufb01cation (ReID). Person reID [22, 23, 24, 25, 26, 27, 28, 29] is a challenging task\ndue to various human poses, domain differences, occlusions, etc. Two main types of methods were\nadopted in previous works, i.e. learning discriminative person representations [30, 31, 32] and metric\nlearning [25, 33, 34, 35]. PAN [1] aligns pedestrians and learn person features simultaneously without\nany extra annotation. Zhao et al. [2] proposed SpindleNet for learning person features of different\nbody regions with additional human pose information. Most recent methods [1, 25, 33, 34, 35, 36, 37,\n38, 39] designed more complicated frameworks to learn more robust representations with increasing\ncomputational cost or requiring extra information during inference.\nInspired by the excellent performances of GAN-based structures for image generation, there were\nprevious works [3, 21, 4] begining to design GAN-based algorithms to improve veri\ufb01cation perfor-\nmance of person reID. Zheng et al. [3] introduced a semi-supervised pipeline for jointly training\ngenerated images and real images from training dataset by the proposed LSRO method for regulariz-\ning unlabelled data. PTGAN [4] was proposed to bridge the domain gap between separate person\nreID datasets. Due to the challenges from pose diversity for person reID datasets, we propose an\nnovel GAN-based framework for distilling identity-related features.\n\n3 Feature Distilling Generative Adversarial Network\n\nOur proposed Feature Distilling Generative Adversarial Network (FD-GAN) aims at learning identity-\nrelated and pose-unrelated person representations, in order to handle large pose variations across\nimages in person reID.\nThe overall framework of our proposed method is shown in Fig. 2. The proposed FD-GAN adopts\na Siamese structure, including an image encoder E, an image generator G, an identity veri\ufb01cation\nclassi\ufb01er V and two adversarial discriminators, i.e., the identity discriminator Did and the pose\ndiscriminator Dpd. For each branch of the network, it takes a person image and a target pose\nlandmark map as inputs. The image encoder E at each branch \ufb01rst transforms the input person\nimage into feature representations. An identity veri\ufb01cation classi\ufb01er is utilized to supervise the\nfeature learning for person reID. However, using only the veri\ufb01cation classi\ufb01er makes the encoder\ngenerally encode not only person identity information but also person pose information, which makes\nthe learned features sensitive to person pose variation. To make the learned features robust and\n\n3\n\nnoise zIdentity lossIdentity lossPose lossVerification lossPose lossInput Image 1Input Image 2Target PoseFake Image 1Fake Image 2ImageEncoderPoseEncoderGeneratorGeneratorPoseDiscriminatorIdentityDiscriminatorPoseDiscriminatorVerificationClassifierImageEncoderIdentityDiscriminatorSame-pose lossVisual FeaturesVisualFeatures\fFigure 3: Network structures of (a) the generator G and the image encoder E, (b) the veri\ufb01cation\nclassi\ufb01er V , (c) the identity discriminator Did, (d) the pose discriminator Dpd.\n\neliminate pose-related information, we added an image generator G conditioned on the features from\nthe encoder and a target pose map. The assumption is intuitive, if the learned person features are\npose-unrelated and identity-related, then it can be used to accurately generate the same person\u2019s\nimage but with different target poses. An identity discriminator Did and a pose discriminator\nDpd are integrated to regularize the image generation process. Both Did and Dpd are conditional\ndiscriminators that classify whether the input image is real or fake conditioned on the input identity\nor pose. They are not used to classify different identities and poses. The image generator together\nwith the image encoder are encouraged to fool the discriminators with fake generated images. Taking\nadvantages of the Siamese structure, a novel same-pose loss minimizing the difference between\nthe fake generated images of the two branches is also utilized, which is shown to further distill\npose-unrelated information from input images. The entire framework is joint trained in an end-to-end\nmanner. For inference, only the image encoder E is used without auxiliary pose information.\n\n3.1\n\nImage encoder and image generator\n\nThe structures of the image encoder E and image generator G are illustrated in Figure 3(a). Given\nan input image x, the image encoder E utilizes ResNet-50 as backbone network to encode the input\nimage into a 2048-dimensional feature vector. The image generator G takes the encoded person\nfeatures and target pose map as inputs, and aims at generating another image of the same person\nspeci\ufb01ed by the target pose. The target pose map is represented by an 18-channel map, where\neach channel represents the location of one pose landmark\u2019s location and the one-dot landmark\nlocation is converted to a Gaussian-like heat map. It is encoded by a 5-block Convolution-BN-ReLU\nsub-network to obtain a 128-dimensional pose feature vector. The visual features, target pose features,\nand an additional 256-dimensional noise vector sampled from standard Gaussian distribution are then\nconcatenated and input into a series of 5 convolution-BN-dropout-ReLU upsampling blocks to output\nthe generated person images.\n\n3.2\n\nIdentity veri\ufb01cation classi\ufb01er\n\nGiven the visual features of the two input images from the image encoder, the identity veri\ufb01cation\nclassi\ufb01er V determines whether the two images belong to the same person. Person identity veri\ufb01cation\nis the main task for person re-identi\ufb01cation and ensures learned features to capture identity information\nof person images. The structure of the classi\ufb01er is shown in Figure 3(b), which takes visual features of\ntwo person images as inputs and feeds them through element-wise subtraction, element-wise square,\na batch normalization layer, a fully-connected layer, and \ufb01nally a sigmoid non-linearity function to\noutput the probability that the input image pair belongs to the same person. This classi\ufb01er is trained\nwith binary cross-entropy loss. Let x1, x2 represent the two input person images, and d(x1, x2)\nrepresents the output same-person con\ufb01dence score by our sub-network. The identity veri\ufb01cation\nclassi\ufb01er V is trained with the following binary cross-entropy loss,\n\nLv = \u2212C log d(x1, x2) \u2212 (1 \u2212 C)(1 \u2212 log d(x1, x2)),\n\n(1)\n\n4\n\nResNetpxzCypyCPatchGAN0/1ResNetResNetxy-BN0/1Sigmoid(a) Image Encoder, Pose Encoder, Generator(c) Identity Discriminator(d) Pose DiscriminatorConv-BN-ReLU blockConv-BN-Dropout-ReLU blockImage EncoderPose EncoderGenerator-BN0/1Sigmoid(b) Verification ClassifierSquareSquareInput ImageTarget PoseNoise VectorFake ImageVisual Features 1Visual Features 2Input ImageFake/Real ImageFake/Real ImageTarget PoseImage-pose Matching MapSamePerson?FCFCSamePerson?\fwhere C is the ground-truth label. C = 1 if x1, x2 belong to the same person and C = 0 otherwise.\n\n3.3\n\nImage generation with identity and pose discriminators\n\n(cid:16)Ey(cid:48)\n\n2(cid:88)\n\nk=1\n\nLid = max\n\nDid\n\nk\u2208Y [log Did(xk, y(cid:48)\n\nTo regularize the image encoder E to learn only identity-related information, the following person\nimage generator G is trained with the identity discriminator Did and the pose discriminator Dpd to\ngenerate person images with target poses. Given the input image xk (k = 1 or 2 for two branches)\nand the target pose p, the generated image yk is required to have the same person identity with xk but\nwith the target pose p. The identity discriminator is utilized to maintain identity-related information in\nthe encoded visual features, while the pose discriminator aims to eliminate pose-related information\nfrom the features.\nIdentity discriminator Did is trained to distinguish whether the generated person image and the\ninput person image of the same branch belong to the same person. The image generator would try to\nfool the identity discriminator to ensure the encoded visual feature contains suf\ufb01cient identity-related\ninformation. The identity discriminator sub-network has a similar network structure (see Figure 3(c))\nto the identity veri\ufb01cation classi\ufb01er V . However, its ResNet-50 sub-network for visual feature\nencoding does not share weights with that of our image encoder E, because the identity discriminator\nDid aims at distinguishing the identity between the real/fake images, while our image encoder targets\nat learning pose-unrelated person features. There is domain gap between the two tasks and sharing\nweights hinders feature learning process of the image encoder. Such an argument is supported by our\nexperiments. Let y(cid:48)\nk represent the real person image having the same identity with input image xk\nand the target pose p. The adversarial loss of the identity discriminator Di can then be de\ufb01ned as\n\n(cid:17)\nk)] + Eyk\u2208Z [log(1 \u2212 Did(xk, yk))]\n\n,\n\n(2)\n\nwhere Y and Z represent the true data distribution and generated data distribution by the image\ngenerator G.\nPose discriminator Dpd is proposed to distinguish whether the generated person image yk (for k = 1\nor 2) matches the given target pose p. The sub-network structure of pose discriminator is shown in\nFigure 3(d). It adopts the PatchGAN [40] structure. The input image and pose map (after Gaussian-\nlike heat-map transformation) is \ufb01rst concatenated along the channel dimension and then processed by\n4 convolution-ReLU blocks and a sigmoid non-linearity to obtain an image-pose matching con\ufb01dence\nmap with values between 0 and 1. Each location of the con\ufb01dence map represents the matching\ndegree between the input person image and the pose landmark map. The image generator G would try\nto fool the pose discriminator Dpd to obtain high matching con\ufb01dences with fake generated images.\nThe adversarial loss of Ddp is then formulated as\nk\u2208Y [log Dpd([p, y(cid:48)\n\n(cid:17)\nk])] + Eyk\u2208Z [log(1 \u2212 Dpd([p, yk]))]\n\n(cid:16)Ey(cid:48)\n\nLpd = max\n\n(3)\n\n,\n\nDpd\n\n2(cid:88)\n\nk=1\n\nwhere Dpd utilizes the concatenated person image and pose landmark map as inputs.\nHowever, we observe that the pose discriminator Dpd might over\ufb01t the poses, i.e., Dpd might\nremember the correspondences between speci\ufb01c poses and person appearances, because each image\u2019s\npose is generally unique. For instance, if we use a blue-top person\u2019s pose as the target pose, the\ngenerated image of a red-top person might end up having blue top. To solve this problem, we propose\nan online pose map augmentation scheme. During training, for each pose landmark, its 1-channel\nGaussian-like heat-map is obtained with a random Gaussian bandwidth in some speci\ufb01c range. In\nthis way, we can create many pose maps for the same pose and mitigate the pose over\ufb01tting problem.\nReconstruction loss. The responsibility of G is not only confusing the discriminators, but also\ngenerating images that are similar to the ground-truth images. However, the discriminators alone\ncannot guarantee generating human-perceivable images. Therefore, a reconstruction loss is introduced\nto minimize the L1 differences between the generated image yk and its corresponding real image y(cid:48)\nk,\nwhich is shown to be helpful for more stable convergence of training the generator.\n\n2(cid:88)\n\nk=1\n\nLr =\n\n(cid:107)yk \u2212 y(cid:48)\n\nk(cid:107)1,\n\n1\nmn\n\n5\n\n(4)\n\n\fk for an input image xk and a target pose p, this loss is not utilized.\n\nwhere mn is the number of pixels in the real/fake images. When there is no corresponding ground-\ntruth image y(cid:48)\nSame-pose loss. The purpose of the image generator G is to help the image encoder distill only\npose-unrelated information. We input the same person\u2019s two different images and the same target pose\nto both branches of our Siamese network, if the conditioning visual features in the two branches are\ntruly only identity-related, then the two generated images should be similar in appearance. Therefore,\nwe propose a same-pose loss to minimize the differences between the two generated images of the\nsame person and with the target pose,\n\nLsp =\n\n1\nmn\n\n(cid:107)y1 \u2212 y2(cid:107)1,\n\n(5)\n\nwhich encourages the learned visual features from E of the two input images to only be identity-\nrelated while ignoring other factors.\nThe overall training objective. The above mentioned classi\ufb01er loss, discriminator losses and recon-\nstruction losses work collaboratively for learning identity-related and pose-unrelated representations.\nThe overall loss function could be de\ufb01ned by\n\nL = Lv + \u03bbidLid + \u03bbpdLpd + \u03bbrLr + \u03bbspLsp,\n\n(6)\n\nwhere \u03bbid, \u03bbpd, \u03bbr, \u03bbsp are the weighting factors for the auxiliary image generation task.\n\n3.4 Training scheme\n\nThere are three stages for training our proposed framework. In the \ufb01rst stage, our Siamese baseline\nmodel, which includes only the image encoder E and identity veri\ufb01cation classi\ufb01er V , is pretrained\non a person reID dataset with only identity cross-entropy loss Lv in Eq. (1). The pre-trained network\nweights are then used to initialize E, V , and identity discriminator Did in stage-II. In the second stage,\nparameters of E and V are \ufb01xed. We then train G, identity discriminator Did, and pose discriminator\nDpd with the overall objective L in Eq. (6). Finally, the whole network is \ufb01netuned jointly in an\nend-to-end manner. For each training mini-batch, it contains 128 person image pairs, with 32 of them\nbelonging to same persons (positive pairs) and 96 of them belonging to different persons (negative\npairs). All images are resized to 256 \u00d7 128. The Gaussian bandwidth for obtaining pose landmark\nheat-map is uniformly sampled in [4, 6].\nIn training stages II and III, the discriminators and other parts of the network are alternatively\noptimized. When jointly optimizing the generator G, the image encoder E and the veri\ufb01cation\nclassi\ufb01er V , the overall objective Eq. (6) is used. When optimizing the discriminators Did and Dpd,\nonly adversarial losses Lid and Lpd are adopted.\nStage I: ReID baseline pretraining. Our Siamese baseline only includes the image encoder E\nand identity veri\ufb01cation classi\ufb01er V . The ResNet-50 sub-network is \ufb01rst initialized with ImageNet-\npretrained weights [41]. The network is optimized by Stochastic Gradient Descent (SGD) with\nmomentum 0.9. The initial learning rates are set to 0.01 for E and 0.1 for V , and they are decreased\nto 0.1 of their previous values every 40 epochs. The stage-I training process iterates for 80 epochs.\nStage II: FD-GAN pretraining. With E and V \ufb01xed, We integrate G, Did, and Dpd into the\nframework in stage-II. Adam optimizer is adopted for optimizing G and SGD for Did and Dpd.\nThe initial learning rates for G, Did, Dpd are set as 10\u22123, 10\u22124, 10\u22122, respectively. Learning rates\nmaintain the same for the \ufb01rst 50 epochs, and then gradually decrease to 0 in the following 50 epochs.\nThe loss weights are set as \u03bbid = 0.1, \u03bbpd = 0.1, \u03bbr = 10, \u03bbsp = 1. We took the label smoothness\nscheme [42] for better balancing between the generator and the discriminator.\nStage III: Global \ufb01netuning. For \ufb01netuning the whole framework end-to-end, we use Adam for\noptimizing E, G and V , and SGD for Did, Dpd after loading the pre-trained weights from stage-II.\nSpeci\ufb01cally, the initial learning rates are set to 10\u22126, 10\u22126, 10\u22125, 10\u22124, 10\u22124 for E, G, V , Did, Dpd,\nrespectively. Learning rates remain the same for the \ufb01rst 25 epochs, and then gradually decay to 0 in\nthe following 25 epochs. Batch normalization layers in E is \ufb01xed to achieve better performance. For\nloss weights, \u03bbid = 0.1, \u03bbpd = 0.1, \u03bbr = 10, \u03bbsp = 1 are set as the weights for different loss terms.\n\u00a7\n\n\u00a7We tune hyperparameters on the validation set of Market-1501 [5], and directly use the same hyperparame-\n\nters for DukeMTMC-reID [7] and CUHK03 [6] datasets.\n\n6\n\n\fTable 1: Component analysis of the proposed FD-GAN on Market-1501 [5] and DukeMTMC-\nreID [7] datasets in terms of top-1 accuracy (%) and mAP (%)\n\nNetworks\n\nnot share E\n\nbaseline (single)\nbaseline (Siamese)\nSiamese DR-GAN[21]\nFD-GAN (share E)\nFD-GAN (no sp.)\nFD-GAN (no veri.)\nFD-GAN (no sp. & no veri.)\nFD-GAN (no Dpd)\nFD-GAN (no Did)\nFD-GAN (no Did & Dpd)\nFD-GAN (no pose aug.)\nFD-GAN\n\nn/a\nn/a\n\u00d7\n\u00d7\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\nComponents\n\nLsp Lv\n\u221a\nn/a\nn/a\n\u221a\nn/a\n\u00d7\n\u221a\n\u221a\n\u221a\n\u00d7\n\u221a\n\u00d7\n\u00d7\n\u00d7\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\nLpd Lid\nn/a\nn/a\n\u221a\n\u221a\nn/a\nn/a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u00d7\n\u221a\n\u00d7\n\u00d7\n\u00d7\n\u221a\n\u221a\n\u221a\n\u221a\n\npose map aug.\n\nn/a\n\u221a\nn/a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u221a\n\u00d7\n\u221a\n\nMarket-1501[5]\ntop-1\nmAP\n81.4\n59.8\n88.2\n72.5\n73.2\n86.7\n86.8\n73.5\n88.9\n75.8\n89.5\n75.7\n88.7\n74.4\n88.0\n73.0\n72.8\n89.2\n84.6\n71.6\n89.5\n77.2\n77.7\n90.5\n\nDukeMTMC-reID[7]\nmAP\n40.7\n61.3\n60.2\n\ntop-1\n62.5\n78.2\n76.9\n\n62.6\n62.4\n\n78.8\n78.6\n\n63.9\n64.5\n\n79.5\n80.0\n\n-\n-\n\n-\n-\n-\n\n-\n-\n\n-\n-\n-\n\n3.5 Comparison to DR-GAN [21]\n\nThere is an existing work, DR-GAN [21] based on conditional GAN [43], which tries to learn pose-\ninvariant identity representations for face recognition. It also adopts an encoder-decoder structure\nwith a discriminator for classifying both identity. Comparison results in Section 4.2 demonstrate the\nadvantages of our proposed method over DR-GAN on the person reID task.\nThis is because there are three key differences between the proposed FD-GAN and DR-GAN, which\nmake our algorithm superior. 1) We adopt a Siamese network structure, which enables us to use the\nsame-pose loss to encourage encoding only learning identity-related information, while DR-GAN\ndoes not have such a loss term. 2) We do not share the weights between the ResNet-50 networks in\nthe image encoder and in the identity discriminator. We observe that identity veri\ufb01cation and real/fake\nimage identity discrimination are two tasks in different domains and therefore their weights should\nnot be shared. 3) Our Siamese structure utilizes a veri\ufb01cation classi\ufb01er instead of a cross-entropy\nclassi\ufb01er, which shows better person reID performance than a single-branch network does.\n\n4 Experiments\n\n4.1 Datasets and evaluation metrics\n\nIn this paper, three datasets are used for performance evaluation, including Market-1501 [5],\nCUHK03 [6], and DukeMTMC-reID [7]. The Market-1501 dataset [5] consists of 12,936 im-\nages of 751 identities for training and 19,281 images of 750 identities in the gallery set for testing.\nThe CUHK03 dataset [6] contains 14,097 training images of 1,467 identities captured from two\ncameras. The original training and testing protocol is used. The DukeMTMC-reID dataset [7] is\na subset of the pedestrian tracking dataset DukeMTMC for image-based reID. It contains 16,522\nimages of 702 identities for training. Mean average precision (mAP) and CMC top-1 accuracy are\nadopted for performance evaluation on all the three datasets.\n\n4.2 Component analysis of the proposed FD-GAN\n\nIn this section, component analysis is conducted to demonstrate the effectiveness of components\nin the FD-GAN framework, including the Siamese structure, and the use of veri\ufb01cation classi\ufb01er\nand same-pose loss. We also compare with DR-GAN [21], which also proposes to learning pose\ndisentangled features. Our Siamese baseline model is only the ResNet-50 image encoder E with our\nidentity veri\ufb01cation classi\ufb01er V . The analysis is conducted on Market-1501 [5] and DukeMTMC-reID\n[7] datasets and the results are shown in Table 1.\nSiamese structure. We \ufb01rst compare the our Siamese reID baseline (denoted as baseline (Siamese))\nwith the single branch ResNet-50 [44] baseline trained with cross-entropy loss on person IDs (denoted\nas baseline (single)). The Siamese baseline outperforms single-branch baseline by 12.7% and 20.6%\nin terms of mAP on the two datasets.\nProposed FD-GAN, with online pose map augmentation and adversarial discriminators. Based\non the Siamese structure, we build our proposed FD-GAN framework. We can observe that the\n\n7\n\n\fTable 2: Experimental comparison of the proposed approach with state-of-the-art methods on\nMarket-1501 [5], CUHK03 [6], and DukeMTMC-reID [7] datasets. Top-1 accuracy(%) and mAP(%)\nare reported.\n\nMethods\n\nBoW+KISSME [5]\nLOMO+XQDA [37]\nOIM Loss [45]\nMSCAN [39]\nDCA [39]\nSpindleNet [2]\nk-reciprocal [46]\nVI+LSRO [3]\nBasel+LSRO [3]\nOL-MANS [47]\nPA [48]\nSVDNet [49]\nJLML [50]\nProposed FD-GAN\n\nMarket-1501 [5]\n\nmAP\n\ntop-1\n\n-\n-\n\n-\n\n-\n-\n\n60.9\n53.1\n57.5\n\n63.6\n66.1\n\n63.4\n62.1\n65.5\n77.7\n\n-\n-\n\n82.1\n76.3\n80.3\n76.9\n77.1\n84.0\n\n-\n\n60.7\n81.0\n82.3\n85.1\n90.5\n\nCUHK03 [6]\n\nmAP\n\n72.5\n\n-\n-\n\n-\n-\n-\n\n-\n-\n-\n\n-\n\n67.6\n87.4\n\n84.8\n\n91.3\n\ntop-1\n\n-\n-\n\n77.5\n74.2\n74.2\n88.5\n61.6\n84.6\n\n-\n\n61.7\n85.4\n81.8\n83.2\n92.6\n\nDukeMTMC-reID [7]\ntop-1\nmAP\n12.1\n25.1\n30.8\n17.0\n47.4\n68.1\n\n-\n-\n-\n-\n-\n\n-\n-\n\n-\n\n47.1\n\n56.8\n\n64.5\n\n-\n-\n-\n-\n-\n\n-\n-\n\n-\n\n67.7\n\n76.7\n\n80.0\n\nproposed FD-GAN gains signi\ufb01cant improvements from our Siamese baseline in terms of both mean\nAP and top-1 accuracy on both two reID datasets. There are 5.2% and 3.2% mAP improvements\nin terms of mAP on the two datasets. To show the effectiveness of our proposed online pose map\naugmentation, we test removing it when training our FD-GAN (denoted as FD-GAN w/o pose aug.\nin Table 1). It results in a .5% performance drop for both datasets. In order to validate the effects\nof the two discriminators Did and Dpd, we test removing them separately and together (denoted as\nFD-GAN w/o Did or Dpd. in Table 1). It results in not only obviously performance drop, but also\npoorer generated images.\nDR-GAN [21], veri\ufb01cation loss, same-pose loss, and not sharing image encoder. We also study\nthe effectiveness of using veri\ufb01cation loss and same-pose loss, and not sharing image encoder weights\nto identity discriminator. Original DR-GAN\u2019s pose discriminator classi\ufb01es each face image into\none of 13 poses. For fair comparison, we \ufb01rst test integrating DR-GAN into our Siamese baseline\n(denoted as Siamese DR-GAN), which could be viewed our FD-GAN without the same-pose loss and\nalso sharing weights between E and Did. Since our network uses pose map as input condition, we\nuse our conditional pose discriminator Dpd to replace DR-GAN\u2019s pose discriminator. The Siamese\nDR-GAN even performs worse than our Siamese baseline on the DukeMTMC-reID dataset. Our\nproposed FD-GAN outperforms it by over 4% mAP on both datasets. We also try removing both\nveri\ufb01cation classi\ufb01er and same-pose loss (denoted as FD-GAN w/o sp. & veri.), removing only\nidentity veri\ufb01cation classi\ufb01er (denoted as FD-GAN w/o veri.), removing only same-pose loss (denoted\nas FD-GAN w/o sp.) from our proposed FD-GAN and only sharing weights between E and Did\n(denoted as FD-GAN share E) . Results in Table 1 show that both the veri\ufb01cation loss and same-pose\nloss are indispensable to achieve superior performance on person reID. Also, not sharing weights\nbetween E and Did results in better performance.\n\n4.3 Comparison with state-of-the-arts\n\nWe compare our proposed FD-GAN with the state-of-the-art person reID methods including\nVI+LSRO [3], JLML [50], PA [48], etc. on the three datasets, Market-1501 [5] , CUHK03 [6]\n, and DukeMTMC-reID [7]. The results are listed in Table 2. Note that only single query results from\npublished papers are compared in order to make a fair comparison.\nBy \ufb01netuning the FD-GAN based on ResNet-50 [44] baseline network structure, our proposed FD-\nGAN outperforms previous approaches and achieves state-of-the-art performance. We can achieve\n90.5% top-1 accuracy and 77.7% mAP on the Market-1501 dataset [5], 92.6% top-1 accuracy and\n91.3% mAP on CUHK03 dataset [6], and 80.0% top-1 accuracy and 64.5% mAP on the DukeMTMC-\nreID dataset [7], which demonstrates the effectiveness of the proposed feature distilling FD-GAN.\n\n4.4 Person image generation and visual analysis\n\nComparison of person image generation [18, 19]. Although generating person images is only an\nauxiliary task in our FD-GAN to learn more robust person features. We are interested in comparing\n\n8\n\n\fInput Pose GT\n\n[18]\n\n[19] Ours\n\nInput Pose GT\n\n[18]\n\n[19] Ours\n\nInput Pose GT\n\n[18]\n\n[19] Ours\n\n(a)\n\n(b)\n\nFigure 4: (a) Generated person images by our proposed method and [18, 19] on test images of Market-\n1501 dataset [5]. (b) Two examples of the generated images from the Market-1501 dataset [5]. (First\nrow) the ground-truth images of target poses. (Second-third rows) input images and the generated\nimages with different target poses on training images of Market-1501 dataset [5].\n\nthe generated images with images by other speci\ufb01cally designed person generation methods [18, 19].\nFigure 4(a) shows the generated person images by state-of-the-art person generation methods [18, 19]\nand our FD-GAN. One can clearly see that our proposed method better understand the concept of\n\u201cbackpack\u201d and could generate correct upper and lower body clothes. We argue that the key is using\nperson identity supervisions to make the encoder learn better identity-related features. Our Siamese\nstructure and the same-pose loss also contribute to achieving consistent generation results.\nVisualization for learned features. The proposed FD-GAN framework not only improves the\ndiscriminative capability of visual features but could also be used as a visualization tool for manually\nexamining learned feature representations. The quality of learned person features have direct impact\non the generated person images. We can therefore tell what aspects of person appearances are\ncaptured by the features. For instance, for \u201cinput 1_b\u201d in Figure 4(b), its generated frontal images\ndo not show colored pattern on the upper body but only the general colors and shapes of the upper\nand lower bodies, which might demonstrate that the learned image encoder focus on embedding the\noverall appearances of persons but fail to capture the distinguishable details in appearance.\n\n5 Conclusion\n\nIn this paper, we proposed the novel FD-GAN for learning identity-related and pose-unrelated\nperson representations with human pose guidance. Novel Siamese network structure as well as\nnovel losses ensure the framework learns more pose-invariant features for robust person reID. Our\nproposed framework achieves state-of-the-art performance on person reID without using additional\ncomputational cost or extra pose information during inference. The generated person images also\nshow higher quality than existing speci\ufb01c person-generation methods.\n\nAcknowledgements\n\nThis work is supported by SenseTime Group Limited, the General Research Fund sponsored\nby the Research Grants Council of Hong Kong (Nos. CUHK14213616, CUHK14206114,\nCUHK14205615, CUHK14203015, CUHK14239816, CUHK419412, CUHK14207814,\nCUHK14208417, CUHK14202217),\nthe Hong Kong Innovation and Technology Support\nProgram (No. ITS/121/15FX).\n\n9\n\n\fReferences\n[1] Zheng, Z., Zheng, L., Yang, Y.: Pedestrian alignment network for large-scale person re-\n\nidenti\ufb01cation. TCSVT (2018)\n\n[2] Zhao, H., Tian, M., Sun, S., Shao, J., Yan, J., Yi, S., Wang, X., Tang, X.: Spindle net: Person\nre-identi\ufb01cation with human body region guided feature decomposition and fusion. In: CVPR.\n(2017)\n\n[3] Zheng, Z., Zheng, L., Yang, Y.: Unlabeled samples generated by gan improve the person\n\nre-identi\ufb01cation baseline in vitro. ICCV (2017)\n\n[4] Wei, L., Zhang, S., Gao, W., Tian, Q.: Person transfer gan to bridge domain gap for person\n\nre-identi\ufb01cation. CVPR (2018)\n\n[5] Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., Tian, Q.: Scalable person re-identi\ufb01cation: A\n\nbenchmark. In: CVPR. (2015)\n\n[6] Li, W., Zhao, R., Xiao, T., Wang, X.: Deepreid: Deep \ufb01lter pairing neural network for person\n\nre-identi\ufb01cation. In: CVPR. (2014)\n\n[7] Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set\n\nfor multi-target, multi-camera tracking. In: ECCV. (2016)\n\n[8] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.,\n\nBengio, Y.: Generative adversarial nets. In: NIPS. (2014) 2672\u20132680\n\n[9] Huang, X., Li, Y., Poursaeed, O., Hopcroft, J., Belongie, S.: Stacked generative adversarial\n\nnetworks. In: CVPR. (2017)\n\n[10] Nowozin, S., Cseke, B., Tomioka, R.:\n\nf-gan: Training generative neural samplers using\n\nvariational divergence minimization. In: NIPS. (2016)\n\n[11] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolu-\n\ntional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)\n\n[12] Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: Interpretable\nrepresentation learning by information maximizing generative adversarial nets. In: NIPS. (2016)\n2172\u20132180\n\n[13] Li, J., Liang, X., Wei, Y., Xu, T., Feng, J., Yan, S.: Perceptual generative adversarial networks\n\nfor small object detection. In: CVPR. (2017)\n\n[14] Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., Krishnan, D.: Unsupervised pixel-level\n\ndomain adaptation with generative adversarial networks. In: CVPR. (2017)\n\n[15] Kaneko, T., Hiramatsu, K., Kashino, K.: Generative attribute controller with conditional \ufb01ltered\n\ngenerative adversarial networks. In: CVPR. (2017)\n\n[16] Nguyen, V., Yago Vicente, T.F., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with\n\nconditional generative adversarial networks. In: ICCV. (2017)\n\n[17] Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.N.: Stackgan: Text to\nphoto-realistic image synthesis with stacked generative adversarial networks. In: ICCV. (2017)\n\n[18] Ma, L., Jia, X., Sun, Q., Schiele, B., Tuytelaars, T., Van Gool, L.: Pose guided person image\n\ngeneration. In: NIPS. (2017) 405\u2013415\n\n[19] Siarohin, A., Sangineto, E., Lathuiliere, S., Sebe, N.: Deformable gans for pose-based human\n\nimage generation. CVPR (2017)\n\n[20] Zan\ufb01r, M., Popa, A.I., Zan\ufb01r, A., Sminchisescu, C.: Human appearance transfer. In: CVPR.\n\n(2018)\n\n[21] Tran, L., Yin, X., Liu, X.: Disentangled representation learning gan for pose-invariant face\n\nrecognition. In: CVPR. (2017)\n\n10\n\n\f[22] Wu, A., Zheng, W.S., Yu, H.X., Gong, S., Lai, J.: Rgb-infrared cross-modality person re-\n\nidenti\ufb01cation. In: ICCV. (2017)\n\n[23] Su, C., Li, J., Zhang, S., Xing, J., Gao, W., Tian, Q.: Pose-driven deep convolutional model for\n\nperson re-identi\ufb01cation. In: ICCV. (2017)\n\n[24] Chung, D., Tahboub, K., Delp, E.J.: A two stream siamese convolutional neural network for\n\nperson re-identi\ufb01cation. In: ICCV. (Oct 2017)\n\n[25] Yu, H.X., Wu, A., Zheng, W.S.: Cross-view asymmetric metric learning for unsupervised\n\nperson re-identi\ufb01cation. In: ICCV. (2017)\n\n[26] Shen, Y., Li, H., Xiao, T., Yi, S., Chen, D., Wang, X.: Deep group-shuf\ufb02ing random walk for\n\nperson re-identi\ufb01cation. In: CVPR. (2018)\n\n[27] Shen, Y., Xiao, T., Li, H., Yi, S., Wang, X.: End-to-end deep kronecker-product matching for\n\nperson re-identi\ufb01cation. In: CVPR. (2018)\n\n[28] Shen, Y., Li, H., Yi, S., Chen, D., Wang, X.: Person re-identi\ufb01cation with deep similarity-guided\n\ngraph neural network. (2018)\n\n[29] Chen, D., Xu, D., Li, H., Sebe, N., Wang, X.: Group consistent similarity learning via deep crf\n\nfor person re-identi\ufb01cation. In: CVPR. (2018)\n\n[30] Xiao, T., Li, H., Ouyang, W., Wang, X.: Learning deep feature representations with domain\n\nguided dropout for person re-identi\ufb01cation. In: CVPR. (2016)\n\n[31] Cheng, D., Gong, Y., Zhou, S., Wang, J., Zheng, N.: Person re-identi\ufb01cation by multi-channel\n\nparts-based cnn with improved triplet loss function. In: CVPR. (2016)\n\n[32] Shi, Z., Hospedales, T.M., Xiang, T.: Transferring a semantic representation for person\n\nre-identi\ufb01cation and search. In: CVPR. (2015)\n\n[33] Zhang, L., Xiang, T., Gong, S.: Learning a discriminative null space for person re-identi\ufb01cation.\n\nIn: CVPR. (2016)\n\n[34] Liu, Z., Wang, D., Lu, H.: Stepwise metric promotion for unsupervised video person re-\n\nidenti\ufb01cation. In: ICCV. (2017)\n\n[35] Bak, S., Carr, P.: One-shot metric learning for person re-identi\ufb01cation. In: CVPR. (2017)\n\n[36] Chen, D., Yuan, Z., Chen, B., Zheng, N.: Similarity learning with spatial constraints for person\n\nre-identi\ufb01cation. In: CVPR. (2016) 1268\u20131277\n\n[37] Liao, S., Hu, Y., Zhu, X., Li, S.Z.: Person re-identi\ufb01cation by local maximal occurrence\n\nrepresentation and metric learning. In: CVPR. (2015)\n\n[38] Li, W., Zhao, R., Xiao, T., Wang, X.: Deepreid: Deep \ufb01lter pairing neural network for person\n\nre-identi\ufb01cation. In: CVPR. (2014)\n\n[39] Li, D., Chen, X., Zhang, Z., Huang, K.: Learning deep context-aware features over body and\n\nlatent parts for person re-identi\ufb01cation. In: CVPR. (2017)\n\n[40] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.:\n\nadversarial networks. CVPR (2017)\n\nImage-to-image translation with conditional\n\n[41] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical\n\nimage database. In: CVPR. (2009)\n\n[42] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.:\n\ntechniques for training gans. In: NIPS. (2016)\n\nImproved\n\n[43] Mirza, M., Osindero, S.:\n\narXiv:1411.1784 (2014)\n\nConditional generative adversarial nets.\n\narXiv preprint\n\n11\n\n\f[44] K. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR.\n\n(2016)\n\n[45] Xiao, T., Li, S., Wang, B., Lin, L., Wang, X.: Joint detection and identi\ufb01cation feature learning\n\nfor person search. In: CVPR. (2017)\n\n[46] Zhong, Z., Zheng, L., Cao, D., Li, S.: Re-ranking person re-identi\ufb01cation with k-reciprocal\n\nencoding. In: CVPR. (2017)\n\n[47] Zhou, J., Yu, P., Tang, W., Wu, Y.: Ef\ufb01cient online local metric adaptation via negative samples\n\nfor person reidenti\ufb01cation. In: CVPR. (2017)\n\n[48] Zhao, L., Li, X., Wang, J., Zhuang, Y.: Deeply-learned part-aligned representations for person\n\nre-identi\ufb01cation. arXiv preprint arXiv:1707.07256 (2017)\n\n[49] Sun, Y., Zheng, L., Deng, W., Wang, S.: Svdnet for pedestrian retrieval. ICCV (2017)\n\n[50] Li, W., Zhu, X., Gong, S.: Person re-identi\ufb01cation by deep joint learning of multi-loss\n\nclassi\ufb01cation. IJCAI (2017)\n\n12\n\n\f", "award": [], "sourceid": 636, "authors": [{"given_name": "Yixiao", "family_name": "Ge", "institution": "The Chinese University of Hong Kong"}, {"given_name": "Zhuowan", "family_name": "Li", "institution": "Johns Hopkins University"}, {"given_name": "Haiyu", "family_name": "Zhao", "institution": "The Chinese University of Hong Kong"}, {"given_name": "Guojun", "family_name": "Yin", "institution": "University of Science and Technology of China"}, {"given_name": "Shuai", "family_name": "Yi", "institution": "The Chinese University of Hong Kong"}, {"given_name": "Xiaogang", "family_name": "Wang", "institution": "The Chinese University of Hong Kong"}, {"given_name": "hongsheng", "family_name": "Li", "institution": "cuhk"}]}