{"title": "Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language", "book": "Advances in Neural Information Processing Systems", "page_first": 42, "page_last": 51, "abstract": "This paper addresses the problem of manipulating images using natural language description. Our task aims to semantically modify visual attributes of an object in an image according to the text describing the new visual appearance. Although existing methods synthesize images having new attributes, they do not fully preserve text-irrelevant contents of the original image. In this paper, we propose the text-adaptive generative adversarial network (TAGAN) to generate semantically manipulated images while preserving text-irrelevant contents. The key to our method is the text-adaptive discriminator that creates word level local discriminators according to input text to classify fine-grained attributes independently. With this discriminator, the generator learns to generate images where only regions that correspond to the given text is modified. Experimental results show that our method outperforms existing methods on CUB and Oxford-102 datasets, and our results were mostly preferred on a user study. Extensive analysis shows that our method is able to effectively disentangle visual attributes and produce pleasing outputs.", "full_text": "Text-Adaptive Generative Adversarial Networks:\n\nManipulating Images with Natural Language\n\nSeonghyeon Nam, Yunji Kim, and Seon Joo Kim\n\nYonsei University\n\n{shnnam,kim_yunji,seonjookim}@yonsei.ac.kr\n\nAbstract\n\nThis paper addresses the problem of manipulating images using natural language\ndescription. Our task aims to semantically modify visual attributes of an object in an\nimage according to the text describing the new visual appearance. Although existing\nmethods synthesize images having new attributes, they do not fully preserve text-\nirrelevant contents of the original image. In this paper, we propose the text-adaptive\ngenerative adversarial network (TAGAN) to generate semantically manipulated\nimages while preserving text-irrelevant contents. The key to our method is the\ntext-adaptive discriminator that creates word-level local discriminators according to\ninput text to classify \ufb01ne-grained attributes independently. With this discriminator,\nthe generator learns to generate images where only regions that correspond to the\ngiven text are modi\ufb01ed. Experimental results show that our method outperforms\nexisting methods on CUB and Oxford-102 datasets, and our results were mostly\npreferred on a user study. Extensive analysis shows that our method is able to\neffectively disentangle visual attributes and produce pleasing outputs.\n\n1\n\nIntroduction\n\nTaking pictures has become a big part of people\u2019s life ever since the emergence of smartphones as\nthe everyday device. With this trend, the demand for manipulating or editing images is growing, to\nmake photos to look better or to meet a user\u2019s need. While one can use commercially available tools\nsuch as the Photoshop to manipulate images, it is dif\ufb01cult for a non-expert user to use those tools for\nimage manipulation. Additionally, manipulating images on a mobile device would be dif\ufb01cult as the\ninterface on mobile devices is rather limited.\nAutomatic image manipulation can remedy this dif\ufb01culty by automatically changing various aspects\nof images from low-level color/texture [1, 2] to high-level semantics [3] without tedious human\noperation. With the advance of deep learning and generative models, many techniques have been\ndeveloped for image editing including the style transfer [2, 4, 5], the drawing based editing [6, 7],\nand the domain/attribute translation [8\u201310]. While manipulating images using these methods become\neasier as most operations are done automatically, editing images speci\ufb01cally per user\u2019s intention\nbecomes more dif\ufb01cult.\nThe goal of this paper is to manipulate images using natural language description. We speci\ufb01cally\nfocus on modifying visual attributes of an object, where the visual attributes are characterized by\nthe color and the texture of an object. Fig. 1 illustrates our problem. In the examples, the colors of\ndifferent parts of an bird are changed according to the given text description. We believe that it is a\nmore intuitive way to manipulate images and would also be ideal for mobile devices.\nConditional deep generative models have shown great potential in multi-modal generative modeling\nof image and text. However, most existing studies concentrate on the text-to-image synthesis [11\u2013\n14], which generates images from text descriptions without the original image. Only few works\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fThis particular bird with a red head and breast\nand features grey wings.\n\nThis small bird has a blue crown and white\nbelly.\n\nOriginal\n\n[11]\n\n[15]\n\nOurs\n\nFigure 1: Examples of image manipulation using natural language description. Existing methods\nproduce reasonable results, but fail to preserve text-irrelevant contents such as the background of\nthe original image. In comparison, our method accurately manipulates images according to the text\nwhile preserving text-irrelevant contents.\n\naddress similar problem as ours [11, 15]. In [11, 15], both approaches train generative adversarial\nnetworks (GANs) using the encoded image and the sentence vector pretrained for visual-semantic\nsimilarity [16, 17]. As shown in Fig. 1, these methods synthesize a new image according to the text\nwhile preserving the image layout and the pose of the object to some extent. However, they are likely\nto generate a new image conditioned on the pose of the original image instead of modifying only the\nparts that are described in the text. This is mainly because the sentence-conditional discriminator\nprovides the generator with coarse training feedback, which prevents the generator from disentangling\ndifferent regions of the image.\nTo overcome this limitation, we propose a novel text-conditioned visual attribute manipulation method\ncalled the Text-Adaptive Generative Adversarial Network (TAGAN). The key idea is to split a single\nsentence-level discriminator into a number of word-level discriminators so that each word-level\ndiscriminator is attached to a speci\ufb01c type of visual attribute. With this approach, the discriminator\nis able to provide \ufb01ne-grained training feedback to the generator to change only the speci\ufb01c visual\nattribute. To this end, we introduce a text-adaptive discriminator that consists of word-level local\ndiscriminators. It classi\ufb01es an image as real or fake by aggregating all word-level matching scores\nfrom the local discriminators with text attention. With the text-adaptive discriminator, our generator\nlearns to change the parts of the image while preserving other contents. To the best of our knowledge,\nnone of the previous works learn this level of \ufb01ne-grained discriminators using text.\nExperimental results show that our method outperforms existing methods on CUB [18] and Oxford-\n102 [19] datasets both quantitatively and qualitatively. We conducted a user study for human\nevaluation, and our results were preferred the most by the participants. In addition, our extensive\nanalysis shows that the TAGAN effectively disentangles visual attributes and accurately manipulates\nimages according to the text, while preserving text-irrelevant contents in the original image.\n\n2 Related Work\n\nWith the success of deep generative models like variational auto-encoders (VAEs) [20] and GANs [21],\nimage generation has been widely studied, and our work is particularly related to conditional image\ngeneration methods [22, 23].\nThere have been many attempts to generate images using conditional variables. cGAN [23] generates\nMNIST digit images from labels. Attribute2Image [24] produces more complex images such as faces\nand birds from visual attributes by disentangling the foreground and the background using cVAE.\nInfoGAN [25] learns interpretable latent variables in an unsupervised manner for conditional image\ngeneration. However, these works address the problem of generating new images from pure noise\nvectors. In comparison, our work focuses on modifying the given image according to the conditional\ninformation.\nAnother line of research is the conditional image manipulation. Zhu et al. [6] introduced modifying\nthe color and the shape of images from user\u2019s sketch by manipulating latent vectors. Similarly,\nBrock et al. [7] also proposed to edit face images with user scribbles. FaderNetworks [8] learns the\nattribute-invariant latent space by preventing correct prediction of attributes for face manipulation.\n\n2\n\n\f(a) GAN structure\n\n(b) Text-adaptive discriminator\n\nFigure 2: The proposed GAN structure. (a) shows the overall GAN architecture and (b) depicts our\ntext-adaptive discriminator. In (b), the attention and the layer-wise weight are omitted for simplicity.\n\nAdditionally, image-to-image domain translation methods have been introduced [9, 10]. However,\nthese methods cannot be directly applied to our task since they do not have mechanisms to process\nthe sequential textual data as inputs. In this paper, we focus on the multi-modal learning of both\nimage and natural language description.\nOur work is also closely related to text-to-image synthesis methods. Reed et al. [11] \ufb01rst proposed\nto generate a 64\u00d764 natural image from a sentence vector using the DCGAN. StackGAN [12, 26]\nproduces high-resolution images progressively by stacking multiple GANs. Zhang et al. [14]\npresented a hierarchically nested adversarial loss to regularize the generator to improve image details.\nAttnGAN [13] incorporated an attention mechanism into [26] and enhanced \ufb01ne-grained details with\na pretrained visual-semantic similarity model. While it uses a word-level visual-semantic attention\nsimilar to our method, it fundamentally relies on a sentence vector to generate images. All of the\nmethods produce various high-quality images from text, but their main focus is in generating a whole\nnew image, not on manipulating a speci\ufb01c part of input image using text.\nIn [11], the authors brie\ufb02y showed an image manipulation method as an application by learning an\nauxiliary image encoder to reconstruct a latent noise vector that contains text-irrelevant information of\nimage. Dong et al.\u2019s work [15] also addressed the problem of manipulating images with semantically\ndifferent text descriptions. Their method trains a conditional GAN to synthesize a manipulated\nversion of image given an original image and a target text description. However, both methods are\nlimited in performance due to the simple sentence modeling discussed previously. In this paper, we\ntackle the problem in a different manner by making a \ufb01ne-grained discriminator to learn to disentangle\nvisual attributes from the image and the text.\n\n3 Text-Adaptive Generative Adversarial Networks\n\nLet x, t, \u02c6t denote an image, a positive text where the description matches the image, and a negative\ntext that does not correctly describe the image, respectively. Given an image x and a target negative\ntext \u02c6t, our task is to semantically manipulate x according to \u02c6t so that the visual attributes of the\nmanipulated image \u02c6y match the description of \u02c6t while preserving other information. We use GAN as\nour framework, in which the generator is trained to produce \u02c6y = G(x, \u02c6t). Similar to text-to-image\nGANs [11, 15], we train our GAN to generate a realistic image that matches the conditional text\nsemantically. In the following, we describe the TAGAN in detail.\n\nGenerator The generator is an encoder-decoder network as shown in Fig. 2 (a)1. It \ufb01rst encodes an\ninput image to a feature representation, then transforms it to a semantically manipulated represen-\ntation according to the features of the given conditional text. For the text representation, we use a\nbidirectional RNN to encode the whole text. Unlike existing works [11, 15], we train the RNN from\nscratch, without pretraining. Additionally, we adopt the conditioning augmentation method [12] for\nsmooth text representation and the diversity of generated outputs. As shown in Fig. 2 (a), manipulated\ncontents are generated through several residual blocks with a skip connection. However, this process\n\n1Note that generating high-resolution images is not the main focus of our paper, therefore we do not use\n\nmulti-stage generation [12\u201314, 26] in our generator.\n\n3\n\nvLDEncoderTextEncoderResidualBlocksDecoderTextEncoderText-AdaptiveDiscriminatorReal/FakewwwvUnconditional lossConditional loss!\"ImageEncoderTextEncoderGAP!\u22121!+1\"\u22121\"+1EncoderGeneratorDiscriminator*vLDLDLDLDLDLDLDLDLD:LocalDiscriminatorvLDEncoderTextEncoderResidualBlocksDecoderTextEncoderText-AdaptiveDiscriminatorReal/FakewwwvUnconditional lossConditional loss!\"ImageEncoderTextEncoderGAP!\u22121!+1\"\u22121\"+1EncoderGeneratorDiscriminator*vLDLDLDLDLDLDLDLDLD:LocalDiscriminator\fmay generate a new background and other contents that are not described in the text. Therefore,\nwe use the reconstruction loss [27] when a positive text is given, which enforces the generator to\nreconstruct the text-irrelevant contents from the input image instead of generating new contents:\n\n(1)\nHowever, learning invariant representation is still dif\ufb01cult unless the discriminator provides useful\nfeedback for disentangling visual attributes. To cope with it, we propose a text-adaptive discriminator.\n\nLrec = (cid:107)x \u2212 G(x, t)(cid:107).\n\nText-adaptive discriminator The motivation of the text-adaptive discriminator is to provide the\ngenerator with a speci\ufb01ed training signal to generate certain visual attributes. To achieve this, the\ndiscriminator classi\ufb01es each attribute independently using word-level local discriminators. By doing\nso, the generator receives feedback from each local discriminator for each visual attribute.\nFig. 2 (b) shows the structure of the text-adaptive discriminator. Similar to the generator, the\ndiscriminator is trained with its own text encoder. For each word vector wi, i-th output from the text\nencoder, we create 1D sigmoid local discriminator fwi, which determines whether a visual attribute\nrelated to wi exists in the image. Formally, fwi is described as:\n\nfwi(v) = \u03c3(W(wi) \u00b7 v + b(wi)),\n\n(2)\nwhere W(wi) and b(wi) are the weight and the bias dependent on wi. v is an 1D image vector\ncomputed by applying global average pooling to the feature map of the image encoder.\nWith the local discriminators, the \ufb01nal classi\ufb01cation decision is made by adding word-level attentions\nto reduce the impact of less important words to the \ufb01nal score. Our attention is a softmax values\nacross T words, which is computed by:\n\n\u03b1i =\n\nexp(uT wi)\ni exp(uT wi)\n\n,\n\n(3)\n\nwhere u is a temporal average of wi. The \ufb01nal score is computed according to the following\nformulation:\n\nD(x, t) =\n\n[fwi(v)]\u03b1i.\n\n(4)\n\n(cid:80)\n\nT(cid:89)\n\ni=1\n\nT(cid:89)\n\n[\ni=1\n\n(cid:88)\n\nj\n\nOur approach has several advantages over existing methods. First, instead of learning sentence-level\ncorrespondence, we train the discriminator to \ufb01rst identify individual attributes in a sentence, then to\n\ufb01nd existence of each attribute in the image. Also, our method can be easily trained by the cross-\nentropy loss without the ranking loss [11] due to our multiplicative aggregation of scores. Although\nour method shares similar spirit to the attention-driven matching score in [13], our method does not\nuse explicit spatial attention on images and does not need any hand-tuned hyperparameters. Also, the\nmost important difference is that we do not use our text-adaptive discriminator as an auxiliary loss.\nInstead, we train the text-adaptive discriminator as our core GAN framework.\nWe additionally consider multi-scale image features to make some attribute detectors to focus on small-\nscale features and others to focus on large-scale features. Therefore, our conditional discriminator is\nrewritten as:\n\nD(x, t) =\n\n\u03b2ijfwi,j(vj)]\u03b1i,\n\n(5)\n\nwhere vj is the image vector of j-th layer, and \u03b2ij is a softmax weight that determines the importance\nof the layer j for each word wi.\n\nGAN objective The \ufb01nal GAN objective consists of unconditional adversarial losses for D(x),\ntext-conditional losses for D(x, \u02c6t), and a reconstruction loss as shown in Fig. 2. The discriminator\nhas one image encoder and two branches of classi\ufb01er on the top of the encoder to compute both the\nunconditional and the conditional losses. Our network is trained by alternatively minimizing both the\ndiscriminator and the generator objectives described as:\n\nLD = E\n+ E\n\nx,t,\u02c6t\u223cpdata\nx,\u02c6t\u223cpdata\n\n[log D(x) + \u03bb1(log D(x, t) + log (1 \u2212 D(x, \u02c6t)))]\n[log (1 \u2212 D(G(x, \u02c6t)))],\n\n(6)\n\n4\n\n\fLG = E\n\nx,\u02c6t\u223cpdata\n\n[log D(x) + \u03bb1 log D(G(x, \u02c6t), \u02c6t)] + \u03bb2Lrec,\n\n(7)\nwhere \u03bb1 and \u03bb2 control the importance of additional losses, and \u02c6t is randomly sampled from a dataset\nregardless of x. Note that we do not penalize generated outputs using the conditional discriminator\nin Eq. (6) due to instability of training. In our experiment, our objective was enough to produce real\nimages having manipulated attributes.\n\nImplementation We implemented our method using PyTorch. For the generator, we adopt the\narchitecture of the generator from [15] to encode and decode 128\u00d7128 images. We additionally\nencode a text using a bidirectional GRU and the pretrained fastText [28] word vectors. In the\ndiscriminator, we use conv3, conv4, and conv5 for the local discriminators.2 We trained our\nnetwork 600 epochs using Adam optimizer [29] with the learning rate of 0.0002, the momentum of\n0.5, and the batch size of 64. Also, we decreased the learning rate by 0.5 for every 100 epochs. For\ndata augmentation, we used random cropping, \ufb02ipping, and rotation. We resized images to 136\u00d7136\nand randomly cropped 128\u00d7128 patches. The random rotation ranged from -10 to 10 degrees. We set\n\u03bb1 and \u03bb2 to 10 and 2 respectively considering both the visual quality and the training stability.\n\n4 Experiments\n\nExperimental setup We evaluated our method on CUB dataset [18] and Oxford-102 dataset [19],\nwhich contain 11,788 bird images of 200 categories and 8,189 \ufb02ower images of 102 categories,\nrespectively. Also, we used natural language captions of both datasets provided in [11], where 10\nindividual sentences were annotated for each image. For evaluation, we compared our method to\ntwo baseline methods: SISGAN [15] and AttnGAN [13]. In particular, we adapted AttnGAN for\nour task based on the source code provided by the authors in order to compare our method with\nthe state-of-the-art text-to-image synthesis method. Speci\ufb01cally, we added an image encoder and a\nreconstruction loss to the original network. For the encoder, we used the same encoder used in our\nmethod. We \ufb01ne-tuned the weight of the reconstruction loss to produce the best results. For SISGAN,\nwe reproduced the same network based on the original paper and the code provided by the authors.\n\nQuantitative results We \ufb01rst conducted a quantitative evaluation of the two baseline methods and\nour method. The quality of synthesized images is a subjective matter, and it is dif\ufb01cult to compare\ndifferent methods using an objective metric. The inception score [30] is also not proper for this\nproblem since it only measures the quality of an image, not the relevance to a text. Therefore, we\nconducted a human evaluation on Amazon Mechanical Turk.\nFor the user study, we randomly selected 10 images and 10 texts from the test set, and produced 200\noutputs from the two datasets for each method. As the spatial resolutions for different methods are\ndifferent, we resized all output images to 64\u00d764 to prevent the users from evaluating the images\nbased on the sharpness. Then, we asked workers to rank three results after looking at both input\nimage/text and outputs. We asked the workers to evaluate images based on the following two criteria:\n(i) whether the visual attributes (colors, textures) of the manipulated image match the text, and the\nbackground irrelevant to the text is preserved, and (ii) whether the manipulated image looks natural,\nand visually pleasing. As a result, we collected a total of 4,000 samples from 20 workers.\nTable 1 shows the results of the user study. In the table, each criterion is referred as (i) Accuracy and\n(ii) Naturalness. The values shown in the table are average ranking values. For both the accuracy and\nthe naturalness, our results were most preferred by the workers. To further verify it, we conducted\nchi-square test on the count values and found that p-value is p < 10\u221230, which indicates that our\nmethod signi\ufb01cantly outperforms baseline methods on this user study. In essence, our method\ngenerates realistic images, where the visual attributes are manipulated accurately while preserving\ntext-irrelevant contents of the original image. The performance of two baseline methods were similar,\nbut AttnGAN was slightly more preferred over SISGAN.\nTo further compare the quality of the content preservation, we also computed L2 reconstruction error\nby forwarding images with positive texts. Due to the randomness, we average 50 trials for each\nmethod. In the table, our method shows the lowest reconstruction error, which indicates that our\nmethod preserves the content of original image better.\n\n2For the detail of the network architecture, please refer to the supplementary material.\n\n5\n\n\fTable 1: Quantitative comparison. Accuracy and Naturalness were evaluated by users, and the values\nindicate the average ranking. L2 reconstruction error was additionally compared.\n\nCUB\n\nOxford-102\n\nMethod\nSISGAN [15]\nAttnGAN [13]\nOurs\n\nAccuracy Naturalness L2 error Accuracy Naturalness L2 error\n\n2.33\n2.19\n1.49\n\n2.34\n2.11\n1.56\n\n0.30\n0.25\n0.11\n\n2.67\n2.21\n1.52\n\n2.28\n2.10\n1.62\n\n0.29\n0.32\n0.11\n\nOriginal\n\nThis bird has wings that are blue\nand has a white belly.\n\nA small bird with white base and\nblack stripes throughout its belly,\nhead, and feathers.\n\nOriginal\n\nThe petals of the \ufb02ower have yellow\nand red stripes.\n\nThis \ufb02ower has petals of pink and\nwhite color with yellow stamens.\n\nFigure 3: Qualitative results of our method on CUB and Oxford-102 datasets.\n\nThis is a black bird with gray\nand white wings and a bright\nyellow belly and chest.\n\nThis \ufb02ower has petals that are\nwhite and has patches of yellow.\n\nOriginal\n\nSISGAN [15]\n\nAttnGAN [13]\n\nOurs\n\nThis pink \ufb02ower has long and oval petals and a large yellow stamen.\n\nFigure 4: Qualitative comparison of three methods. In most cases, our method outperforms baseline\nmethods qualitatively. The rightmost column shows a failure case using our method.\n\n6\n\n\fThis bird is brown with black wings and tail and\nlong legs.\n\nThis \ufb02ower has petals that are yellow and are very\nstringy.\n\nFigure 5: Visualization of the text-adaptive discriminator. From top to bottom, the top-3 word\nattentions are shown. From left to right, the saliency maps of 3 layer-wise local discriminators are\n\nvisualized. Each fractional number is \u03b2ij. Note that(cid:80)\n\nj \u03b2ij = 1.\n\nQualitative results Fig. 3 shows qualitative results of our method on CUB and Oxford-102. In\nmost cases, the visual attributes of images are accurately changed according to the text. Note that\nsome attributes only exist in a speci\ufb01c class of birds or \ufb02owers. For example, the petals of \u201cyellow\nand red stripes\u201d in Fig. 3 refers to the \ufb02ower gazania. As can be seen from the results, our method\nis able to generate textures on images of different classes. It indicates that our network effectively\ndisentangles visual attributes invariant to pose, shape, and background.\nFig. 4 shows comparisons with the baselines methods. Both baselines usually preserve the pose and\nthe layout of the original images. However, the methods are likely to generate a new image based\non the original layout and do not preserve the contents that are not relevant to the text. In some\ncases, the methods generate similar images because the sentence is highly correlated to a particular\nclass of birds or \ufb02owers. On the other hand, our method preserves the text-irrelevant contents while\ntransferring visual attributes accurately.\nOur method may sometimes generate failure cases. As shown in the rightmost column in Fig. 4, our\nmethod fails to generate the exact shape described in the text on the original image. It is mainly due\nto the trade-off between generating new contents and preserving original contents. While we can\ncontrol it by adjusting the power of reconstruction loss, some examples need more sophisticated\ncontrol. Similar to [27, 8], our potential extension is to incorporate an explicit control variable into\nour generator to decide the trade-off example by example.\n\nComponent analysis As described in Sec. 3, our text-adaptive discriminator learns both word-level\nattention and local discriminators. In Fig. 5, we visualized class activation maps (CAMs) [31] to\nshow the saliency map that a local discriminator in Eq. (2) attends to. 3 The \ufb01gure shows the top-3\nword attentions vertically and 3 layer-wise weights horizontally. We can see that the discriminator\nlearns distinguishable visual attributes in the text and their corresponding classi\ufb01ers. Additionally, the\nnetwork adaptively selects different levels of layers to detect coarse or \ufb01ne-grained attributes for each\nword. To show the effectiveness of it, we also ran an ablation study as shown in Fig. 6. When we use\nonly conv5, the network usually learns the global object colors. The network generates more details\nwhen trained using conv3, but the quality is not satisfying as various scales of visual attributes are\nhard to learn within a single scale layer. On the other hand, our multi-scale network (conv3,4,5)\neffectively learns to generate both coarse and \ufb01ne-grained visual attributes by separating different\nscales of the attributes in the latent space.\n\n3Since the grid of conv5 is coarse (4\u00d74), the salient region may not exactly match to an object region.\n\n7\n\nOriginalOutputwingsbrownblackconv5conv4conv3FineCoarse0.250.280.470.040.010.950.100.100.80++= 1++= 1++= 1OriginalOutputyellowandflower0.050.270.680.030.140.830.040.270.69++= 1++= 1++= 1conv5conv4conv3FineCoarse\fconv5\n\nconv3\n\nconv3,4,5\n\nThis \ufb02ower has petals that are yellow\nand are very stringy.\n\nThis \ufb02ower is purple in color, and has\npetals that are very skinny.\n\nFigure 6: Ablation study of multi-scale layers.\n\nTable 2: Quantitative comparison of\nmulti-modal retrieval task on CUB\ndataset. Our method is competitive\nto the state-of-the-art method [32]\n\nImage-Text\n\nText-Image\nTop-1 Acc (%) AP@50 (%)\n\n[34]\n[32]\n[13]\nOurs\n\n56.8\n61.5\n55.1\n61.3\n\n48.7\n57.6\n51.0\n62.8\n\nLeft: A small brightly colored yellow bird with a black crown.\n\nRight: This is a black and white shaded bird with a very small beak.\n\nFigure 7: Sentence interpolation results. Our generator smoothly generates new visual attributes\nwithout loosing original image.\n\nSince our method shares similar word attention paradigm with image-text matching methods [32, 13],\nwe further provide a comparison on this task. We trained our text-adaptive classi\ufb01er on top of the\nInception-v3 network [33] to compute the image-text similarity scores. Similar to [34], we compared\nthe top-1 image-to-text retrieval accuracy (Top-1 Acc), and the percentage of the matching images\nin the top-50 text-to-image retrieval results (AP@50) on CUB dataset as shown in Table 2. Our\nmethod outperforms the auxiliary matching network in AttnGAN [13], and is competitive with the\nstate-of-the-art matching method [32]. We attribute this result to the word-level local discriminators,\nand the use of multi-scale layers. In addition to performance, our method is more convenient to train\nsince it does not need hard sample mining [32], and data-dependent hyperparameters [13].\nLastly, we conducted a text interpolation experiment for the generator to analyze the inference\nability. We interpolated a text embedding vector in the generator using two different sentences, and\ngenerated smoothly changing images. As shown in Fig. 7, our method generates reasonable images\nof interpolated text while preserving the contents of original image. This indicates that our generator\neffectively learns invariant latent variables and is able to infer with respect to various texts without\nmemorizing them.\n\n5 Conclusion\n\nIn this paper, we proposed a text-adaptive generative adversarial network to semantically manipulate\nimages using natural language description. Our text-adaptive discriminator disentangles \ufb01ne-grained\nvisual attributes in the text using word-level local discriminators created on the \ufb02y according to\nthe text. By doing so, our generator learns to generate particular visual attributes while preserving\nirrelevant contents in the original image. Experimental results show that our method outperforms\nexisting methods both quantitatively and qualitatively.\n\nAcknowledgement This work was supported by Global Ph.D. Fellowship Program through the National\nResearch Foundation of Korea (NRF) funded by the Ministry of Education (NRF2015H1A2A1033924), Institute\nfor Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIP)\n(2018-0-01858, Video Manipulation and Language-based Image Editing Technique for Detecting Manipulated\nImage/Video), and the ICT R&D program of MSIT/IITP (2017-0-01772, Development of QA systems for Video\nStory Understanding to pass the Video Turing Test).\n\n8\n\n\fReferences\n[1] R. Zhang, P. Isola, and A. A. Efros, \u201cColorful image colorization,\u201d in ECCV, 2016.\n\n[2] L. A. Gatys, A. S. Ecker, and M. Bethge, \u201cImage style transfer using convolutional neural networks,\u201d in\n\nCVPR, pp. 2414\u20132423, 2016.\n\n[3] X. Liang, H. Zhang, and E. P. Xing, \u201cGenerative semantic manipulation with contrasting gan,\u201d arXiv\n\npreprint arXiv:1708.00315, 2017.\n\n[4] J. Johnson, A. Alahi, and L. Fei-Fei, \u201cPerceptual losses for real-time style transfer and super-resolution,\u201d\n\nin ECCV, pp. 694\u2013711, 2016.\n\n[5] F. Luan, S. Paris, E. Shechtman, and K. Bala, \u201cDeep photo style transfer,\u201d in CVPR, pp. 6997\u20137005, 2017.\n\n[6] J.-Y. Zhu, P. Kr\u00e4henb\u00fchl, E. Shechtman, and A. A. Efros, \u201cGenerative visual manipulation on the natural\n\nimage manifold,\u201d in ECCV, 2016.\n\n[7] A. Brock, T. Lim, J. M. Ritchie, and N. Weston, \u201cNeural photo editing with introspective adversarial\n\nnetworks,\u201d arXiv preprint arXiv:1609.07093, 2016.\n\n[8] G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. Denoyer, et al., \u201cFader networks: Manipulating\n\nimages by sliding attributes,\u201d in NeurIPS, pp. 5969\u20135978, 2017.\n\n[9] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, \u201cImage-to-image translation with conditional adversarial\n\nnetworks,\u201d in CVPR, pp. 1125\u20131134, 2017.\n\n[10] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, \u201cUnpaired image-to-image translation using cycle-consistent\n\nadversarial networks,\u201d in ECCV, pp. 2223\u20132232, 2017.\n\n[11] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, \u201cGenerative adversarial text-to-image\n\nsynthesis,\u201d in ICML, 2016.\n\n[12] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas, \u201cStackgan: Text to photo-realistic\n\nimage synthesis with stacked generative adversarial networks,\u201d in ICCV, 2017.\n\n[13] Q. H. H. Z. Z. G. X. H. X. H. Tao Xu, Pengchuan Zhang, \u201cAttngan: Fine-grained text to image generation\n\nwith attentional generative adversarial networks,\u201d in CVPR, 2018.\n\n[14] Z. Zhang, Y. Xie, and L. Yang, \u201cPhotographic text-to-image synthesis with a hierarchically-nested adver-\n\nsarial network,\u201d in CVPR, 2018.\n\n[15] H. Dong, S. Yu, C. Wu, and Y. Guo, \u201cSemantic image synthesis via adversarial learning,\u201d in ICCV, Oct\n\n2017.\n\n[16] R. Kiros, R. Salakhutdinov, and R. S. Zemel, \u201cUnifying visual-semantic embeddings with multimodal\n\nneural language models,\u201d arXiv preprint arXiv:1411.2539, 2014.\n\n[17] S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee, \u201cLearning what and where to draw,\u201d in\n\nNeurIPS, pp. 217\u2013225, 2016.\n\n[18] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, \u201cThe Caltech-UCSD Birds-200-2011 Dataset,\u201d\n\nTech. Rep. CNS-TR-2011-001, California Institute of Technology, 2011.\n\n[19] M.-E. Nilsback and A. Zisserman, \u201cAutomated \ufb02ower classi\ufb01cation over a large number of classes,\u201d in\n\nICCVGIP, Dec 2008.\n\n[20] D. P. Kingma and M. Welling, \u201cAuto-encoding variational bayes,\u201d arXiv preprint arXiv:1312.6114, 2013.\n\n[21] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio,\n\n\u201cGenerative adversarial nets,\u201d in NeurIPS, pp. 2672\u20132680, 2014.\n\n[22] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, \u201cSemi-supervised learning with deep generative\n\nmodels,\u201d in NeurIPS, pp. 3581\u20133589, 2014.\n\n[23] M. Mirza and S. Osindero, \u201cConditional generative adversarial nets,\u201d arXiv preprint arXiv:1411.1784,\n\n2014.\n\n[24] X. Yan, J. Yang, K. Sohn, and H. Lee, \u201cAttribute2image: Conditional image generation from visual\n\nattributes,\u201d in ECCV, pp. 776\u2013791, 2016.\n\n9\n\n\f[25] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, \u201cInfogan: Interpretable\nrepresentation learning by information maximizing generative adversarial nets,\u201d in NeurIPS, pp. 2172\u2013\n2180, 2016.\n\n[26] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas, \u201cStackgan++: Realistic image\n\nsynthesis with stacked generative adversarial networks,\u201d arXiv: 1710.10916, 2017.\n\n[27] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen, \u201cArbitrary facial attribute editing: Only change what you\n\nwant,\u201d arXiv preprint arXiv:1711.10678, 2017.\n\n[28] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, \u201cEnriching word vectors with subword information,\u201d\n\narXiv preprint arXiv:1607.04606, 2016.\n\n[29] D. P. Kingma and J. Ba, \u201cAdam: A method for stochastic optimization,\u201d arXiv preprint arXiv:1412.6980,\n\n2014.\n\n[30] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, \u201cImproved techniques for\n\ntraining gans,\u201d in NeurIPS, pp. 2234\u20132242, 2016.\n\n[31] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, \u201cLearning deep features for discriminative\n\nlocalization,\u201d in CVPR, pp. 2921\u20132929, 2016.\n\n[32] S. Li, T. Xiao, H. Li, W. Yang, and X. Wang, \u201cIdentity-aware textual-visual matching with latent co-\n\nattention,\u201d in CVPR, pp. 1890\u20131899, 2017.\n\n[33] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, \u201cRethinking the inception architecture for\n\ncomputer vision,\u201d in CVPR, pp. 2818\u20132826, 2016.\n\n[34] S. Reed, Z. Akata, H. Lee, and B. Schiele, \u201cLearning deep representations of \ufb01ne-grained visual descrip-\n\ntions,\u201d in CVPR, pp. 49\u201358, 2016.\n\n10\n\n\f", "award": [], "sourceid": 40, "authors": [{"given_name": "Seonghyeon", "family_name": "Nam", "institution": "Yonsei University"}, {"given_name": "Yunji", "family_name": "Kim", "institution": "Yonsei University"}, {"given_name": "Seon Joo", "family_name": "Kim", "institution": "Yonsei University"}]}