{"title": "Turbo Learning for CaptionBot and DrawingBot", "book": "Advances in Neural Information Processing Systems", "page_first": 6455, "page_last": 6465, "abstract": "We study in this paper the problems of both image captioning and\ntext-to-image generation, and present a novel turbo learning\napproach to jointly training an image-to-text generator (a.k.a.\nCaptionBot) and a text-to-image generator (a.k.a. DrawingBot). The\nkey idea behind the joint training is that image-to-text\ngeneration and text-to-image generation as dual problems can form\na closed loop to provide informative feedback to each other. Based\non such feedback, we introduce a new loss metric by comparing the\noriginal input with the output produced by the closed loop. In\naddition to the old loss metrics used in CaptionBot and\nDrawingBot, this extra loss metric makes the jointly trained\nCaptionBot and DrawingBot better than the separately trained\nCaptionBot and DrawingBot. Furthermore, the turbo-learning\napproach enables semi-supervised learning since the closed loop\ncan provide peudo-labels for unlabeled samples. Experimental\nresults on the COCO dataset demonstrate that the proposed turbo\nlearning can significantly improve the performance of both\nCaptionBot and DrawingBot by a large margin.", "full_text": "Turbo Learning for CaptionBot and DrawingBot\n\nQiuyuan Huang\nMicrosoft Research\nRedmond, WA, USA\n\nPengchuan Zhang\nMicrosoft Research\nRedmond, WA, USA\n\nqihua@microsoft.com\n\npenzhan@microsoft.com\n\nDapeng Wu\n\nUniversity of Florida\nGainesville, FL, USA\n\ndpwu@ieee.org\n\nLei Zhang\n\nMicrosoft Research\nRedmond, WA, USA\n\nleizhang@microsoft.com\n\nAbstract\n\nWe study in this paper the problems of both image captioning and text-to-image\ngeneration, and present a novel turbo learning approach to jointly training an\nimage-to-text generator (a.k.a. CaptionBot) and a text-to-image generator (a.k.a.\nDrawingBot). The key idea behind the joint training is that image-to-text generation\nand text-to-image generation as dual problems can form a closed loop to provide\ninformative feedback to each other. Based on such feedback, we introduce a new\nloss metric by comparing the original input with the output produced by the closed\nloop. In addition to the old loss metrics used in CaptionBot and DrawingBot,\nthis extra loss metric makes the jointly trained CaptionBot and DrawingBot better\nthan the separately trained CaptionBot and DrawingBot. Furthermore, the turbo-\nlearning approach enables semi-supervised learning since the closed loop can\nprovide pseudo-labels for unlabeled samples. Experimental results on the COCO\ndataset demonstrate that the proposed turbo learning can signi\ufb01cantly improve the\nperformance of both CaptionBot and DrawingBot by a large margin.\n\n1\n\nIntroduction\n\nDue to the breakthrough of deep learning, recent years have witnessed great progresses in both\ncomputer vision and natural language processing. As a result, two fundamental problems \u2013 image\ncaptioning and text-to-image generation \u2013 that requires cross modality understanding have also\nbeen intensely studied in the past few years. Image captioning (a.k.a. CaptionBot) is to generate\na meaningful caption for any given input image, whereas text-to-image (a.k.a. DrawingBot) is to\ngenerate a realistic image for any input sentence. Regardless of their different implementations, it\nis interesting to see that the two problems can be regarded as dual problems as both can take each\nother\u2019s output as input. However, despite their duality in terms of their input and output forms, the\ntwo problems were largely studied separately in the past, leaving a signi\ufb01cant room for improvement.\nTo leverage their duality, for the \ufb01rst time, this paper proposes a turbo-learning approach, which can\njointly train a CaptionBot and a DrawingBot together in a way similar to an engine turbo-charger,\nwhich feeds the output back to the input to reuse the exhaust gas for increased back pressure and\nbetter engine ef\ufb01ciency. The key idea of the proposed turbo-learning approach is that an image-to-text\ngenerator and a text-to-image generator can form a closed loop and generate informative feedback\nsignals to each other. In this way, we can obtain a new loss metric (for updating the neural network\nweights during training) by comparing the original input data with the output data produced by the\nwhole closed loop, in addition to the old loss metric which measures the difference between the\noutput and the ground-truth of the CaptionBot or the DrawingBot. This extra loss metric effectively\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fleads to improved performance of both of the jointly trained CaptionBot and DrawingBot compared\nwith their separately trained counterparts.\nTo jointly train a CaptionBot and a DrawingBot, we utilize the state-of-the-art long short-term\nmemory (LSTM) image captioner [1] and text-to-image generation algorithm [2] as building blocks\nand use stochastic gradient descent to iteratively learn the network parameters of both blocks. More\nspeci\ufb01cally, as illustrated in Fig. 1, training in each iteration consists of two steps. In Step 1, the\nDrawingBot serves as the primal module while the CaptionBot serves as the dual module. In Step\n2, the CaptionBot serves as the primal module while the DrawingBot serves as the dual module. In\neach step, the primal module takes an input and produces/forwards its output to its dual module, and\nthe dual module further feeds back its output to the input of the primal module. The weights of the\nDrawingBot and the CaptionBot are updated simultaneously in each step by using the loss function of\ntwo pairs \u2013 (gold image vs. generated image) and (gold sentence vs. generated sentence) to calculate\nthe gradient.\nClosing the loop between CaptionBot and DrawingBot also brings another merit: it enables semi-\nsupervised learning since the closed loop can provide pseudo-labels for unlabeled samples. For image\ncaptioning, such a semi-supervised learning capability is particularly useful since human annotation of\nimages is very costly. Note that between the two bots in each step we also have a constraint to ensure\nthe sentence (or image) generated from the primal module is natural (or realistic). The constraint\ncould be a Generative Adversarial Network (GAN) loss if we only have unsupervised training data,\ni.e., images and sentences without correspondence. In this work, we use the semi-supervised setting\nfor simplicity and use the ground truth sentence (or image) to supervise/constrain the sentence (or\nimage) generated from the primal module, while letting the unlabeled data to pass through the two\nmodules to improve the feature learning. This loss term effectively prevents the model from learning\ntrivial identity mappings for both the CaptionBot and DrawingBot.\nWe conducted experimental evaluation on the COCO dataset. Experimental results show that the\nturbo learning approach signi\ufb01cantly improves the performance of CaptionBot and DrawingBot. For\nexample, under supervised learning, the CaptionBot is improved by 13% in BLEU-4 metric, and the\nDrawingBot is improved by 4.3% in Inception score; under semi-supervised learning, the CaptionBot\nis improved by 77% in BLEU-4 metric.\nThe rest of the paper is organized as follows. Section 2 discusses the related work. In Section 3, we\nintroduce our proposed turbo learning approach and present its theoretical underpinning. Section 4\ndescribes turbo learning for supervised training and presents experimental results. In Section 5, we\npresent our semi-supervised learning for CaptionBot and experimental results. Finally, Section 6\nconcludes the paper.\n\n2 Related work\n\nMost existing image captioning systems exploit end-to-end deep learning with a convolutional neural\nnetwork (CNN) image-analysis front end producing a distributed representation that is then used to\ndrive a natural-language generation process, typically using a recurrent neural network (RNN) or\nLSTM [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14].\nGenerating photo-realistic images from text descriptions has been an active research area in recent\nyears. There are different approaches working toward this grand goal, such as variational infer-\nence [15, 16], approximate Langevin process [17], conditional PixelCNN via maximal likelihood\nestimation [18, 17], and conditional generative adversarial networks [19, 20, 21, 22]. Compared with\nother approaches, generative adversarial networks (GANs) [23] have shown great performance for\ngenerating sharper samples [24, 25, 26, 27, 28]. AttnGAN [2] proposes an attentional multi-stage\ngenerator that can synthesize \ufb01ne-grained details at different image regions by paying attentions to\nthe relevant words in the text description, which achieves the state-of-the-art performance of the\ntext-to-image synthesis task on MS-COCO dataset.\nDifferent from previous works which study the problems of image captioning and text-to-image\ngeneration separately, this work aims at developing a turbo learning approach to jointly training\nCaptionBot and DrawingBot. Our proposed turbo learning is similar to dual learning [29], which is\napplied to automatic machine translation. Under dual learning, any machine translation task has a\ndual task, e.g., English-to-French translation (primal) versus French-to-English translation (dual); the\n\n2\n\n\fprimal and dual tasks can form a closed loop, and generate feedback signals to train the translation\nmodels, even if without the involvement of a human labeler. In the dual-learning mechanism, one\nagent represents the model for the primal task and the other agent represents the model for the dual\ntask, then the two agents teach each other through a reinforcement learning process. Dual-learning is\nsolely developed for NLP. In contrast, our turbo learning is applied to both NLP and computer vision,\nwhich is more challenging.\n\n3 Turbo learning structure\n\nIn this section, we present the turbo learning approach for training CaptionBot and DrawingBot\njointly.\n\n3.1 Turbo learning for CaptionBot and DrawingBot\n\nFigure 1: The proposed turbo butter\ufb02y structure for training CaptionBot and DrawingBot jointly\n\nFig. 1 shows the turbo architecture for the proposed joint training of CaptionBot and DrawingBot.\nAs shown in the left hand side of Fig. 1, for a given gold sentence S\u2217, the DrawingBot generates an\nimage \u02c6Il; the generated image is supplied to the CaptionBot, which produces a captioning sentence\n\u02c6Sl. Then we use the loss function of pairs (\u02c6Sl, S\u2217) and (\u02c6Il, I\u2217) to calculate the gradients and update\n\u03b8draw and \u03b8cap, the parameters of the DrawingBot and CaptionBot, simultaneously. As shown in\nthe right hand side of Fig. 1, for a given gold image I\u2217, the CaptionBot generates a sentence \u02c6Sr;\nthe generated sentence is supplied to the DrawingBot, which produces an image \u02c6Ir. Then we use\nthe loss function of pairs (\u02c6Sr, S\u2217) and (\u02c6Ir, I\u2217) to calculate the gradients and update \u03b8draw and \u03b8cap\nsimultaneously.\n\n3.2 Optimization theoretical underpinning of turbo learning\nConsidering the left hand side of Fig. 1, for a given gold sentence S\u2217, the DrawingBot generates an\nimage \u02c6Il; mathematically, we can represent this process by\n\u02c6Il = D(S\u2217|\u03b8draw)\n\n(1)\nwhere D(\u00b7) denotes the DrawingBot as a nonlinear operator parameterized by \u03b8draw. The generated\nimage \u02c6Il is supplied to the CaptionBot, which produces a captioning sentence \u02c6Sl; mathematically, we\n\n3\n\n\fcan represent this process by\n\n\u02c6Sl = C(\u02c6Il|\u03b8cap)\n\n(2)\nwhere C(\u00b7) denotes the CaptionBot as a nonlinear operator parameterized by \u03b8cap; conceptually,\noperator C(\u00b7) can be considered as an approximate inverse of operator D(\u00b7). Similarly in the right\nbranch, we have \u02c6Sr = C(I\u2217|\u03b8cap) and \u02c6Ir = D(\u02c6Sr|\u03b8draw).\nIdeally, we intend to minimize both E(S\u2217,I\u2217) [Ll(\u03b8draw, \u03b8cap)] and E(S\u2217,I\u2217) [Lr(\u03b8draw, \u03b8cap)], in\nwhich Ll(\u03b8draw, \u03b8cap) and Lr(\u03b8draw, \u03b8cap) are the loss function of the left hand side and the right\nhand side of Fig. 1, respectively. We solve this multi-objective optimization problem by converting it\nto a single objective optimization problem by summing up these two loss functions, i.e.,\n\nmin\n\n\u03b8draw,\u03b8cap\n\nE(S\u2217,I\u2217) [Ll(\u03b8draw, \u03b8cap)] + E(S\u2217,I\u2217) [Lr(\u03b8draw, \u03b8cap)] .\n\n(3)\n\nOur turbo training is intended to solve the optimization problem in (3) by the stochastic gradient\ndescent algorithm, where we randomly select one of the two terms and approximate the expectation\nby an empirical average over a mini-batch. In practice, we select these two terms alternatively, and\nuse the same mini-batch to compute the gradient for both Ll and Lr. This reuse of samples cuts the\nI/O cost by half during the training.\n\n3.3\n\nInsight of turbo learning\n\nLater in Section 4.4, the experimental results show that our proposed turbo joint training of CaptionBot\nand DrawingBot achieves signi\ufb01cant gains over separate training of CaptionBot and DrawingBot.\nThe reason is as follows. If we train CaptionBot and DrawingBot jointly under the turbo approach,\nthe turbo CaptionBot has both sentence information (generated sentence \u02c6S vs. gold sentence S\u2217) and\nimage information (generated image \u02c6I vs. gold image I\u2217) to guide the update of \u03b8cap; in other words,\nthe turbo CaptionBot is trained to generate a sentence, which not only is close to the ground truth\nsentence, but also captures as many classes/objects in the original input image as possible. In contrast,\nthe baseline CaptionBot only has sentence information (generated sentence \u02c6S vs. gold sentence S\u2217)\nfor updating \u03b8cap. The same reasoning is also true for the turbo DrawingBot.\n\n4 Turbo learning for supervised training\n\nThis section is organized as follows. Subsection 4.1 and Subsection 4.2 describe CaptionBot and\nDrawingBot, respectively; Subsection 4.3 describes turbo joint training of CaptionBot and Drawing-\nBot. Subsection 4.4 shows the experimental results.\n\n4.1 LSTM model for CaptionBot\n\nTo illustrate the bene\ufb01t of the turbo structure, in this section, we choose a simple baseline LSTM\nmodel as shown in Fig. 2 for the CaptionBot .\n\nFigure 2: Baseline LSTM model for CaptionBot\n\nConsider an image I with caption S. Assume that caption S consists of T words. We de\ufb01ne\nS = [x1,\u00b7\u00b7\u00b7 , xT ], where xt is a one-hot encoding vector of dimension V , where V is the size of the\n\n4\n\n\fvocabulary. The length T may be different for different captions. The t-th word in a caption, xt, is\nembedded into an nx-dimensional real-valued vector wt = Wext, where We \u2208 Rnx\u00d7V is a word\nembedding matrix.\nThe following is the basic LSTM model for captioning a single image which is \ufb01rst processed by a\nConvolutional Neural Network (CNN) such as ResNet and then conveyed to the LSTM [30, 31, 1].\n(4)\n\nht, ct = LST M (xt\u22121, ht\u22121, ct\u22121)\n\nwhere ht, ct are the hidden state and the cell state of the LSTM at time t, respectively.\nThe LSTM model in Eq. (4) for image captioning can be trained via minimizing the cross-entropy\n(XE) loss function, which is called Baseline LSTM-XE in this paper.\nTo directly optimize NLP metrics such as CIDEr [32] and address the exposure bias issue, reinforce-\nment learning (RL) can be used to train the LSTM model in Eq. (4) [12], which is called Baseline\nLSTM-RL in this paper. Under the RL terminology, the LSTM is regarded as an \u201cagent\u201d that interacts\nwith an external \u201cenvironment\u201d (e.g., words and image features). The parameters of the LSTM, \u03b8,\nde\ufb01ne a policy p\u03b8, that produces an \u201caction\u201d (e.g., prediction of the current word). After each action,\nthe agent updates its internal \u201cstate\u201d (e.g., cell and hidden states of the LSTM). Upon generating\nthe end-of-sentence (EOS) symbol, the agent receives a \u201creward\u201d denoted as r(S) (e.g., the CIDEr\nmetric of the generated sentence w.r.t. the ground-truth sentence), where S is the generated sentence.\nMinimizing the reinforcement loss (or maximizing rewards) does not ensure the readability and\n\ufb02uency of the generated caption [33]. Using a mixed loss function, which is a weighted combination\nof the cross-entropy (XE) loss LXE and the reinforcement learning (RL) loss LRL, helps improve\nreadability and \ufb02uency since the cross-entropy loss is based on a conditioned language model, with a\ngoal of producing \ufb02uent captions. That is, the mixed loss function is given by\n\nwhere \u03b3 \u2208 [0, 1].\n\nLmix(\u03b8) = \u03b3LRL(\u03b8) + (1 \u2212 \u03b3)LXE(\u03b8)\n\n(5)\n\n4.2 AttnGAN for DrawingBot\n\nAttnGAN [2] introduces an attentional multi-stage generative network, which can synthesize \ufb01ne-\ngrained details at different sub-regions of the image by paying attentions to the relevant words in the\nnatural language description. Trained by multi-level discriminators and a deep attentional multimodal\nsimilarity model (DAMSM) that computes a \ufb01ne-grained image-text matching loss, the AttnGAN\nachieves the state-of-the-art performance on the text-to-image generation tasks. In this work, we use\nAttnGAN, more precisely, its attentional multi-stage generative network, as the drawing bot.\nAs shown in Figure 3, the proposed attentional generative network has m stages Fi (i = 0, 1, . . . , m),\neach of which outputs an hidden state hi. On the last stage, we obtain the generated high-resolution\nimage after passing the last hidden state hm\u22121 through a convolutional layer with 3 \u00d7 3 kernel size.\nSpeci\ufb01cally, the drawing bot, denoted as \u02c6I = C(S, z), is decomposed as\n\nh0 = F0(z, F ca(e));\nhi = Fi(hi\u22121, F attn\n\u02c6I = Gm\u22121(hm\u22121).\n\ni\n\n(e, hi\u22121)) for i = 1, 2, ..., m \u2212 1;\n\n(6)\n\nHere, z is a noise vector usually sampled from a standard normal distribution. e is the global sentence\nvector of input sentence S, and e is the matrix of word vectors. The embedding was pretrained\nby the DAMSM model proposed by [2]. F ca represents the Conditioning Augmentation [21] that\nconverts the sentence vector e to the conditioning vector. F attn\nis the proposed attention model at\nthe ith stage in [2]. F ca, F attn\n, Fi, and Gm\u22121 are all modeled as neural networks. The AttnGAN is\ntrained by minimizing the sum of 1) the GAN matching loss that jointly approximates conditional\nand unconditional distributions of multi-scale images, and 2) a word-level image-text matching loss.\n\ni\n\ni\n\n4.3 Turbo training procedure for supervised learning\n\nNow, we describe the proposed turbo training procedure. As mentioned before, we se-\nlect Ll(\u03b8draw, \u03b8cap) and Lr(\u03b8draw, \u03b8cap) alternatively,\ni.e., minimizing Ll(\u03b8draw, \u03b8cap) and\nLr(\u03b8draw, \u03b8cap) alternatively. Hence, each iteration of the turbo training procedure consists of\nthe following three steps:\n\n5\n\n\fFigure 3: Drawingbot: the attentional multi-stage generative model from AttnGAN [2]\n\n\u2022 Step 1: minimizing Ll(\u03b8draw, \u03b8cap). As shown in the left hand side of Fig. 1, for a given\ngold sentence S\u2217, the DrawingBot generates an image \u02c6Il; the generated image is supplied to\nthe CaptionBot, which produces a captioning sentence \u02c6Sl. Then we use the following loss\nfunction to calculate the gradients and update \u03b8draw and \u03b8cap simultaneously:\nLl(\u03b8draw, \u03b8cap) = \u03b22(\u03b1(\u02c6Il, I\u2217) \u2212 r(\u02c6Sl)) + (1 \u2212 \u03b22)(\u03b21\u03b1(\u02c6Il, I\u2217)) + (1 \u2212 \u03b21)LXE(\u03b8cap)),\nLl(\u03b8draw, \u03b8cap) = \u03b22(\u03b1(\u02c6Il, I\u2217) \u2212 r(\u02c6Sl)) + (1 \u2212 \u03b22)(\u03b21\u03b1(\u02c6Il, I\u2217)) + (1 \u2212 \u03b21)Lmix(\u03b8cap)),\nwhere \u03b21, \u03b22 \u2208 [0, 1]; (7) and (8) are for the CaptionBot being LSTM-XE and LSTM-\nRL, respectively; r(\u02c6Sl) is a reward (e.g., the CIDEr metric of the generated sentence\n\u02c6Sl w.r.t. the ground-truth sentence S\u2217); \u03b1(\u02c6Il, I\u2217) is KL(p(y|I\u2217)||p(y|\u02c6Il)) where p(y|I\u2217)\nis the conditional distribution of label y given the ground-truth image I\u2217 and p(y|\u02c6Il) is\nthe conditional distribution of label y given the generated image \u02c6Il, and KL(p||q) is the\nKullback-Leibler divergence from q to p; label y is generated by Inception-v3 model in\nTensorFlow [34]. A label y is selected from 1,000 meaningful classes, such as \u201cZebra\u201d and\n\u201cDishwasher\u201d.\n\u2022 Step 2: minimizing Lr(\u03b8draw, \u03b8cap). As shown in the right hand side of Fig. 1, for a\ngiven gold image I\u2217, the CaptionBot generates a sentence \u02c6Sr; the generated sentence is\nsupplied to the DrawingBot, which produces an image \u02c6Ir. Then we use the loss func-\ntion Lr(\u03b8draw, \u03b8cap), which is de\ufb01ned by replacing (\u02c6Sl, \u02c6Il) with (\u02c6Sr, \u02c6Ir) in (7) or (8), to\ncalculate the gradients and update \u03b8draw and \u03b8cap simultaneously.\n\n\u2022 Step 3: Go to Step 1 until convergence.\n\n(7)\n(8)\n\nNote that it is critical to choose the right reconstruction loss to enforce cycle consistency in turbo\nlearning. After experimenting with pixel-wise loss, Deep Structured Semantic Models (DSSM)\nloss [35], and our \u201cperceptual\u201d-like loss \u03b1(\u02c6Il, I\u2217), we found that our \u201cperceptual\u201d-like loss achieves\nsigni\ufb01cantly better performance.\nIt is worth mentioning that the above turbo training is different from GAN training. GAN training\nis adversarial while our turbo training is collaborative. Actually, our method is more similar to the\ntraining of Auto-Encoder (AE). Each branch of the turbo training in Fig. 1) can be viewed as an AE.\nFor example, the left branch is an AE where the DrawingBot is the encoder and the CaptionBot is\nthe decoder. Different from vinilla AE (VAE), our encoding space is semantically meaningful, and\nsupervised signal is available for the encoding procedure because image-caption pairs are available.\nFor the decoder/reconstruction loss, instead of using pixel-wise loss in VAE, we propose to use a\n\u201cperceptual\u201d-like loss to better capture its semantics.\n\n4.4 Experimental results\n\n4.4.1 Dataset\n\nTo evaluate the performance of our proposed approach, we use the COCO dataset [36]. The COCO\ndataset contains 123,287 images, each of which is annotated with at least 5 captions. We use the\n\n6\n\n\fsame pre-de\ufb01ned splits as in [8, 1]: 113,287 images for training, 5,000 images for validation, and\n5,000 images for testing. We use the same vocabulary as that employed in [1], which consists of\n8,791 words.\n\nTable 1: Performance of CaptionBot and corresponding DrawingBot with BLEU-4 as the reward\nr(\u02c6S).\n\nCaptionbot\n\nBaseline LSTM-XE\nTurbo LSTM-XE\n\nBaseline LSTM-RL\nTurbo LSTM-RL\n\nBLEU-4\n0.2684\n0.3168\n0.2831\n0.3183\n\n4.4.2 Evaluation\n\nCIDEr-D ROUGE-L METEOR SPICE\n0.1663\n0.7362\n0.7364\n0.1720\n0.1733\n0.7238\n0.7327\n0.1735\n\n0.2193\n0.2196\n0.2190\n0.2191\n\n0.4937\n0.4938\n0.4963\n0.4974\n\nDrawingbot\n\nAttnGAN\nTurbo AttnGAN\n\nAttnGAN\nTurbo AttnGAN\n\nInception\n25.68\n26.69\n25.68\n26.88\n\nFor the CNN, which is used to extract features from an image, we used ResNet-50 [37] pretrained\non the ImageNet dataset. The reason to use ResNet-50 instead of ResNet-152 is mainly for the\nconsideration of training and inference ef\ufb01ciency. We will report the experimental results for ResNet-\n152 in our future work.\nThe feature vector v has 2048 dimensions. Word embedding vectors in We are downloaded from\nthe web [38]. The model is implemented in TensorFlow [34] with the default settings for random\ninitialization and optimization by backpropagation. We empirically set \u03b21 = \u03b22 = 0.5.\nThe widely-used BLEU [39], METEOR [40], CIDEr [32], and SPICE [41] metrics are reported in\nour quantitative evaluation of the performance of the proposed approach.\nTable 1 shows the experimental results on the COCO dataset with 113,287 training samples, for which\nwe use BLEU-4 as the reward r(\u02c6S) in (7), and (8). As shown in the table, our proposed turbo approach\nachieves signi\ufb01cant gain over separate training of CaptionBot and DrawingBot. Following [2], we use\nthe Inception score [42] as the quantitative evaluation measure. The larger Inception score, the better\nperformance. Table 1 shows the Inception score for AttnGAN and Turbo AttnGAN. It is observed\nthat turbo AttnGANs achieve higher Inception scores than AttnGAN.\nTable 2 shows the experimental results on the COCO dataset with 113,287 training samples, for\nwhich we use CIDEr-D as the reward r(\u02c6S) in (7), and (8). As shown in the table, our proposed turbo\napproach achieves signi\ufb01cant gain over separate training of CaptionBot and DrawingBot.\n\nTable 2: Performance of CaptionBot and corresponding DrawingBot with CIDEr-D as the reward\nr(\u02c6S).\n\nCaptionbot\n\nBaseline LSTM-XE\nTurbo LSTM-XE\n\nBaseline LSTM-RL\nTurbo LSTM-RL\n\nBLEU-4\n0.2684\n0.3033\n0.2831\n0.3162\n\nCIDEr-D ROUGE-L METEOR SPICE\n0.7362\n0.1663\n0.1721\n0.7473\n0.1733\n0.7238\n0.7478\n0.1748\n\n0.2193\n0.2195\n0.2190\n0.2192\n\n0.4937\n0.4942\n0.4963\n0.4981\n\nDrawingbot\n\nAttnGAN\nTurbo AttnGAN\n\nAttnGAN\nTurbo AttnGAN\n\nInception\n25.68\n26.72\n25.68\n26.83\n\nBecause we use ResNet-50 instead of ResNet-152, the performance of the baseline LSTM CaptionBot\nis not as good as the state-of-the-art LSTM, which uses ResNet-152 or better features.\nIt is worth mentioning that this paper aims at developing a turbo learning approach to training\nLSTM-based CaptionBot; therefore, it is directly comparable to an LSTM baseline. Therefore, in the\nexperiments, we focus on the comparison to a strong CNN-LSTM baseline. We acknowledge that\nmore recent papers [11, 12, 13, 14, 1] reported better performance on the task of image captioning.\nPerformance improvements in these more recent models are mainly due to using better image features\nsuch as those obtained by region-based convolutional neural networks (R-CNN), or using more\ncomplex attention mechanisms [1] to provide a better context vector for caption generation, or using\nan ensemble of multiple LSTMs, among others. However, the LSTM is still playing a core role in\nthese works and we believe improvement over the core LSTM by turbo learning is still very valuable\nand orthogonal to most existing works; that is why we compare the turbo LSTM with a native LSTM.\n\n7\n\n\f5 Semi-supervised learning for CaptionBot\n\n5.1 Semi-supervised learning approach for CaptionBot\n\nIn this section, we present a semi-supervised learning approach for training CaptionBot.\nHuman annotation is costly. Hence, many images on the Internet has no caption and semi-supervised\nlearning for CaptionBot is desirable. In fact, the right hand side of Fig. 1 forms a loop, which\nenables semi-supervised learning for CaptionBot since the closed loop can provide pseudo-labels for\nunlabeled samples.\nAs shown in the right hand side of Fig. 1, for a given gold image I\u2217, the CaptionBot generates a\nsentence \u02c6S; the generated sentence is supplied to the DrawingBot, which produces an image \u02c6I. The\ngenerated image \u02c6I is supplied to the CaptionBot, which generates a sentence \u02dcS.\nThen we can use the following equation to calculate the loss for an unlabeled sample:\n\nfor LSTM-XE CaptionBot,\nfor LSTM-RL CaptionBot,\n\nL(\u03b8cap) = \u03b21 \u00d7 \u03b1(\u02c6I, I\u2217)) + (1 \u2212 \u03b21) \u00d7 LXE(\u03b8cap)\nL(\u03b8cap) = \u03b21 \u00d7 \u03b1(\u02c6I, I\u2217)) + (1 \u2212 \u03b21) \u00d7 CIDEr(\u02c6S, \u02dcS)\n\n(9)\n(10)\nwhere \u03b21 \u2208 [0, 1]; \u03b1(\u02c6I, I\u2217) is KL(p(y|I\u2217)||p(y|\u02c6I)), and LXE(\u03b8cap) is the cross-entropy between \u02c6S\nand \u02dcS; CIDEr(\u02dcS, \u02c6S) is the CIDEr metric of the sentence \u02dcS w.r.t. the sentence \u02c6S.\nIn our experiments, in order to smoothly transit from the initial model trained from labeled data to\nthe model training from labeled and unlabeled data, we adopted the following strategy. For each\nmini-batch, we use a half number of samples from unlabeled data and another half number of samples\nfrom labeled data (sampled from the labeled dataset used to train the initial model). The objective is\nto minimize the total loss of all labeled and unlabeled samples. We compute the gradient of the total\nloss of all labeled and unlabeled samples in a mini-batch and update the neural weights \u03b8cap of the\nCaptionBot.\nHere, we would like to relate our semi-supervised learning to the CycleGAN scheme [43]. Both\nschemes use cycle consistency, but they are implemented in different ways. Speci\ufb01cally, CycleGAN\nuses pixel-wise L1 loss while we use \u201cperceptual\u201d-like loss for image semantic consistency. Cycle-\nGAN has not been applied to text reconstruction, while we use CIDEr score for caption consistency.\nWhile CycleGAN was mainly used in homogeneous modality for image-to-image translation, our\nframework works for multi-modality problems, i.e., image-to-text and text-to-image tasks.\n\n5.2 Performance Evaluation\n\nSame as Section 4.4, we also use COCO dataset to evaluate the performance of the proposed\nsemi-supervised learning approach for CaptionBot. We set \u03b21 = 0.85.\nTable 3 shows the experimental results for semi-supervised learning with 1,000 labeled training\nsamples, 112,287 unlabeled training samples and CIDEr-D as the reward. It is observed that turbo\nLSTMs signi\ufb01cantly outperform baseline LSTMs for semi-supervised image captioning.\n\nTable 3: Performance of CaptionBot with 1,000 labeled training samples, 112,287 unlabeled training\nsamples and CIDEr-D as the reward r(\u02c6S).\n\nMethods\nBaseline LSTM-XE\nTurbo LSTM-XE\nBaseline LSTM-RL\nTurbo LSTM-RL\n\nBLEU-4 CIDEr-D ROUGE-L METEOR SPICE\n0.1231\n0.1036\n0.1432\n0.1832\n0.1361\n0.1181\n0.1982\n0.1466\n\n0.4331\n0.6841\n0.4924\n0.6954\n\n0.3462\n0.4174\n0.3518\n0.4215\n\n0.1864\n0.2071\n0.1863\n0.2082\n\nFig. 4 shows some sample results of CaptionBot and DrawingBot for supervised and semi-supervised\nlearning.\n\n8\n\n\fFigure 4: Sample results of CaptionBot and DrawingBot for supervised and semi-supervised learning\n\n6 Conclusion\n\nWe have presented a novel turbo learning approach to jointly training a CaptionBot and a DrawingBot.\nTo the best of our knowledge, this is the \ufb01rst work that studies both problems in one framework. The\nframework leverages the duality of image captioning and text-to-image generation and forms a closed\nloop, which results in a new loss metric by comparing the initial input with the feedback produced by\nthe whole loop. This not only leads to better CaptionBot and DrawingBot by joint training, but also\nmakes semi-supervised learning possible. Experimental results on the COCO dataset have effectively\nvalidated the advantages of the proposed joint learning approach.\nIn our future work, we will explore if adding more unlabeled data can further improve the performance\nof both bots, and extend the turbo learning approach to other domains, for example, speech recognition\nvs. text to speech, question answering vs. question generation, search (matching queries to documents)\nvs. keyword extraction (extracting keywords/queries for documents).\n\nReferences\n[1] Z. Gan, C. Gan, X. He, Y. Pu, K. Tran, J. Gao, L. Carin, and L. Deng, \u201cSemantic compositional networks\nfor visual captioning,\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\n2017.\n\n[2] Q. H. H. Z. Z. G. X. H. X. H. Tao Xu, Pengchuan Zhang, \u201cAttngan: Fine-grained text to image generation\n\nwith attentional generative adversarial networks,\u201d 2018.\n\n[3] J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, \u201cDeep captioning with multimodal recurrent\nneural networks (m-rnn),\u201d in Proceedings of International Conference on Learning Representations, 2015.\n[4] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, \u201cShow and tell: A neural image caption generator,\u201d in\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3156\u20133164.\n[5] J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell, \u201cLanguage models\n\nfor image captioning: The quirks and what works,\u201d arXiv preprint arXiv:1505.01809, 2015.\n\n[6] X. Chen and L. Zitnick, \u201cMind\u2019s eye: A recurrent visual representation for image caption generation,\u201d in\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2422\u20132431.\n[7] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell,\n\u201cLong-term recurrent convolutional networks for visual recognition and description,\u201d in Proceedings of the\nIEEE conference on computer vision and pattern recognition, 2015, pp. 2625\u20132634.\n\n[8] A. Karpathy and L. Fei-Fei, \u201cDeep visual-semantic alignments for generating image descriptions,\u201d in\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3128\u20133137.\n[9] R. Kiros, R. Salakhutdinov, and R. Zemel, \u201cMultimodal neural language models,\u201d in Proceedings of the\n\n31st International Conference on Machine Learning (ICML-14), 2014, pp. 595\u2013603.\n\n[10] R. Kiros, R. Salakhutdinov, and R. S. Zemel, \u201cUnifying visual-semantic embeddings with multimodal\n\nneural language models,\u201d arXiv preprint arXiv:1411.2539, 2014.\n\n[11] K. Xu, H. Wang, and P. Tang, \u201cImage captioning with deep lstm based on sequential residual,\u201d in\n\nProceedings of IEEE International Conference on Multimedia and Expo (ICME), 2017, pp. 361\u2013366.\n\n[12] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, \u201cSelf-critical sequence training for image\ncaptioning,\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.\n\n9\n\n\f[13] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, \u201cBoosting image captioning with attributes,\u201d in Proceedings of\n\nInternational Conference on Computer Vision, 2017.\n\n[14] J. Lu, C. Xiong, D. Parikh, and R. Socher, \u201cKnowing when to look: Adaptive attention via a visual sentinel\nfor image captioning,\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), vol. 6, 2017.\n\n[15] E. Mansimov, E. Parisotto, L. J. Ba, and R. Salakhutdinov, \u201cGenerating images from captions with\n\nattention,\u201d in ICLR, 2016.\n\n[16] K. Gregor, I. Danihelka, A. Graves, D. J. Rezende, and D. Wierstra, \u201cDRAW: A recurrent neural network\n\nfor image generation,\u201d in ICML, 2015.\n\n[17] S. E. Reed, A. van den Oord, N. Kalchbrenner, S. G. Colmenarejo, Z. Wang, Y. Chen, D. Belov, and\n\nN. de Freitas, \u201cParallel multiscale autoregressive density estimation,\u201d in ICML, 2017.\n\n[18] A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu, \u201cConditional\n\nimage generation with pixelcnn decoders,\u201d in NIPS, 2016.\n\n[19] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, \u201cGenerative adversarial text-to-image\n\nsynthesis,\u201d in ICML, 2016.\n\n[20] S. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, and H. Lee, \u201cLearning what and where to draw,\u201d in\n\nNIPS, 2016.\n\n[21] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. Metaxas, \u201cStackgan: Text to photo-realistic\n\nimage synthesis with stacked generative adversarial networks,\u201d in ICCV, 2017.\n\n[22] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, \u201cStackgan++: Realistic image\n\nsynthesis with stacked generative adversarial networks,\u201d arXiv: 1710.10916, 2017.\n\n[23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio,\n\u201cGenerative adversarial nets,\u201d in Advances in neural information processing systems, 2014, pp. 2672\u20132680.\n[24] A. Radford, L. Metz, and S. Chintala, \u201cUnsupervised representation learning with deep convolutional\n\ngenerative adversarial networks,\u201d in ICLR, 2016.\n\n[25] E. L. Denton, S. Chintala, A. Szlam, and R. Fergus, \u201cDeep generative image models using a laplacian\n\npyramid of adversarial networks,\u201d in NIPS, 2015.\n\n[26] T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, \u201cImproved techniques\n\nfor training gans,\u201d in NIPS, 2016.\n\n[27] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, \u201cPhoto-\n\nrealistic single image super-resolution using a generative adversarial network,\u201d in CVPR, 2017.\n\n[28] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, \u201cImage-to-image translation with conditional adversarial\n\nnetworks,\u201d in CVPR, 2017.\n\n[29] Y. Xia, D. He, T. Qin, L. Wang, N. Yu, T.-Y. Liu, and W.-Y. Ma, \u201cDual learning for machine translation,\u201d\n\narXiv preprint arXiv:1611.00179, 2016.\n\n[30] Q. Wu, C. Shen, L. Liu, A. Dick, and A. van den Hengel, \u201cWhat value do explicit high level concepts have\nin vision to language problems?\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, 2016, pp. 203\u2013212.\n\n[31] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, \u201cImage captioning with semantic attention,\u201d in Proceedings\n\nof the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4651\u20134659.\n\n[32] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, \u201cCider: Consensus-based image description evaluation,\u201d\nin Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4566\u20134575.\n\n[33] R. Pasunuru and M. Bansal, \u201cReinforced video captioning with entailment rewards,\u201d in EMNLP, 2017.\n[34] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean,\nM. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser,\nM. Kudlur, J. Levenberg, D. Man\u00e9, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens,\nB. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Vi\u00e9gas, O. Vinyals,\nP. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, \u201cTensorFlow: Large-scale machine\nlearning on heterogeneous systems,\u201d 2015, software available from tensor\ufb02ow.org. [Online]. Available:\nhttps://www.tensor\ufb02ow.org/\n\n[35] P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck, \u201cLearning deep structured semantic models\nfor web search using clickthrough data,\u201d in Proceedings of the 22Nd ACM International Conference\non Information & Knowledge Management, ser. CIKM\u201913. New York, NY, USA: ACM, 2013, pp.\n2333\u20132338. [Online]. Available: http://doi.acm.org/10.1145/2505515.2505665\n\n[36] COCO, \u201cCoco dataset for image captioning,\u201d http://mscoco.org/dataset/#download, 2017.\n\n10\n\n\f[37] K. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep residual learning for image recognition,\u201d in Proceedings of the\n\nIEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770\u2013778.\n\n[38] J. Pennington, R. Socher, and C. Manning, \u201cStanford glove: Global vectors for word representation,\u201d\n\nhttps://nlp.stanford.edu/projects/glove/, 2017.\n\n[39] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, \u201cBleu: a method for automatic evaluation of machine\ntranslation,\u201d in Proceedings of the 40th annual meeting on association for computational linguistics.\nAssociation for Computational Linguistics, 2002, pp. 311\u2013318.\n\n[40] S. Banerjee and A. Lavie, \u201cMeteor: An automatic metric for mt evaluation with improved correlation with\nhuman judgments,\u201d in Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for\nmachine translation and/or summarization. Association for Computational Linguistics, 2005, pp. 65\u201372.\n[41] P. Anderson, B. Fernando, M. Johnson, and S. Gould, \u201cSpice: Semantic propositional image caption\n\nevaluation,\u201d in ECCV, 2016.\n\n[42] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, \u201cImproved techniques for\n\ntraining gans,\u201d arXiv preprint arXiv:1606.03498, 2016.\n\n[43] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, \u201cUnpaired image-to-image translation using cycle-consistent\n\nadversarial networks,\u201d in IEEE International Conference on Computer Vision (ICCV), 2017.\n\n11\n\n\f", "award": [], "sourceid": 3171, "authors": [{"given_name": "Qiuyuan", "family_name": "Huang", "institution": "Microsoft Research AI"}, {"given_name": "Pengchuan", "family_name": "Zhang", "institution": "Microsoft Research"}, {"given_name": "Dapeng", "family_name": "Wu", "institution": "University of Florida"}, {"given_name": "Lei", "family_name": "Zhang", "institution": "Microsoft Research"}]}