{"title": "Teaching Machines to Describe Images with Natural Language Feedback", "book": "Advances in Neural Information Processing Systems", "page_first": 5068, "page_last": 5078, "abstract": "Robots will eventually be part of every household.  It is thus critical to enable algorithms to learn from and be guided by non-expert users. In this paper, we bring a human in the loop, and enable a human teacher to give feedback to a learning agent in the form of natural language. A descriptive sentence can provide a stronger learning signal than a numeric reward in that it can easily point to where the mistakes are and how to correct them. We focus on the problem of image captioning in which the quality of the output can easily be judged by non-experts.  We propose a phrase-based captioning model trained with policy gradients, and design a critic that provides reward to the learner by conditioning on the human-provided feedback. We show  that by exploiting descriptive feedback our model learns to perform better than when given independently written human captions.", "full_text": "Teaching Machines to Describe Images via Natural\n\nLanguage Feedback\n\nHuan Ling1, Sanja Fidler1,2\n\nUniversity of Toronto1, Vector Institute2\n{linghuan,fidler}@cs.toronto.edu\n\nAbstract\n\nRobots will eventually be part of every household. It is thus critical to enable\nalgorithms to learn from and be guided by non-expert users. In this paper, we\nbring a human in the loop, and enable a human teacher to give feedback to a\nlearning agent in the form of natural language. We argue that a descriptive sentence\ncan provide a much stronger learning signal than a numeric reward in that it can\neasily point to where the mistakes are and how to correct them. We focus on\nthe problem of image captioning in which the quality of the output can easily be\njudged by non-experts. In particular, we \ufb01rst train a captioning model on a subset\nof images paired with human written captions. We then let the model describe new\nimages and collect human feedback on the generated descriptions. We propose a\nhierarchical phrase-based captioning model, and design a feedback network that\nprovides reward to the learner by conditioning on the human-provided feedback.\nWe show that by exploiting descriptive feedback on new images our model learns\nto perform better than when given human written captions on these images.\n\n1\n\nIntroduction\n\nIn the era where A.I. is slowly \ufb01nding its way into everyone\u2019s lives, be in the form of social bots [36, 2],\npersonal assistants [24, 13, 32], or household robots [1], it becomes critical to allow non-expert users\nto teach and guide their robots [37, 18]. For example, if a household robot keeps bringing food served\non an ashtray thinking it\u2019s a plate, one should ideally be able to educate the robot about its mistakes,\npossibly without needing to dig into the underlying software.\nReinforcement learning has become a standard way of training arti\ufb01cial agents that interact with an\nenvironment. There have been signi\ufb01cant advances in a variety of domains such as games [31, 25],\nrobotics [17], and even \ufb01elds like vision and NLP [30, 19]. RL agents optimize their action policies\nso as to maximize the expected reward received from the environment. Training typically requires a\nlarge number of episodes, particularly in environments with large action spaces and sparse rewards.\nSeveral works explored the idea of incorporating humans in the learning process, in order to help\nthe reinforcement learning agent to learn faster [35, 12, 11, 6, 5]. In most cases, a human teacher\nobserves the agent act in an environment, and is allowed to give additional guidance to the learner.\nThis feedback typically comes in the form of a simple numerical (or \u201cgood\u201d/\u201cbad\u201d) reward which is\nused to either shape the MDP reward [35] or directly shape the policy of the learner [5].\nIn this paper, we aim to exploit natural language as a way to guide an RL agent. We argue that a\nsentence provides a much stronger learning signal than a numeric reward in that it can easily point\nto where the mistakes occur and suggests how to correct them. Such descriptive feedback can thus\nnaturally facilitate solving the credit assignment problem as well as to help guide exploration. Despite\nits clear bene\ufb01ts, very few approaches aimed at incorporating language in Reinforcement Learning.\nIn pioneering work, [22] translated natural language advice into a short program which was used to\nbias action selection. While this is possible in limited domains such as in navigating a maze [22] or\nlearning to play a soccer game [15], it can hardly scale to the real scenarios with large action spaces\nrequiring versatile language feedback.\n\n\fMachine\n( a cat ) ( sitting ) ( on a sidewalk ) ( next to a street . )\n\nHuman Teacher\nFeedback: There is a dog on a sidewalk, not a cat.\nType of mistake: wrong object\nSelect the mistake area:\n( a cat ) ( sitting ) ( on a sidewalk ) ( next to a street . )\nCorrect the mistake:\n( a dog ) ( sitting ) ( on a sidewalk ) ( next to a street . )\n\nFigure 1: Our model accepts feedback from a human teacher in the form of natural language. We generate\ncaptions using the current snapshot of the model and collect feedback via AMT. The annotators are requested to\nfocus their feedback on a single word/phrase at a time. Phrases, indicated with brackets in the example, are part\nor our captioning model\u2019s output. We also collect information about which word the feedback applies to and its\nsuggested correction. This information is used to train our feedback network.\n\nHere our goal is to allow a non-expert human teacher to give feedback to an RL agent in the form of\nnatural language, just as one would to a learning child. We focus on the problem of image captioning\nin which the quality of the output can easily be judged by non-experts.\nTowards this goal, we make several contributions. We propose a hierarchical phrase-based RNN as\nour image captioning model, as it can be naturally integrated with human feedback. We design a web\ninterface which allows us to collect natural language feedback from human \u201cteachers\u201d for a snapshot of\nour model, as in Fig. 1. We show how to incorporate this information in Policy Gradient RL [30], and\nshow that we can improve over RL that has access to the same amount of ground-truth captions. Our\ncode and data will be released (http://www.cs.toronto.edu/~linghuan/feedbackImageCaption/)\nto facilitate more human-like training of captioning models.\n\n2 Related Work\n\nSeveral works incorporate human feedback to help an RL agent learn faster. [35] exploits humans\nin the loop to teach an agent to cook in a virtual kitchen. The users watch the agent learn and\nmay intervene at any time to give a scalar reward. Reward shaping [26] is used to incorporate this\ninformation in the MDP. [6] iterates between \u201cpractice\u201d, during which the agent interacts with the real\nenvironment, and a critique session where a human labels any subset of the chosen actions as good or\nbad. In [12], the authors compare different ways of incorporating human feedback, including reward\nshaping, Q augmentation, action biasing, and control sharing. The same authors implement their\nTAMER framework on a real robotic platform [11]. [5] proposes policy shaping which incorporates\nright/wrong feedback by utilizing it as direct policy labels. These approaches mostly assume that\nhumans provide a numeric reward, unlike in our work where the feedback is given in natural language.\nA few attempts have been made to advise an RL agent using language. [22]\u2019s pioneering work\ntranslated advice to a short program which was then implemented as a neural network. The units in\nthis network represent Boolean concepts, which recognize whether the observed state satis\ufb01es the\nconstraints given by the program. In such a case, the advice network will encourage the policy to\ntake the suggested action. [15] incorporated natural language advice for a RoboCup simulated soccer\ntask. They too translate the advice in a formal language which is then used to bias action selection.\nParallel to our work, [7] exploits textual advice to improve training time of the A3C algorithm in\nplaying an Atari game. Recently, [37, 18] incorporates human feedback to improve a text-based QA\nagent. Our work shares similar ideas, but applies them to the problem of image captioning. In [27],\nthe authors incorporate human feedback in an active learning scenario, however not in an RL setting.\nCaptioning represents a natural way of showing that our algorithm understands a photograph to a\nnon-expert observer. This domain has received signi\ufb01cant attention [8, 39, 10], achieving impressive\nperformance on standard benchmarks. Our phrase model shares the most similarity with [16],\nbut differs in that exploits attention [39], linguistic information, and RL to train. Several recent\napproaches trained the captioning model with policy gradients in order to directly optimize for the\ndesired performance metrics [21, 30, 3]. We follow this line of work. However, to the best of our\nknowledge, our work is the \ufb01rst to incorporate natural language feedback into a captioning model.\n\n2\n\n\fFigure 2: Our hierarchical phrase-based cap-\ntioning model, composed of a phrase-RNN at\nthe top level, and a word-level RNN which out-\nputs a sequence of words for each phrase. The\nuseful property of this model is that it directly\nproduces an output sentence segmented into\nlinguistic phrases. We exploit this information\nwhile collecting and incorporating human feed-\nback into the model. Our model also exploits\nattention, and linguistic information (phrase\nlabels such as noun, preposition, verb, and con-\njunction phrase). Please see text for details.\n\nRelated to our efforts is also work on dialogue based visual representation learning [40, 41], however\nthis work tackles a simpler scenario, and employs a slightly more engineered approach.\nWe stress that our work differs from the recent efforts in conversation modeling [19] or visual\ndialog [4] using Reinforcement Learning. These models aim to mimic human-to-human conversations\nwhile in our work the human converses with and guides an arti\ufb01cial learning agent.\n\n3 Our Approach\n\nOur framework consists of a new phrase-based captioning model trained with Policy Gradients that\nincorporates natural language feedback provided by a human teacher. While a number of captioning\nmethods exist, we design our own which is phrase-based, allowing for natural guidance by a non-\nexpert. In particular, we argue that the strongest learning signal is provided when the feedback\ndescribes one mistake at a time, e.g. a single wrong word or a phrase in a caption. An example can\nbe seen in Fig. 1. This is also how one most effectively teaches a learning child. To avoid parsing the\ngenerated sentences at test time, we aim to predict phrases directly with our captioning model. We\n\ufb01rst describe our phrase-based captioner, then describe our feedback collection process, and \ufb01nally\npropose how to exploit feedback as a guiding signal in policy gradient optimization.\n\n3.1 Phrase-based Image Captioning\n\nOur captioning model, forming the base of our approach, uses a hierarchical Recurrent Neural\nNetwork, similar to [34, 14]. In [14], the authors use a two-level LSTM to generate paragraphs,\nwhile [34] uses it to generate sentences as a sequence of phrases. The latter model shares a similar\noverall structure as ours, however, our model additionally reasons about the type of phrases and\nexploits the attention mechanism over the image.\nThe structure of our model is best explained through Fig. 2. The model receives an image as input and\noutputs a caption. It is composed of a phrase RNN at the top level, and a word RNN that generates a\nsequence of words for each phrase. One can think of the phrase RNN as providing a \u201ctopic\u201d at each\ntime step, which instructs the word RNN what to talk about.\nFollowing [39], we use a convolutional neural network in order to extract a set of feature vectors\na = (a1, . . . , an), with aj a feature in location j in the input image. We denote the hidden state of\nthe phrase RNN at time step t with ht, and ht,i to denote the i-th hidden state of the word RNN for\nthe t-th phrase. Computation in our model can be expressed with the following equations:\n\nht = fphrase(ht\u22121, lt\u22121, ct\u22121, et\u22121)\nlt = softmax(fphrase\u2212label(ht))\nct = fatt(ht, lt, a)\n\nN(cid:124)\nN\nR\n-\ne\ns\na\nr\nh\np\n\n(cid:123)\n(cid:122)\n\n(cid:125)\n\net = fword\u2212phrase(wt,1, . . . , wt,end)\n\nht,0 = fphrase\u2212word(ht, lt, ct)\nht,i = fword(ht,i\u22121, ct, wt,i)\nwt,i = fout(ht,i, ct, wt,i\u22121)\n\nN(cid:124)\nN\nR\n-\nd\nr\no\nw\n\n(cid:123)\n(cid:122)\n\n(cid:125)\n\nfphrase\nfphrase\u2212label\nfatt\nfphrase\u2212word\n\nLSTM, dim 256\n\n3-layer MLP\n\n2-layer MLP with ReLu\n3-layer MLP with ReLu\n\nfword\nfout\nfword\u2212phrase mean+3-lay. MLP with ReLu\n\nLSTM, dim 256\ndeep decoder [28]\n\n3\n\n\fImage\n\nRef. caption\n\nFeedback\n\nCorr. caption\n\nImage\n\nRef. caption\n\nFeedback\n\nCorr. caption\n\n( a woman ) ( is sit-\nting ) ( on a bench\n) ( with a plate ) (\nof food . )\n\nthe\nWhat\nwoman\nis\nsitting on is\nnot visible.\n\n( a woman ) ( is\nsitting ) ( with a\nplate ) ( of food .\n)\n\n( a man ) ( rid-\ning a motorcy-\ncle ) ( on a city\nstreet . )\n\nis\n\nThere\na\nman and a\nwoman.\n\n( a man and a\nwoman ) ( riding a\nmotorcycle ) ( on\na city street . )\n\n( a horse ) ( is\nstanding ) ( in a\nbarn ) ( in a \ufb01eld\n. )\n\nThere is no\nbarn. There\nis a fence.\n\n( a horse ) ( is\nstanding ) ( in\na fence ) ( in a\n\ufb01eld . )\n\n(\n( a man )\nis swinging a\nbaseball bat ) (\non a \ufb01eld . )\n\nThe baseball\nplayer is not\nswinging\na\nbate.\n\n( a man ) ( is play-\ning baseball ) ( on\na \ufb01eld . )\n\nTable 1: Examples of collected feedback. Reference caption comes from the MLE model.\n\nTable 2: Statistics for our collected feedback information. The table on the right shows how many times the\nfeedback sentences mention words to be corrected and suggest correction.\n\nNum. of evaluated examples (annot. round 1)\n\nEvaluated as containing errors\n\nTo ask for feedback (annot. round 2)\n\nAvg. num. of feedback rounds per image\n\nAvg. num. of words in feedback sent.\nAvg. num. of words needing correction\n\nAvg. num. of modi\ufb01ed words\n\n9000\n5150\n4174\n2.22\n8.04\n1.52\n1.46\n\nSomething should be replaced\n\nmistake word is in description\ncorrect word is in description\n\nSomething is missing\n\nmissing word is in description\n\nSomething should be removed\n\nremoved word is in description\n\n2999\n2664\n2674\n334\n246\n841\n779\n\nfeedback round: number of correction rounds for the same example, description: natural language feedback\n\nFigure 3: Caption quality evalua-\ntion by the human annotators. Plot on\nthe left shows evaluation for captions\ngenerated with our reference model\n(MLE). The right plot shows evalua-\ntion of the human-corrected captions\n(after completing at least one round\nof feedback).\n\nAs in [39], ct denotes a context vector obtained by applying the attention mechanism to the image.\nThis context vector essentially represents the image area that the model \u201clooks at\u201d in order to generate\nthe t-th phrase. This information is passed to both the word-RNN as well as to the next hidden state\nof the phrase-RNN. We found that computing two different context vectors, one passed to the phrase\nand one to the word RNN, improves generation by 0.6 points (in weighted metric, see Table 4) mainly\nhelping the model to avoid repetition of words. Furthermore, we noticed that the quality of attention\nsigni\ufb01cantly improves (1.5 points, Table 4) if we provide it with additional linguistic information. In\nparticular, at each time step t our phrase RNN also predicts a phrase label lt, following the standard\nde\ufb01nition from the Penn Tree Bank. For each phrase, we predict one out of four possible phrase labels,\ni.e., a noun (NP), preposition (PP), verb (VP), and a conjunction phrase (CP). We use additional\n<EOS> token to indicate the end of the sentence. By conditioning on the NP label, we help the model\nlook at the objects in the image, while VP may focus on more global image information.\nAbove, wt,i denotes the i-th word output of the word-RNN in the t-th phrase, encoded with a one-hot\nvector. Note that we use an additional <EOP> token in word-RNN\u2019s vocabulary, which signals the\nend-of-phrase. Further, et encodes the generated phrase via simple mean-pooling over the words,\nwhich provides additional word-level context to the next phrase. Details about the choices of the\nfunctions are given in the table. Following [39], we use a deep output layer [28] in the LSTM and\ndouble stochastic attention.\nImplementation details.\nTo train our hierarchical model, we \ufb01rst process MS-COCO image\ncaption data [20] using the Stanford Core NLP toolkit [23]. We \ufb02atten each parse tree, separate a\nsentence into parts, and label each part with a phrase label (<NP>, <PP>, <CP>, <VP>). To simplify\nthe phrase structure, we merge some NPs to its previous phrase label if it is not another NP.\nPre-training. We pre-train our model using the standard cross-entropy loss. We use the ADAM\noptimizer [9] with learning rate 0.001. We discuss Policy Gradient optimization in Subsec. 3.4.\n\n3.2 Crowd-sourcing Human Feedback\n\nWe aim to bring a human in the loop when training the captioning model. Towards this, we create a\nweb interface that allows us to collect feedback information on a larger scale via AMT. Our interface\n\n4\n\nperfectaccecptablegrammarminor_errormajor_error050010001500200025003000evaluation after correction\fFigure 4: The architecture of our feedback network (FBN) that classi\ufb01es each phrase (bottom left) in a sampled\nsentence (top left) as either correct, wrong or not relevant, by conditioning on the feedback sentence.\n\nis akin to that depicted in Fig. 1, and we provide further visualizations in the Appendix. We also\nprovide it online on our project page. In particular, we take a snapshot of our model and generate\ncaptions for a subset of MS-COCO images [20] using greedy decoding. In our experiments, we take\nthe model trained with the MLE objective.\nWe do two rounds of annotation. In the \ufb01rst round, the annotator is shown a captioned image and\nis asked to assess the quality of the caption, by choosing between: perfect, acceptable, grammar\nmistakes only, minor or major errors. We asked the annotators to choose minor and major error if the\ncaption contained errors in semantics, i.e., indicating that the \u201crobot\u201d is not understanding the photo\ncorrectly. We advised them to choose minor for small errors such as wrong or missing attributes or\nawkward prepositions, and go with major errors whenever any object or action naming is wrong.\nFor the next (more detailed, and thus more costly) round of annotation, we only select captions which\nare not marked as either perfect or acceptable in the \ufb01rst round. Since these captions contain errors,\nthe new annotator is required to provide detailed feedback about the mistakes. We found that some of\nthe annotators did not \ufb01nd errors in some of these captions, pointing to the annotator noise in the\nprocess. The annotator is shown the generated caption, delineating different phrases with the \u201c(\u201d and\n\u201c)\u201d tokens. We ask the annotator to 1) choose the type of required correction, 2) write feedback in\nnatural language, 3) mark the type of mistake, 4) highlight the word/phrase that contains the mistake,\n5) correct the chosen word/phrase, 6) evaluate the quality of the caption after correction. We allow\nthe annotator to submit the HIT after one correction even if her/his evaluation still points to errors.\nHowever, we plea to the good will of the annotators to continue in providing feedback. In the latter\ncase, we reset the webpage, and replace the generated caption with their current correction.\nThe annotator \ufb01rst chooses the type of error, i.e., something \u201c should be replaced\u201d, \u201cis missing\u201d, or\n\u201cshould be deleted\u201d. (S)he then writes a sentence providing feedback about the mistake and how\nit should be corrected. We require that the feedback is provided sequentially, describing a single\nmistake at a time. We do this by restricting the annotator to only select mistaken words within a single\nphrase (in step 4). In 3), the annotator marks further details about the mistake, indicating whether it\ncorresponds to an error in object, action, attribute, preposition, counting, or grammar. For 4) and 5)\nwe let the annotator highlight the area of mistake in the caption, and replace it with a correction.\nThe statistics of the data is provided in Table 2, with examples shown in Table 1. An interesting fact\nis that the feedback sentences in most cases mention both the wrong word from the caption, as well\nas the correction word. Fig. 3 (left) shows evaluation of the caption quality of the reference (MLE)\nmodel. Out of 9000 captions, 5150 are marked as containing errors (either semantic or grammar),\nand we randomly choose 4174 for the second round of annotation (detailed feedback). Fig. 3 (left)\nshows the quality of all the captions after correction, i.e. good reference captions as well as 4174\ncorrected captions as submitted by the annotators. Note that we only paid for one round of feedback,\nthus some of the captions still contained errors even after correction. Interestingly, on average the\nannotators still did 2.2 rounds of feedback per image (Table 2).\n3.3 Feedback Network\nOur goal is to incorporate natural language feedback into the learning process. The collected feedback\ncontains rich information of how the caption can be improved: it conveys the location of the mistake\nand typically suggests how to correct it, as seen in Table 2. This provides strong supervisory signal\nwhich we want to exploit in our RL framework. In particular, we design a neural network which will\nprovide additional reward based on the feedback sentence. We refer to it as the feedback network\n(FBN). We \ufb01rst explain our feedback network, and show how to integrate its output in RL.\n\n5\n\n\fSampled caption\nA cat on a sidewalk.\nA dog on a sidewalk.\nA cat on a sidewalk.\n\nFeedback\n\nThere is a dog on a sidewalk not a cat.\n\nPhrase\nA cat\nA dog\non a sidewalk\n\nPrediction\nwrong\ncorrect\nnot relevant\n\nTable 3: Example classif. of each phrase in a newly sampled caption into correct/wrong/not-relevant conditioned\non the feedback sentence. Notice that we do not need the image to judge the correctness/relevance of a phrase.\n\nNote that RL training will require us to generate samples (captions) from the model. Thus, during\ntraining, the sampled captions for each training image will change (will differ from the reference\nMLE caption for which we obtained feedback for). The goal of the feedback network is to read a\nnewly sampled caption, and judge the correctness of each phrase conditioned on the feedback. We\nmake our FBN to only depend on text (and not on the image), making its learning task easier. In\nparticular, our FBN performs the following computation:\n\n(1)\n(2)\n(3)\n(4)\n\nfsent\nfphrase\nff bn\n\nhcaption\nt\nhf eedback\nt\n\nt and wf\n\n3-layer MLP with dropout\n\n+3-way softmax\n\nLSTM, dim 256\nlinear+mean pool\n\n= fsent(hcaption\n= fsent(hf eedback\n\n, wc\nt )\n, wf\nt )\ni,1, . . . , wc\ni,N )\nT (cid:48), qi, m)\n\nt\u22121\nt\u22121\nqi = fphrase(wc\nT , hf\noi = ff bn(hc\nt denote the one-hot encoding of words in the sampled caption and feedback sentence,\nHere, wc\nrespectively. By wc\ni,\u00b7 we denote words in the i-th phrase of the sampled caption. FBN thus encodes\nboth the caption and feedback using an LSTM (with shared parameters), performs mean pooling over\nthe words in a phrase to represent the phrase i, and passes this information through a 3-layer MLP.\nThe MLP additionally accepts information about the mistake type (e.g., wrong object/action) encoded\nas a one hot vector m (denoted as \u201cextra information\u201d in Fig. 4). The output layer of the MLP is a\n3-way classi\ufb01cation layer that predicts whether the phrase i is correct, wrong, or not relevant (wrt\nfeedback sentence). An example output from FBN is shown in Table 3.\nImplementation details. We train our FBN with the ground-truth data that we collected.\nIn\nparticular, we use (reference, feedback, marked phrase in reference caption) as an example of a wrong\nphrase, (corrected sentence, feedback, marked phrase in corrected caption) as an example of the\ncorrect phrase, and treat the rest as the not relevant label. Reference here means the generated caption\nthat we collected feedback for, and marked phrase means the phrase that the annotator highlighted\nin either the reference or the corrected caption. We use the standard cross-entropy loss to train our\nmodel. We use ADAM [9] with learning rate 0.001, and a batch size of 256. When a reference\ncaption has several feedback sentences, we treat each one as independent training data.\n\n3.4 Policy Gradient Optimization using Natural Language Feedback\n\nWe follow [30, 29] to directly optimize for the desired image captioning metrics using the Policy\nGradient technique. For completeness, we brie\ufb02y summarize it here [30].\nOne can think of an caption decoder as an agent following a parameterized policy p\u03b8 that selects an\naction at each time step. An \u201caction\u201d in our case requires choosing a word from the vocabulary (for\nthe word RNN), or a phrase label (for the phrase RNN). An \u201cagent\u201d (our captioning model) then\nreceives the reward after generating the full caption, i.e., the reward can be any of the automatic\nmetrics, their weighted sum [30, 21], or in our case will also include the reward from feedback.\nThe objective for learning the parameters of the model is the expected reward received when com-\npleting the caption ws = (ws\nt is the word sampled from the model at time step t):\n\n1, . . . , ws\n\nT ) (ws\nL(\u03b8) = \u2212Ews\u223cp\u03b8 [r(ws)]\n\n(5)\nTo optimize this objective, we follow the reinforce algorithm [38], as also used in [30, 29]. The\ngradient of (5) can be computed as\n\n\u2207\u03b8L(\u03b8) = \u2212Ews\u223cp\u03b8 [r(ws)\u2207\u03b8 log p\u03b8(ws)],\n\nwhich is typically estimated by using a single Monte-Carlo sample:\n\u2207\u03b8L(\u03b8) \u2248 \u2212r(ws)\u2207\u03b8 log p\u03b8(ws)\n\n6\n\n(6)\n\n(7)\n\n\fWe follow [30] to de\ufb01ne the baseline b as the reward obtained by performing greedy decoding:\n\n\u02c6wt = arg max p(wt|ht)\n\nb = r( \u02c6w),\n\u2207\u03b8L(\u03b8) \u2248 \u2212(r(ws) \u2212 r( \u02c6w))\u2207\u03b8 log p\u03b8(ws)\n\n(8)\n\nNote that the baseline does not change the expected gradient but can drastically reduce its variance.\nReward. We de\ufb01ne two different rewards, one at the sentence level (optimizing for a performance\nmetrics), and one at the phrase level. We use human feedback information in both. We \ufb01rst de\ufb01ne the\nsentence reward wrt to a reference caption as a weighted sum of the BLEU scores:\n\n(cid:88)\n\nr(ws) = \u03b2\n\n\u03bbi \u00b7 BLEUi(ws, ref )\n\n(9)\n\ni\n\nIn particular, we choose \u03bb1 = \u03bb2 = 0.5, \u03bb3 = \u03bb4 = 1, \u03bb5 = 0.3. As reference captions to compute\nthe reward, we either use the reference captions generated by a snapshot of our model which were\nevaluated as not having minor and major errors, or ground-truth captions. The details are given in the\nexperimental section. We weigh the reward by the caption quality as provided by the annotators. In\nparticular, \u03b2 = 1 for perfect (or GT), 0.8 for acceptable, and 0.6 for grammar/\ufb02uency issues only.\nWe further incorporate the reward provided by the feedback network. In particular, our FBN allows\nus to de\ufb01ne the reward at the phrase level (thus helping with the credit assignment problem). Since\nour generated sentence is segmented into phrases, i.e., ws = wp\nt denotes the\n(sequence of words in the) t-th phrase, we de\ufb01ne the combined phrase reward as:\n\nP , where wp\n\n2 . . . wp\n\n1wp\n\nr(wp\n\n(10)\nNote that FBN produces a classi\ufb01cation of each phrase. We convert this into reward, by assigning\ncorrect to 1, wrong to \u22121, and 0 to not relevant. We do not weigh the reward by the con\ufb01dence of the\nnetwork, which might be worth exploring in the future. Our \ufb01nal gradient takes the following form:\n\nt ) = r(ws) + \u03bbf ff bn(ws, f eedback, wp\nt )\n\n\u2207\u03b8L(\u03b8) = \u2212 P(cid:88)\n\np=1\n\n(r(wp) \u2212 r( \u02c6wp))\u2207\u03b8 log p\u03b8(wp)\n\n(11)\n\nImplementation details. We use Adam with learning rate 1e\u22126 and batch size 50. As in [29], we\nfollow an annealing schedule. We \ufb01rst optimize the cross entropy loss for the \ufb01rst K epochs, then\nfor the following t = 1, . . . , T epochs, we use cross entropy loss for the \ufb01rst (P \u2212 f loor(t/m))\nphrases (where P denotes the number of phrases), and the policy gradient algorithm for the remaining\nf loor(t/m) phrases. We choose m = 5. When a caption has multiple feedback sentences, we take\nthe sum of the FBN\u2019s outputs (converted to rewards) as the reward for each phrase. When a sentence\ndoes not have any feedback, we assign it a zero reward.\n4 Experimental Results\nTo validate our approach we use the MS-COCO dataset [20]. We use 82K images for training, 2K for\nvalidation, and 4K for testing. In particular, we randomly chose 2K val and 4K test images from the\nof\ufb01cial validation split. To collect feedback, we randomly chose 7K images from the training set, as\nwell as all 2K images from our validation. In all experiments, we report the performance on our (held\nout) test set. For all the models (including baselines) we used a pre-trained VGG [33] network to\nextract image features. We use a word vocabulary size of 23,115.\nPhrase-based captioning model. We analyze different instantiations of our phrase-based caption-\ning in Table 4, showing the importance of predicting phrase labels. To sanity check our model we\ncompare it to a \ufb02at approach (word-RNN only) [39]. Overall, our model performs slightly worse\nthan [39] (0.66 points). However, the main strength of our model is that it allows a more natural\nintegration with feedback. Note that these results are reported for the models trained with MLE.\nFeedback network. As reported in Table 2, our dataset which contains detailed feedback (descrip-\ntions) contains 4173 images. We randomly select 9/10 of them to serve as a training set for our\nfeedback network, and use 1/10 of them to be our test set. The classi\ufb01cation performance of our FBN\nis reported in Table 5. We tried exploiting additional information in the network. The second line\nreports the result for FBN which also exploits the reference caption (for which the feedback was\nwritten) as input, represented with a LSTM. The model in the third line uses the type of error, i.e.\nthe phrase is \u201cmissing\u201d, \u201cwrong\u201d, or \u201credundant\u201d. We found that by using information about what\nkind of mistake the reference caption had (e.g., corresponding to misnaming an object, action, etc)\nachieves the best performance. We use this model as our FBN used in the following experiments.\n\n7\n\n\f\ufb02at (word level) with att\n\nphrase with att.\n\nBLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L Weighted metric\n65.36\n64.69\n65.46\n65.37\n\nphrase with att +phrase label\nphrase with 2 att +phrase label\nTable 4: Comparing performance of the \ufb02at captioning model [39], and different instantiations of our phrase-\nbased captioning model. All these models were trained using the cross-entropy loss.\n\n104.78\n102.14\n103.64\n104.12\n\n51.04\n50.80\n51.40\n50.90\n\n44.03\n43.37\n44.59\n44.02\n\n29.68\n28.80\n29.36\n29.51\n\n20.40\n19.31\n19.25\n19.91\n\nFeedback network\nno extra information\nuse reference caption\n\nuse \"missing\"/\"wrong\"/\"redundant\"\nuse \"action\"/\"object\"/\"preposition\"/etc\n\nAccuracy\n\n73.30\n73.24\n72.92\n74.66\n\nTable 5: Classi\ufb01cation results of our feedback\nnetwork (FBN) on a held-out feedback data. The\nFBN predicts correct/wrong/not relevant for each\nphrase in a caption. See text for details.\n\nRL with Natural Language Feedback.\nIn Table 6 we report the performance for several instan-\ntiations of the RL models. All models have been pre-trained using cross-entropy loss (MLE) on\nthe full MS-COCO training set. For the next rounds of training, all the models are trained only\non the 9K images that comprise our full evaluation+feedback dataset from Table 2. In particular,\nwe separate two cases. In the \ufb01rst, standard case, the \u201cagent\u201d has access to 5 captions for each\nimage. We experiment with different types of captions, e.g. ground-truth captions (provided by\nMS-COCO), as well as feedback data. For a fair comparison, we ensure that each model has access\nto (roughly) the same amount of data. This means that we count a feedback sentence as one source\nof information, and a human-corrected reference caption as yet another source. We also exploit\nreference (MLE) captions which were evaluated as correct, as well as corrected captions obtained\nfrom the annotators. In particular, we tried two types of experiments. We de\ufb01ne \u201cC\u201d captions as all\ncaptions that were corrected by the annotators and were not evaluated as containing minor or major\nerror, and ground-truth captions for the rest of the images. For \u201cA\u201d, we use all captions (including\nreference MLE captions) that did not have minor or major errors, and GT for the rest. A detailed\nbreak-down of these captions is reported in Table 7.\nWe \ufb01rst test a model using the standard cross-entropy loss, but which now also has access to the\ncorrected captions in addition to the 5GT captions. This model (MLEC) is able to improve over the\noriginal MLE model by 1.4 points. We then test the RL model by optimizing the metric wrt the 5GT\ncaptions (as in [30]). This brings an additional point, achieving 2.4 over the MLE model. Our RL\nagent with feedback is given access to 3GT captions, the \u201cC\" captions and feedback sentences. We\nshow that this model outperforms the no-feedback baseline by 0.5 points. Interestingly, with \u201cA\u201d\ncaptions we get an additional 0.3 boost. If our RL agent has access to 4GT captions and feedback\ndescriptions, we achieve a total of 1.1 points over the baseline RL model and 3.5 over the MLE\nmodel. Examples of generated captions are shown in Fig. 6.\nWe also conducted a human evaluation using AMT. In particular, Turkers are shown an image\ncaptioned by the baseline RL and our method, and are asked to choose the better caption. As shown\nin Fig. 5, our RL with feedback is 4.7 percent higher than the RL baseline. We additionally count\nhow much human interaction is required for either the baseline RL and our approach. In particular,\nwe count every interaction with the keyboard as 1 click. In evaluation, choosing the quality of the\ncaption counts as 1 click, and for captions/feedback, every letter counts as a click. The main save\ncomes from the \ufb01rst evaluation round, in which we only as for the quality of captions. Overall, there\nis almost half clicks saved in our setting.\nWe also test a more realistic scenario, in which the models have access to either a single GT caption,\nor in our case \u201cC\" (or \u201cA\u201d) and feedback. This mimics a scenario in which the human teacher observes\nthe agent and either gives feedback about the agent\u2019s mistakes, or, if the agent\u2019s caption is completely\nwrong, the teacher writes a new caption. Interestingly, RL when provided with the corrected captions\nperforms better than when given GT captions. Overall, our model outperforms the base RL (no\nfeedback) by 1.2 points. We note that our RL agents are trained (not counting pre-training) only on a\nsmall (9K) subset of the full MS-COCO training set. Further improvements are thus possible.\nDiscussion. These experiments make an important point. Instead of giving the RL agent a com-\npletely new target (caption), a better strategy is to \u201cteach\u201d the agent about the mistakes it is doing\nand suggest a correction. Natural language thus offers itself as a rich modality for providing such\nguidance not only to humans but also to arti\ufb01cial agents.\n\n8\n\n\fTable 6: Comparison of our RL with feedback information to baseline RL and MLE models.\n\nMLE (5 GT)\n\nMLEC (5 GT + C)\nMLEC (5 GT + A)\n\nRLB (5 GT)\n\nRLF (3GT+FB+C)\nRLF (3GT+FB+A)\nRLF (4GT + FB)\n\nRLB (1 GT)\n\nRLB (C)\nRLB (A)\n\nRLF (C + FB)\nRLF (A + FB)\n\n.\nt\nn\ne\ns\n5\n\n.\nt\nn\ne\ns\n1\n\nBLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L Weighted metric\n65.37\n66.85\n66.14\n66.90\n66.52\n66.98\n67.10\n65.68\n65.84\n65.81\n65.76\n66.23\n\n104.12\n105.58\n105.47\n106.55\n107.02\n107.31\n107.67\n104.93\n105.50\n105.31\n106.03\n106.12\n\n50.90\n51.20\n51.32\n51.10\n51.41\n51.54\n51.30\n51.07\n51.06\n51.28\n51.35\n51.58\n\n44.02\n45.19\n44.87\n45.10\n45.23\n45.54\n45.50\n44.58\n44.64\n44.58\n44.65\n45.00\n\n19.91\n19.79\n20.27\n20.30\n20.66\n20.53\n20.30\n19.97\n20.23\n20.24\n20.62\n20.34\n\n29.51\n29.89\n30.17\n30.10\n30.48\n30.52\n30.60\n29.81\n30.01\n29.87\n30.20\n30.15\n\nGT: ground truth captions; FB: feedback; MLE(A)(C): MLE model using \ufb01ve GT sentences + either C or A\ncaptions (see text and Table 7); RLB: baseline RL (no feedback network); RLF: RL with feedback (here we\n\nalso use C or A captions as well as FBN);\n\nground-truth perfect acceptable grammar error only\n\nA\nC\n\n3107\n6326\n\n2661\n1502\n\n2790\n1502\n\n442\n234\n\nTable 7: Detailed break-down of what\ncaptions were used as \u201cA\u201d or \u201cC\u201d in Table 6\nfor computing additional rewards in RL.\n\n(a)\n\n(b)\n\nFigure 5: (a) Human preferences: RL baseline vs RL with feedback (our approach), (b) Number of human\n\u201cclicks\u201d required for MLE/baseline RL, and ours. A click is counted when an annotator hits the keyboard: in\nevaluation, choosing the quality of the caption counts as 1 click, and for captions/feedback, every letter counts as\na click. The main save comes from the \ufb01rst evaluation round, in which we only as for the quality of captions.\n\nMLE: ( a man ) ( walking ) ( in front of a\nbuilding ) ( with a cell phone . )\nRLB: ( a man ) ( is standing ) ( on a sidewalk )\n( with a cell phone . )\nRLF: ( a man ) ( wearing a black suit ) ( and\ntie ) ( on a sidewalk . )\n\nMLE: ( a clock tower ) ( with a clock ) ( on\ntop . )\nRLB: ( a clock tower ) ( with a clock ) ( on top\nof it . )\nRLF: ( a clock tower ) ( with a clock ) ( on the\nfront . )\n\nMLE: ( two giraffes ) ( are standing ) ( in a\n\ufb01eld ) ( in a \ufb01eld . )\nRLB: ( a giraffe ) ( is standing ) ( in front of a\nlarge building . )\nRLF: ( a giraffe ) ( is ) ( in a green \ufb01eld ) ( in a\nzoo . )\n\nMLE: ( two birds ) ( are standing ) ( on the\nbeach ) ( on a beach . )\nRLB: ( a group ) ( of birds ) ( are ) ( on the\nbeach . )\nRLF: ( two birds ) ( are standing ) ( on a beach\n) ( in front of water . )\n\nFigure 6: Qualitative examples of captions from the MLE and RLB models (baselines), and our RBF model.\n\n5 Conclusion\nIn this paper, we enable a human teacher to provide feedback to the learning agent in the form of\nnatural language. We focused on the problem of image captioning. We proposed a hierarchical\nphrase-based RNN as our captioning model, which allowed natural integration with human feedback.\nWe crowd-sourced feedback for a snapshot of our model, and showed how to incorporate it in Policy\nGradient optimization. We showed that by exploiting descriptive feedback our model learns to\nperform better than when given independently written captions.\n\nAcknowledgment\nWe gratefully acknowledge the support from NVIDIA for their donation of the GPUs used for this research. This\nwork was partially supported by NSERC. We also thank Relu Patrascu for infrastructure support.\n\n9\n\n47.7 52.3 0 10 20 30 40 50 60 RLB RLF Human preferences 0 50000 100000 150000 200000 250000 300000 350000 400000 RLB RLF # of clicks \fReferences\n[1] Cmu\u2019s herb robotic platform, http://www.cmu.edu/herb-robot/.\n[2] Microsoft\u2019s tay, https://twitter.com/tayandyou.\n[3] Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. Towards diverse and natural image descriptions via\n\na conditional gan. In arXiv:1703.06029, 2017.\n\n[4] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra. Visual dialog. In\n\narXiv:1611.08669, 2016.\n\n[5] Shane Grif\ufb01th, Kaushik Subramanian, Jonathan Scholz, Charles L. Isbell, and Andrea Lockerd Thomaz.\n\nPolicy shaping: Integrating human feedback with reinforcement learning. In NIPS, 2013.\n\n[6] K. Judah, S. Roy, A. Fern, and T. Dietterich. Reinforcement learning via practice and critique advice. In\n\nAAAI, 2010.\n\n[7] Russell Kaplan, Christopher Sauer, and Alexander Sosa. Beating atari with natural language guided\n\nreinforcement learning. In arXiv:1704.05539, 2017.\n\n[8] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR,\n\n2015.\n\n[9] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\n[10] Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. Unifying visual-semantic embeddings with\n\nmultimodal neural language models. CoRR, abs/1411.2539, 2014.\n\n[11] W. Bradley Knox, Cynthia Breazeal, , and Peter Stone. Training a robot via human feedback: A case study.\n\nIn International Conference on Social Robotics, 2013.\n\n[12] W. Bradley Knox and Peter Stone. Reinforcement learning from simultaneous human and mdp reward. In\n\nIntl. Conf. on Autonomous Agents and Multiagent Systems, 2012.\n\n[13] Jacqueline Kory Westlund, Jin Joo Lee, Luke Plummer, Fardad Faridi, Jesse Gray, Matt Berlin, Harald\nQuintus-Bosz, Robert Hartmann, Mike Hess, Stacy Dyer, Kristopher dos Santos, Sigurdhur \u00d6rn Ad-\nhalgeirsson, Goren Gordon, Samuel Spaulding, Marayna Martinez, Madhurima Das, Maryam Archie,\nSooyeon Jeong, and Cynthia Breazeal. Tega: A social robot. In International Conference on Human-Robot\nInteraction, 2016.\n\n[14] Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating\n\ndescriptive image paragraphs. In CVPR, 2017.\n\n[15] G. Kuhlmann, P. Stone, R. Mooney, and J. Shavlik. Guiding a reinforcement learner with natural language\nadvice: Initial results in robocup soccer. In AAAI Workshop on Supervisory Control of Learning and\nAdaptive Systems, 2004.\n\n[16] Remi Lebret, Pedro O. Pinheiro, and Ronan Collobert.\n\narXiv:1502.03671, 2015.\n\nPhrase-based image captioning.\n\nIn\n\n[17] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor\n\npolicies. J. Mach. Learn. Res., 17(1):1334\u20131373, 2016.\n\n[18] Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc\u2019Aurelio Ranzato, and Jason Weston. Dialogue learning\n\nwith human-in-the-loop. In arXiv:1611.09823, 2016.\n\n[19] Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. Deep reinforcement\n\nlearning for dialogue generation. In arXiv:1606.01541, 2016.\n\n[20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r,\nand C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740\u2013755. 2014.\n[21] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via\n\npolicy gradient optimization of spider. In arXiv:1612.00370, 2016.\n\n[22] Richard Maclin and Jude W. Shavlik. Incorporating advice into agents that learn from reinforcements. In\n\nNational Conference on Arti\ufb01cial Intelligence, pages 694\u2013699, 1994.\n\n[23] Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David\n\nMcClosky. The Stanford CoreNLP natural language processing toolkit. In ICLR, 2014.\n\n[24] Maja J. Matari\u02c7c. Socially assistive robotics: Human augmentation vs. automation. Science Robotics, 2(4),\n\n2017.\n\n[25] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare,\nAlex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie,\nAmir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis\nHassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529\u2013533, 2015.\n\n10\n\n\f[26] Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations:\n\nTheory and application to reward shaping. In ICML, pages 278\u2013287, 1999.\n\n[27] Amar Parkash and Devi Parikh. Attributes for classi\ufb01er feedback. In European Conference on Computer\n\nVision (ECCV), 2012.\n\n[28] Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent\nneural networks. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55\u201360,\n2014.\n\n[29] Marc\u2019Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with\n\nrecurrent neural networks. In arXiv:1511.06732, 2015.\n\n[30] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical\n\nsequence training for image captioning. In arXiv:1612.00563, 2016.\n\n[31] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche,\nJulian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik\nGrewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray\nKavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks\nand tree search. Nature, 529(7587):484\u2013489, 2016.\n\n[32] Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, and Raquel Urtasun. Neuroaesthetics in fashion:\n\nModeling the perception of beauty. In CVPR, 2015.\n\n[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-\n\ntion. 23, 2015.\n\n[34] Ying Hua Tan and Chee Seng Chan. phi-lstm: A phrase-based hierarchical lstm model for image captioning.\n\nIn ACCV, 2016.\n\n[35] A. Thomaz and C. Breazeal. Reinforcement learning with human teachers: Evidence of feedback and\n\nguidance. In AAAI, 2006.\n\n[36] Oriol Vinyals and Quoc Le. A neural conversational model. In arXiv:1506.05869, 2015.\n[37] Jason Weston. Dialog-based language learning. In arXiv:1604.06045, 2016.\n[38] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement\n\nlearning. In Machine Learning, 1992.\n\n[39] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard\nZemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention.\nIn ICML, 2015.\n\n[40] Yanchao Yu, Arash Eshghi, and Oliver Lemon. Training an adaptive dialogue policy for interactive learning\n\nof visually grounded word meanings. In Proc. of SIGDIAL, 2016.\n\n[41] Yanchao Yu, Arash Eshghi, Gregory Mills, and Oliver Lemon. The burchak corpus: a challenge data set\nfor interactive learning of visually grounded word meanings. In Workshop on Vision and Language, 2017.\n\n11\n\n\f", "award": [], "sourceid": 2623, "authors": [{"given_name": "huan", "family_name": "ling", "institution": "university of toronto"}, {"given_name": "Sanja", "family_name": "Fidler", "institution": "University of Toronto"}]}