{"title": "Dialog-to-Action: Conversational Question Answering Over a Large-Scale Knowledge Base", "book": "Advances in Neural Information Processing Systems", "page_first": 2942, "page_last": 2951, "abstract": "We present an approach to map utterances in conversation to logical forms, which will be executed on a large-scale knowledge base. To handle enormous ellipsis phenomena in conversation, we introduce dialog memory management to manipulate historical entities, predicates, and logical forms when inferring the logical form of current utterances. Dialog memory management is embodied in a generative model, in which a logical form is interpreted in a top-down manner following a small and flexible grammar. We learn the model from denotations without explicit annotation of logical forms, and evaluate it on a large-scale dataset consisting of 200K dialogs over 12.8M entities. Results verify the benefits of modeling dialog memory, and show that our semantic parsing-based approach outperforms a memory network based encoder-decoder model by a huge margin.", "full_text": "Dialog-to-Action: Conversational Question\n\nAnswering Over a Large-Scale Knowledge Base\n\nDaya Guo1\u2217, Duyu Tang2, Nan Duan2, Ming Zhou2, and Jian Yin1\n1 The School of Data and Computer Science, Sun Yat-sen University.\n\nGuangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, P.R.China\n\n2 Microsoft Research Asia, Beijing, China\n\n{guody5@mail2,issjyin@mail}.sysu.edu.cn\n{dutang,nanduan,mingzhou}@microsoft.com\n\nAbstract\n\nWe present an approach to map utterances in conversation to logical forms, which\nwill be executed on a large-scale knowledge base. To handle enormous ellipsis phe-\nnomena in conversation, we introduce dialog memory management to manipulate\nhistorical entities, predicates, and logical forms when inferring the logical form of\ncurrent utterances. Dialog memory management is embodied in a generative model,\nin which a logical form is interpreted in a top-down manner following a small and\n\ufb02exible grammar. We learn the model from denotations without explicit annotation\nof logical forms, and evaluate it on a large-scale dataset consisting of 200K dialogs\nover 12.8M entities. Results verify the bene\ufb01ts of modeling dialog memory, and\nshow that our semantic parsing-based approach outperforms a memory network\nbased encoder-decoder model by a huge margin.\n\n1\n\nIntroduction\n\nWe consider the problem of mapping conversational natural language questions to formal represen-\ntations (e.g., logical form) of their underlying meanings, which would be executed to produce the\nanswer (denotation) [1\u20137]. We study the problem in a realistic setting that (1) only denotations are\navailable for model training while the underlying logical forms remain unknown, and (2) logical\nforms will be executed on a large-scale knowledge base (KB) consisting of tens of millions of entities.\nWe believe that KB-based conversational question answering plays an important role in both search\nengines and intelligent personal assistants (e.g., Siri, Alexa, Cortana/Xiaoice, and Google Now) [8]\nto improve the ability of multi-turn question answering.\nThe major challenge of this task is how to interpret the meaning of an utterance in interaction where\nellipsis phenomena are frequently encountered. Let\u2019s consider the example in Figure 1. The ellipsis\nof the entity \u201che\u201d in Q2 refers to \u201cPresident of the United States\u201d in Q1. The ellipsis of the entity \u201cit\u201d\nin Q3 means the answer R2. In Q4, the ellipsis of the predicate (\u201cyearEstablished\u201d) comes from Q3.\nWe see that understanding the meaning of conversational utterances requires a good understanding of\ndialog history. Another challenge is how to ef\ufb01ciently learn the semantic parser from denotations.\nOnline learning by searching legitimate logical forms requires repeated execution on a large-scale\nknowledge base, which is extremely time-consuming and intolerable.\nIn this work, we regard the generation of a logical form as the prediction of a sequence of actions\n[9, 10, 6, 11\u201316], each of which corresponds to a derivation rule in a simple and \ufb02exible grammar. We\nintroduce a generative model that interprets the logical form of an utterance in a top-down manner. A\ngrammar-guided decoder is developed to generate possible action sequences following the grammar.\n\n\u2217Work done while this author was an intern at Microsoft Research.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: An example illustrating the task of conversational question answering.\n\nTo cope with ellipsis phenomena in conversation, we introduce a dialog memory management\ncomponent that leverages historical entities, predicates, and action subsequences when generating the\nlogical form for a current utterance. To avoid the time-consuming procedure of repeatedly executing\non a large-scale knowledge base during training [6], we conduct a breadth-\ufb01rst-search algorithm in\nadvance to obtain pairs of utterances and their action sequences that lead to correct answers. The\nmodel is learned by maximizing the likelihood of generating the expected action sequences [5, 11].\nWe conduct experiments on a large-scale dataset [17] for conversation question answering, which\nconsists of 200K dialogs with 1.6M turns over 12.8M entities from Wikidata. Compared to a memory\nnetwork enhanced encoder-decoder method [17], our semantic parsing-based approach achieves\nbetter performance. We show the bene\ufb01ts of using dialog memory, and observe that our approach\nperforms well on those questions which rely on dialog contexts for resolving ellipsis phenomena.\n\n2 Problem Statement\n\nOur goal is to answer questions (utterances) in conversations based on a large-scale open-domain\nknowledge base (KB). We tackle the problem in a semantic parsing manner that \ufb01rst maps the\nquestion into executable logical forms, and then executes the generated logical form on a KB to\nproduce the answer. We would like to learn the semantic parser from denotations, having no luxury of\naccess to the annotated logical form for each utterance. Formally, let I be an interaction consisting of\nn utterances/questions {q1, q2, ..., qn}2. During training, each question qi is paired with the correct\nanswer ai, without explicit annotation of the correct logical form zi. In the inference process, a\ni is derived based on the current question qi and its preceding questions {q1, q2, .., qi\u22121}.\nlogical form z(cid:48)\ni on knowledge base K produces the outcome a(cid:48)\nExecuting z(cid:48)\ni.\n\n3 Grammar\n\nIn this section, we describe the actions we de\ufb01ne in this work for generating logical forms. A\nsummary of all the actions are given in Table 1.\n\nNote\n\nTable 1: The base actions we use in this work for generating logical forms.\nOperation\nstart \u2192 set|num|bool\nset \u2192 f ind(set, r)\nnum \u2192 count(set)\nbool \u2192 in(e, set)\nset \u2192 union(set1, set2)\nset \u2192 inter(set1, set2)\nset \u2192 dif f (set1, set2)\nset \u2192 larger(set, r, num)\nset \u2192 less(set, r, num)\nset \u2192 equal(set, r, num)\nset \u2192 argmax(set, r)\nset \u2192 argmin(set, r)\nset \u2192 {e}\ne|r|num \u2192 constant\nset|num|bool \u2192 actioni\u22121\n\nset of entities with a r edge to e\ntotal number of set\nwhether e is in set\nunion of set1 and set2\nintersection of set1 and set2\ninstances included in set1 but not included in set2\nsubset of set linking to more than num entities with relation r\nsubset of set linking to less than num with relation r\nsubset of set linking to num entities with relation r\nsubset of set linking to most entities with relation r\nsubset of set linking to least entities with relation r\n\ninstantiation for entity e, predicate r or number num\nreplicate previous action subsequence (w/o or w/ instantiation)\n\nAction\nA1-A3\n\nA4\nA5\nA6\nA7\nA8\nA9\nA10\nA11\nA12\nA13\nA14\nA15\n\nA16-A18\nA19-A21\n\n2We use the terms utterance and question interchangeably.\n\n2\n\nQ1: Where was the President of the United States born?R3:1881Q2:Where did he graduate from?R1:New York CityR2:Wharton School of the University of PennsylvaniaQ3: What year was it established?R4:1636Q4: How about Harvard university?FreebaseDBpediaYAGONELLOpenIE/ReVerb\fAnalogous to the meaning representation of [9], each action in this work consists of three components:\na semantic category, a function symbol which might be omitted, and a list of arguments. An argument\ncan be a semantic category, a constant, or an action subsequence. Take A5 for example: it has a\nsemantic category num, a function symbol count, and a semantic category set as the only argument.\nWe add A16-A18 to instantiate entity e, predicate r, and number num, respectively. We add A19-A21\nfor replicating a subsequence of previously predicated action sequence from the dialog memory. The\nderivation of a logical form starts from the semantic category start. As derivation processes, the\nmodel recursively rewrites the leftmost nonterminal (i.e semantic category) in the logical form by\napplying a legitimate action. The parsing process terminates until no nonterminals remain.\n\n4 Dialog-to-Action\n\nWe describe our semantic parsing-based model in this section. Based on sequence-to-sequence\nlearning [18, 19], the model takes a question and its context from interaction history as the input and\ngenerates an action sequence. We develop a grammar-guided decoder to control the generation of an\naction sequence, and a dialog memory management component to leverage historical contexts.\n\n4.1 Encoder\n\nFigure 2 illustrates an overview of the proposed model. Since previous questions and answer-\ns/responses in conversation are useful contexts, we concatenate them with the current question as an\ninput x = (x1, ..., xT ). A bidirectional RNN with a gated recurrent unit (GRU) [20] is used as the\nencoder to convert the input to a sequence of context vector. The forward RNN reads the input in\n\u2212\u2192\nleft-to-right direction, obtaining hidden states (\nhT ). The backward RNN reads reversely and\n\u2190\u2212\nhT ). We then get the \ufb01nal representation (h1, ..., hT ) for each word in the source\noutputs (\n\u2190\u2212\nsequence, where hj = [\nh1]) is used as\ninitial hidden state of the decoder.\n\n\u2190\u2212\nhj]. The representation of the source sequence hx = ([\n\n\u2190\u2212\nh1, ...,\n\n\u2212\u2192\nh1, ...,\n\n\u2212\u2192\nhT ;\n\n\u2212\u2192\nhj;\n\n4.2 Grammar-guided Decoder\n\nWe use GRU with an attention mechanism as a decoder, which generates an action sequence a1, ..., an\nin a sequential way. As we can see from Figure 2, the decoder parses the dialog to an action sequence,\nwhich corresponds to the parsing tree shown in the lower right side. At each time-step t, we apply an\nattention mechanism to obtain the context vector ct that is computed in the same way as [21]. The\nconcatenation of the context vector ct, the last hidden state sdec\nt\u22121 and the embedding vt\u22121 of previously\npredicted action is fed to the decoder to get the current hidden state sdec\nt\u22121, vt\u22121, ct). If\nthe previously predicted action is an instantiated action, the embedding vt\u22121 is the representation\nis used with the same attention mechanism\nof the selected constant. The current hidden state sdec\nover the inputs to get the context vector sc\nt to get \ufb01nal hidden states\nst. In order to generate a valid logical form, we incorporate an action-constrained grammar to \ufb01lter\nillegal actions. An action is legitimate if its left-hand semantic category is the same as the leftmost\nnonterminal in the partial logical form parsed so far. We denote the set of legitimate actions at the\ntime step t as At = {a1, ..., aN}. The probability distribution over the set is calculated as Equation\n1, where vi is the one-hot indicator vector for ai, Wa is model parameter, and a<t stands for the\npreceding actions of the t-th time step.\n\nt. We then concatenate sdec\n\nt = GRU (sdec\n\nand sc\n\nt\n\nt\n\n(1)\n\np(ai|a<t, x) =\n\n4.3 Dialog Memory\n\n(cid:80)\n\nexp(vT\n\ni Wast)\n\naj\u2208At\n\nexp(vT\n\nj Wast)\n\nInteraction history is very important to generate the logical form of the following utterance. Therefore,\nwe incorporate a dialog memory to maintain information from interaction history. As illustrated in\nFigure 2, the dialog memory includes three types of information, including entities, predicates, and\naction subsequences. We describe these aspects one after another.\nEntity We consider two types of entities from interaction history, coming from the previous question\nutterance and the previous answer, respectively. The \ufb01rst type is suitable for a common co-reference\n\n3\n\n\fFigure 2: An illustration of the proposed approach. Our approach is the encoder-decoder structure\nwith dialog memory management component. The lower right side is a parsing tree corresponding to\nthe action sequence generated by the decoder.\n\ncase where the ellipsis entity comes from the previous utterance, such as \u201cQ1: Where was the\nPresident of the United States born\u201d, \u201cQ2: Where did he graduate from\u201d. The second type is suitable\nfor the case in ellipsis entities comes from the previous answer, such as Q3 and R2 in Figure 1.\nPredicate We record the predicates of the previous utterance. This is useful for the scenario where the\nellipsis of the predicate occurs. Let us take Q3 and Q4 from Figure 1 as an example. The predicate\n\u201cyearEstablished\u201d is not explicitly expressed in Q4, yet mentioned in Q3.\nAction Subsequences An action subsequence could be roughly categorized as instantiated or not.\nIndeed, an action subsequence with instantiation stands for a full or a partial logical form. For\nexample, the \ufb01rst action subsequence in the dialog memory of Figure 2 is identical to the logical\nform f ind(U nitedStates, isP residentOf ), which means the president of the United States. The\nellipsis of the entity \u201che\u201d in the current question \u201cWhere did he graduate from?\u201d actually refers to\nthe president of the United States. Therefore, the model executes an action (i.e. A19) to replicate the\n\ufb01rst action subsequence. An action subsequence without instantiation conveys the soft pattern of a\nlogical form. For example, the current question \u201cAnd how about China?\u201d has the same soft pattern\nas the previous question, but the country mentioned in the previous question should be replaced by\n\u201cChina\u201d.\nFor more details on establishment of the dialog memory, see Appendix A.\n\n4.4\n\nIncorporating Dialog Memory\n\nIn this section, we present our strategy to replicate contents from dialog memory as decoding\nprocesses. This has an in\ufb02uence on A16-A21, which we would list as follows.\nInstantiation: We allow instantiated actions (i.e. A16-A18) to access to the dialog memory when\nthe decoder instantiates an entity, predicate, or number. Taking entities as an example. Each entity is\nassigned one of three tags: previous question, previous answer, or current question. The probability of\nan entity et being instantiated at time-step t is calculated as Equation 2, where pg(\u00b7) is the probability\nof the tag gt to be chosen, and pe(\u00b7) is the probability distribution over entities for each tag.\n\np(et|a<t, x) = pe(et|gt, a<t, x)pg(gt|a<t, x)\n\n(2)\nThe probability distribution of entities pe(\u00b7) is calculated as Equation 3, where ve is the embedding of\nentity et, We is model parameter, and Egt is the set of entities having tag gt. The probability pg(\u00b7) is\nimplemented by a linear layer followed by a softmax function, and the input is st. The instantiations\nof predicates and numbers are similar to entities, except that predicates have two kinds of tags (i.e.\n\n4\n\nDialog MemoryEntity{United States, tag=utterance}{New York City, tag=answer}Predicate {isPresidentOf}{placeOfBirth}ActionSubsequence\ud835\udc60\ud835\udc52\ud835\udc61\u2192\ud835\udc344\ud835\udc3415\ud835\udc52\ud835\udc48\ud835\udc46\ud835\udc5f\ud835\udc5d\ud835\udc5f\ud835\udc52\ud835\udc60\ud835\udc60\ud835\udc52\ud835\udc61\u2192\ud835\udc344\ud835\udc3415\ud835\udc60\ud835\udc52\ud835\udc61\u2192\ud835\udc344\ud835\udc344\ud835\udc3415\ud835\udc52\ud835\udc48\ud835\udc46\ud835\udc5f\ud835\udc5d\ud835\udc5f\ud835\udc52\ud835\udc60\ud835\udc5f\ud835\udc4f\ud835\udc61\u210e\ud835\udc60\ud835\udc52\ud835\udc61\u2192\ud835\udc344\ud835\udc344\ud835\udc3415Where did president of the United States born?New York CityWhere did he graduate from?\ud835\udc5f\ud835\udc54\ud835\udc5f\ud835\udc4e\ud835\udc51\ud835\udc5f\ud835\udc54\ud835\udc5f\ud835\udc4e\ud835\udc51\ud835\udc52\ud835\udc5b\ud835\udc51\ud835\udc344\ud835\udc3419\ud835\udc344\ud835\udc3415\ud835\udc3419\ud835\udc52\ud835\udc48\ud835\udc46\ud835\udc5f\ud835\udc5d\ud835\udc5f\ud835\udc52\ud835\udc60replicated action sequence w/ instantiationPrevious Question Previous Answer Current Question SsetA1find(set, r1)A4graduateFromA17find(set, r2)A4{e}United StatesA15A16isPresidentOfA17\ud835\udc46\ud835\udc341\ud835\udc341\ud835\udc344copy\fprevious question and current question) and numbers have only one tag (i.e current question).\n\npe(et|gt, a<t, x) =\n\nexp(vT\n\ne tanh(West))\nexp(vT\n\ne(cid:48)tanh(West))\n\ne(cid:48)\u2208Egt\n\n(cid:80)\n\n(3)\n\n(5)\n\nReplication: The model learns to copy a previous action subsequence through choosing A19-A21.\nIt has two modes that replicate instantiated or non-instantiated action subsequences. Figures 2\nillustrates how the model replicates instantiated action subsequences. In order to obtain instantiated\naction subsequences of the previous question, we parse the whole previous logical form to a tree and\nenumerate all subtrees, each of which corresponds to an instantiated action subsequence. Another\nmode will be described in appendix B. In our model, the probability of a subsequence to be copied is\ncalculated as Equation 4, where pm(\u00b7) is the probability of the mode mt to be chosen, and ps(\u00b7) is\nthe probability distribution over subsequences for each mode.\n\np(subt|a<t, x) = ps(subt|mt, a<t, x)pm(mt|a<t, x)\n\n(4)\nThe probability of copying the subsequence subt, namely ps(subt|mt, a<t, x), is calculated as\nfollows, where vsub is the representation of subt, and Emt is the set of subsequences given mode mt.\nvsub is obtained by encoding subt using a GRU. The calculation of pm(\u00b7) is analogous to pg(\u00b7).\n\nps(subt|mt, a<t, x) =\n\n(cid:80)\n\nexp(vT\n\nsi\u2208Emt\n\nsubtanh(Wsst))\nexp(vT\n\nsitanh(Wsst))\n\nAfter replicating a subsequence action, the decoder clamps the generation of subsequence length by\ncontinuously feeding the subsequence actions one by one. In the inference process, we obtain action\nsubsequences from the predicted logical form with the highest score. Error propagation might occur\nwhen the model replicates an incorrect previous logical form, which hurts performance. Therefore,\nwe consider the score of action subsequence as a degree of con\ufb01dence, which is calculated in the\nsame way as the probability of action subsequences without replication.\n\n5 Learning and Inference\n\nAt the training phase, instances from training data are labeled with answers while action sequences\nremain unknown. In order to train our model, we \ufb01rst generate action sequences for each example,\nand then use an approximate marginal log-likelihood as the objective function. We use a breadth-\n\ufb01rst-search algorithm from root to generate a set of action sequences Sa that are executed to the\ncorrect answer. To cover the replication of action subsequences from dialog memory, we regard\naction subsequences in Sa which appear in the dialog memory as replicated action subsequences.\nIn order to guarantee the quality of training instances with replication actions, we have a constraint\nthat at least one instantiated constant should be the same. The objective function is the sum of log\nprobabilities of actions, instantiations, and replications, where \u03b4(ins, at) is 1 if at is an instantiation\naction otherwise 0, and \u03b4(rep, at) is the same as \u03b4(ins, at), where rep means a replication action.\n\u03b4(rep, at)logp(subt|a<t, x)\n(6)\nWe use beam search at the inference phase. For more details on the training and inference procedures\nused in the experiments, see Appendix C.\n\n\u03b4(ins, at)logp(et|a<t, x) \u2212(cid:88)\n\nlogp(at|a<t, x) \u2212(cid:88)\n\nloss = \u2212(cid:88)\n\nt\n\nt\n\nt\n\n6 Experiment\n\nWe conduct the experiment on the CSQA dataset3. The dataset is created based on Wikidata4,\nincluding 152K dialogs for training, and 16K/28K dialogs for development/testing. Questions in\ndialogs are classi\ufb01ed as kinds of types, examples of which are shown in Figure 3. We use the same\nevaluation metrics employed in [17]. Precision and recall are used as evaluation metrics for questions\nwhose answers are entities, which measures the percentage of correct entities in the output and\nthe percentage of correct entities that are retrieved, respectively. Accuracy is used to measure the\nperformance for questions which produce boolean and numerical answers.\n\n3https://amritasaha1812.github.io/CSQA/\n4https://www.wikidata.org\n\n5\n\n\fTable 2: Performance of different approaches on the CSQA dataset.\n\nMethods\nQuestion Type\nOverall\nSimple Question (Direct)\nSimple Question (Co-referenced)\nSimple Question (Ellipsis)\nLogical Reasoning (All)\nQuantitative Reasoning (All)\nComparative Reasoning (All)\nClari\ufb01cation\nQuestion Type\nVeri\ufb01cation (Boolean)\nQuantitative Reasoning (Count)\nComparative Reasoning (Count)\n\nHRED+KVmem\nPrecision\nRecall\n6.30%\n18.40%\n8.58%\n33.30%\n5.09%\n12.67%\n6.98%\n17.30%\n5.75%\n15.11%\n1.01%\n0.91%\n2.11%\n4.97%\n25.09% 12.13%\n\nAccuracy\n21.04%\n12.13%\n8.67%\n\nContxIndp-SP\n\nRecall\nPrecision\n42.18% 40.88%\n94.04% 88.32%\n40.29% 38.55%\n14.09% 13.28%\n36.23% 35.91%\n43.75% 49.91%\n41.49% 38.91%\n0.01%\n0.01%\n\nAccuracy\n20.38%\n30.60%\n15.54%\n\nD2A\n\nRecall\nPrecision\n64.04% 61.76%\n93.67% 89.26%\n71.31% 68.41%\n86.58% 77.85%\n42.49% 44.82%\n48.59% 52.03%\n44.73% 43.69%\n19.36% 17.36%\n\nAccuracy\n45.05%\n40.94%\n17.78%\n\n6.1 Model Comparisons\n\nTable 2 shows the results of different methods on CSQA data. HRED+KVmem [17] is a sequence-\nto-sequence learning method, which uses a hierarchical encoder and a key-value memory network\n[22] to compute the representation for the question and its contexts, and then uses an RNN as\nthe decoder to directly produce answers. To demonstrate the effectiveness of dialog memory, we\nimplement a context-independent semantic parser ContxIndp-SP, in which the dialog memory is\ntotally removed from the full Dialog-to-Action model. Our full model is abbreviated as D2A (short\nfor Dialog-to-Action).\nOur approach is a semantic parsing based method, which explicitly manipulates the actions/functions\nand lets the Seq2Seq model learn how these actions are used to derive the logical form of the question.\nIt could naturally leverage parsed results of previous turn including entities, predicates and action\nsubsequences to handle various ellipsis phenomena. HRED+KVmem is a text generation based\napproach that puts the entire burden of doing reasoning and compositionality to the Seq2Seq model,\nwhich struggles at handling all these problems in an implicit way. Results demonstrate that namely\nsemantic parsing approach is more effective to handle complex questions, including quantitative,\ncomparative and logical reasoning. We can also see that incorporating the dialog memory brings\nsigni\ufb01cant improvements in co-referenced and ellipsis categories. The results also show that the\ndialog memory is very important to handle ellipsis phenomena in conversation.\n\n6.2 Model Analysis\n\nWe conduct ablation analysis to better understand how various components in the dialog memory\nimpact overall performance. We remove entity memory (EM), predicate memory (PM) and action\nsubsequence memory (AM), respectively, to analyze their contribution.\n\nD2A w/o PM\n\nD2A w/o AM\n\nTable 3: Performance of different approaches on the CSQA dataset. EM, PM and AM stand for\nentities, predicates, and subsequent action sequences from dialog memory, respectively.\nMethods\nQuestion Type\nOverall\nSimple Question (Direct)\nSimple Question (Co-referenced)\nSimple Question (Ellipsis)\nLogical Reasoning (All)\nQuantitative Reasoning (All)\nComparative Reasoning (All)\nClari\ufb01cation\nQuestion Type\nVeri\ufb01cation (Boolean)\nQuantitative Reasoning (Count)\nComparative Reasoning (Count)\n\nRecall\nPrecision\n57.52% 56.20%\n93.39% 88.76%\n70.42% 67.89%\n15.35% 13.73%\n38.20% 42.37%\n44.18% 48.30%\n39.40% 38.58%\n0.86%\n1.11%\n\nRecall\nPrecision\n64.02% 62.85%\n93.55% 88.63%\n73.36% 72.01%\n85.96% 80.44%\n38.69% 40.11%\n43.57% 50.89%\n41.95% 43.65%\n17.76% 16.16%\n\nAccuracy\n18.44%\n38.89%\n16.51%\n\nD2A w/o EM\n\nRecall\nPrecision\n44.93% 44.13%\n93.09% 88.59%\n37.95% 36.54%\n81.82% 76.69%\n40.85% 42.76%\n43.87% 52.16%\n42.47% 44.74%\n1.44%\n1.79%\n\nAccuracy\n47.92%\n34.04%\n15.38%\n\nAccuracy\n50.84%\n39.14%\n16.79%\n\n6\n\n\fTable 3 shows that the recall and precision of co-referenced questions drop from \u223c70% to \u223c37%\nwhen ablating entity memory (D2A w/o EM), which reveals the importance of entity memory\nin a co-referenced scenario. We can see that the accuracy of veri\ufb01cation questions drops from\n45.05% to 18.44%, which means this type of question also needs information on entities from history\ninteraction. After removing the predicate memory, the model (D2A w/o PM) performs poorly in\nellipsis questions, dropping from \u223c80% to \u223c15%. This is consistent with our intuition that the\npredicate of an ellipsis question comes from the previous question. Results show that removing action\nsubsequence memory (D2A w/o AM) hurts the performance on complex questions including logical\nreasoning and quantitative reasoning. After analyzing examples of these two types, we observe that\nellipsis and co-reference phenomena occur in complex questions, the understanding of which needs\nto copy complex logical form from previous questions.\nTo better understand the ability of our semantic parser, we show examples to illustrate the parsing\nresults by our approach (D2A) in Figure 3. As shown, our parser is capable of parsing various types\nof questions. The 2nd and 3rd examples show that the dialog memory helps the parser replicate\nentity and predicate from history interaction. Furthermore, replication actions work well in complex\nquestions such as 8th and 9th examples, where previous un-instantiated action subsequences are\nreplicated and instantiation follows.\n\nFigure 3: Examples of the parsing results of D2A. Q1, R1 and Q2 stand for previous utterance,\nprevious answer and current question, respectively; copy() stands for one of the action from A19-A21\nthat replicates previous action subsequence; reverse() is a speci\ufb01c function that could be applied on\nany predicate, resulting in doubled predicates.\n\n6.3 Discussion\n\nTo understand the limitations of our approach and shed light on future directions to make further\nimprovements, we randomly select 100 wrongly predicted instances for each category, and summary\nfour main classes of errors as follows.\nEntity Linking. A common problem is entity linking error when different entities have exactly the\nsame surface name. Based on a balance between accuracy and latency, we represent an entity based\non the words it contains in this work, so that there\u2019s no difference in their representation. A potential\nway to alleviate this problem is to learn better word representations by considering the contexts from\na knowledge graph [23, 24].\nSpurious Program. We collect referenced action sequence in an automatic way based on an\nassumption that a logical form is correct if it could be executed to the correct answer. However, some\n\n7\n\nidquestiontypecurrent question + previous turnpredicted logical form1Simple Question (Direct)Q1: N/A R1: N/AQ2: Who was the dad of Jorgen OttesenBrahe?\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc3d\ud835\udc5c\ud835\udc5f\ud835\udc54\ud835\udc52\ud835\udc5b\ud835\udc42\ud835\udc61\ud835\udc61\ud835\udc52\ud835\udc60\ud835\udc52\ud835\udc5b\ud835\udc35\ud835\udc5f\ud835\udc4e\u210e\ud835\udc52},\ud835\udc53\ud835\udc4e\ud835\udc61\u210e\ud835\udc52\ud835\udc5f)2Simple Question (Coreferenced)Q1: Who was the dad of Jorgen OttesenBrahe?R1: OtteBraheQ2: Who is the spouse of that one?\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc42\ud835\udc61\ud835\udc61\ud835\udc52\ud835\udc35\ud835\udc5f\ud835\udc52\u210e\ud835\udc52},\ud835\udc60\ud835\udc5d\ud835\udc5c\ud835\udc62\ud835\udc60\ud835\udc52)3Simple Question (Ellipsis)Q1: What is the profession of MkihailBeliaiev?R1: Military personnelQ2: And also tell me about Brett MacLean\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc35\ud835\udc5f\ud835\udc52\ud835\udc61\ud835\udc61\ud835\udc40\ud835\udc4e\ud835\udc50\ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5b},\ud835\udc5c\ud835\udc50\ud835\udc50\ud835\udc62\ud835\udc5d\ud835\udc4e\ud835\udc61\ud835\udc56\ud835\udc5c\ud835\udc5b)4Logical Reasoning (All)Q1: N/A R1: N/AQ2: Which administrative territories have diplomatic relations with Italy and are not Derikhapresent in?\ud835\udc4e\ud835\udc5b\ud835\udc51(\ud835\udc51\ud835\udc56\ud835\udc53\ud835\udc53(\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc3c\ud835\udc61\ud835\udc4e\ud835\udc59\ud835\udc66},\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52(\ud835\udc51\ud835\udc56\ud835\udc5d\ud835\udc59\ud835\udc5c\ud835\udc5a\ud835\udc4e\ud835\udc61\ud835\udc56\ud835\udc50\ud835\udc5f\ud835\udc52\ud835\udc59\ud835\udc4e\ud835\udc61\ud835\udc56\ud835\udc5c\ud835\udc5b)),\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc37\ud835\udc52\ud835\udc5f\ud835\udc56\ud835\udc58\u210e\ud835\udc4e,\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61\ud835\udc5f\ud835\udc66),\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51\ud835\udc4e\ud835\udc51\ud835\udc5a\ud835\udc56\ud835\udc5b\ud835\udc56\ud835\udc60\ud835\udc61\ud835\udc5f\ud835\udc4e\ud835\udc61\ud835\udc56\ud835\udc63\ud835\udc52\ud835\udc61\ud835\udc52\ud835\udc5f\ud835\udc5f\ud835\udc56\ud835\udc61\ud835\udc5c\ud835\udc5f\ud835\udc56\ud835\udc52\ud835\udc60,\ud835\udc56\ud835\udc60\ud835\udc34)5Quantitative ReasoningQ1: N/A R1: N/AQ2: Which works did min number of people do the dubbing for?\ud835\udc4e\ud835\udc5f\ud835\udc54\ud835\udc5a\ud835\udc56\ud835\udc5b(\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc63\ud835\udc5c\ud835\udc56\ud835\udc50\ud835\udc52\ud835\udc4e\ud835\udc50\ud835\udc61\ud835\udc5c\ud835\udc5f},\ud835\udc56\ud835\udc60\ud835\udc4e),\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52(\ud835\udc64\ud835\udc5c\ud835\udc5f\ud835\udc58))6Comparative Reasoning Q1: N/A R1: N/AQ2: Which musical instruments are played by more number of people than electronic keyboard?\ud835\udc59\ud835\udc4e\ud835\udc5f\ud835\udc54\ud835\udc52\ud835\udc5f(\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc5a\ud835\udc62\ud835\udc60\ud835\udc56\ud835\udc50\ud835\udc4e\ud835\udc59\ud835\udc56\ud835\udc5b\ud835\udc60\ud835\udc61\ud835\udc5f\ud835\udc62\ud835\udc5a\ud835\udc52\ud835\udc5b\ud835\udc61\ud835\udc60},\ud835\udc56\ud835\udc60\ud835\udc34),\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52\ud835\udc56\ud835\udc5b\ud835\udc60\ud835\udc61\ud835\udc5f\ud835\udc62\ud835\udc5a\ud835\udc52\ud835\udc5b\ud835\udc61,\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61(\ud835\udc4e\ud835\udc5b\ud835\udc51(\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc52\ud835\udc59\ud835\udc52\ud835\udc50\ud835\udc61\ud835\udc5f\ud835\udc5c\ud835\udc5b\ud835\udc56\ud835\udc50\ud835\udc58\ud835\udc52\ud835\udc66\ud835\udc4f\ud835\udc5c\ud835\udc4e\ud835\udc5f\ud835\udc51},\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52(\ud835\udc56\ud835\udc5b\ud835\udc60\ud835\udc61\ud835\udc5f\ud835\udc62\ud835\udc5a\ud835\udc52\ud835\udc5b\ud835\udc61)),\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc5d\ud835\udc52\ud835\udc5c\ud835\udc5d\ud835\udc59\ud835\udc52},\ud835\udc56\ud835\udc60\ud835\udc34))))7Verification(Boolean)Q1: N/A R1: N/AQ2: Is Arizona Coyotes present in United States of America?\ud835\udc56\ud835\udc5b(\ud835\udc34\ud835\udc5f\ud835\udc56\ud835\udc67\ud835\udc5c\ud835\udc5b\ud835\udc4e\ud835\udc36\ud835\udc5c\ud835\udc66\ud835\udc5c\ud835\udc61\ud835\udc52\ud835\udc60,\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc48\ud835\udc5b\ud835\udc56\ud835\udc61\ud835\udc52\ud835\udc51\ud835\udc46\ud835\udc61\ud835\udc4e\ud835\udc61\ud835\udc52\ud835\udc60\ud835\udc5c\ud835\udc53\ud835\udc34\ud835\udc5a\ud835\udc52\ud835\udc5f\ud835\udc56\ud835\udc50\ud835\udc4e},\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52(\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61\ud835\udc5f\ud835\udc66)))8Quantitative Reasoning (Count)Q1: How many people have birthplace at Provence? R1: 15Q2: And how about Peterborough?\ud835\udc50\ud835\udc5c\ud835\udc5d\ud835\udc66(\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61(\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc43\ud835\udc52\ud835\udc61\ud835\udc52\ud835\udc5f\ud835\udc4f\ud835\udc5c\ud835\udc5f\ud835\udc5c\ud835\udc62\ud835\udc54\u210e},\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52\ud835\udc5d\ud835\udc59\ud835\udc4e\ud835\udc50\ud835\udc52\ud835\udc5c\ud835\udc53\ud835\udc4f\ud835\udc56\ud835\udc5f\ud835\udc61\u210e)))9Comparative Reasoning (Count)Q1: How many musical instruments are played by greater number of people than Body percussion ? R1: 30Q2: And also tell me about timpani?\ud835\udc50\ud835\udc5c\ud835\udc5d\ud835\udc66(\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61(\ud835\udc59\ud835\udc4e\ud835\udc5f\ud835\udc54\ud835\udc52\ud835\udc5f(\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc5a\ud835\udc62\ud835\udc60\ud835\udc56\ud835\udc50\ud835\udc4e\ud835\udc59\ud835\udc56\ud835\udc5b\ud835\udc60\ud835\udc61\ud835\udc5f\ud835\udc62\ud835\udc5a\ud835\udc52\ud835\udc5b\ud835\udc61},\ud835\udc56\ud835\udc60\ud835\udc34),\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52(\ud835\udc56\ud835\udc5b\ud835\udc60\ud835\udc61\ud835\udc5f\ud835\udc62\ud835\udc5a\ud835\udc52\ud835\udc5b\ud835\udc61),\ud835\udc50\ud835\udc5c\ud835\udc62\ud835\udc5b\ud835\udc61(\ud835\udc53\ud835\udc56\ud835\udc5b\ud835\udc51({\ud835\udc61\ud835\udc56\ud835\udc5a\ud835\udc5d\ud835\udc4e\ud835\udc5b\ud835\udc56},\ud835\udc5f\ud835\udc52\ud835\udc63\ud835\udc52\ud835\udc5f\ud835\udc60\ud835\udc52(\ud835\udc56\ud835\udc5b\ud835\udc60\ud835\udc61\ud835\udc5f\ud835\udc62\ud835\udc5a\ud835\udc52\ud835\udc5b\ud835\udc61))))))\fof these logical forms are spurious [5], in the sense that they do not represent the meaning of question\nbut get the correct answer. Filtering rules might be useful to \ufb01lter out spurious logical forms.\nError Propagation. The problem of error propagation occurs because our model learns to replicate\npreviously generated action sequences, which might be incorrect despite we consider the probability\nof the previous logical form. The problem might be alleviated if we incorporate more signals to\nmeasure the correctness of a logical form.\nUnsupported Actions. There exist examples whose logical forms could be not covered by our\ngrammar. An example is \u201cHow many political and administrative territories have diplomatic\nrelationships with France?\u201d, whose answer is \u201c3 and 15\u201d. Incorporating more actions might improve\nthe coverage, however, the aim of this paper is not to explore dataset-speci\ufb01c grammar, but to show\nthat a \ufb02exible grammar works well and dialog memory helps.\n\n7 Related Work\n\nOur task closely relates to two lines of works on content-dependent semantic parsing, categorized by\nthe type of supervision used for model learning.\nThe \ufb01rst line of work learns a context-dependent semantic parser from fully annotated logical forms.\n[1] \ufb01rst learn a context-independent CCG parser, and then conduct context-dependent substitution and\nelaboration. [3] produce logical forms using a set of classi\ufb01cation models. [7] propose a sequence-to-\nsequence model with a copying mechanism to replicate previously generated logical form. The main\ndifference between our task and this line of work is that we learn from denotations with no access to\nannotated logical forms.\nThe second line of work learns a model from denotations, which could be the answer [6] or the \ufb01nal\nworld state [4]. [2] jointly learn a weighted CCG parser and execute spatial/instructional language in\nnavigation environments. [4] develop a shift-reduce parser and use model projection to reduce the\nsearch space. [5] generate tokens of action, constant, and function with a sequence-to-sequence model,\nand use meritocratic gradient weights and randomized beam search to alleviate the spurious program\nproblem. [25] mapping context-sequential instructions to actions sequence, and propose a learning\nalgorithm that take advantage of single-step reward observations and immediate expected reward\nmaximization. [6] regard SQL generation as action sequence prediction, and search legitimate action\nsequences through online learning. A special \u201csubsequent\u201d action is de\ufb01ned to replicate the entire\nSQL query of the previously contiguous utterance. Generated SQL query will be executed on a web\ntable to produce the answer. Similar to [6], our de\ufb01nitions of action and structure constraint depend\non the language of the target logical form. Compared to their method that only learns to replicate\nthe entire logical form of previous utterance, our model is more \ufb02exible and capable of replicating\nvarious information from dialog memory including entities, predicates, and action subsequences\n(i.e. partial logical forms). Our task differs from this line of work in that our logical forms interact\nwith a large-scale knowledge base, which poses new challenges for model training. There also exist\nmemory or encoder-decoder based methods [17, 26, 27] that directly generate an answer utterance as\nthe output of the decoder. Our semantic parsing-based model is essentially different from them in\nthat deep question understanding is required to produce the explicit logical form of the underlying\nmeaning. Our task differs from the \u201cQA+recommendation dialog\u201d task [28, 29] in that they only ask\nquestion in the second turn, the intention of which is about the recommended entity of the \ufb01rst turn.\n\n8 Conclusion\n\nWe present the Dialog-to-Action, a generative model that converts an utterance in conversation to a\nlogical form, which will be executed on a large-scale knowledge base to produce the answer. The\nmodel works in a top-down manner following a small and \ufb02exible grammar, in which the generation of\na logical form is equivalent to the prediction of a sequence of actions. A dialog memory management\nis developed and naturally integrated in the model, so that historical entities, predicates, and action\nsubsequences could be selectively replicated. The model is effectively learned from denotations\nwithout using annotated logical forms. Results on a large-scale dataset demonstrate the effectiveness\nof considering the dialog memory, and show that our model performs signi\ufb01cantly better than a strong\nmemory network-based encoder-decoder model.\n\n8\n\n\fAcknowledgments\n\nThis work is supported by the National Natural Science Foundation of China (61472453, U1401256,\nU1501252, U1611264, U1711261, U1711262). Thanks to the anonymous reviewers and Junwei Bao\nfor their helpful comments and suggestions.\n\nReferences\n[1] Luke S Zettlemoyer and Michael Collins. Learning context-dependent mappings from sentences to\nlogical form. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th\nInternational Joint Conference on Natural Language Processing of the AFNLP, pages 976\u2013984, 2009.\n\n[2] Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions\n\nto actions. Transactions of the Association of Computational Linguistics, 1:49\u201362, 2013.\n\n[3] Andreas Vlachos and Stephen Clark. A new corpus and imitation learning framework for context-dependent\n\nsemantic parsing. Transactions of the Association for Computational Linguistics, 2:547\u2013559, 2014.\n\n[4] Reginald Long, Panupong Pasupat, and Percy Liang. Simpler context-dependent logical forms via model\nprojections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics\n(Volume 1: Long Papers), pages 1456\u20131465. Association for Computational Linguistics, August 2016.\n\n[5] Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. From language to programs: Bridging\nreinforcement learning and maximum marginal likelihood. In Proceedings of the 55th Annual Meeting of\nthe Association for Computational Linguistics, pages 1051\u20131062, 2017.\n\n[6] Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for sequential\nquestion answering. In Proceedings of the 55th Annual Meeting of the Association for Computational\nLinguistics (Volume 1: Long Papers), volume 1, pages 1821\u20131831, 2017.\n\n[7] Alane Suhr, Srinivasan Iyer, and Yoav Artzi. Learning to map context-dependent sentences to executable\n\nformal queries. arXiv preprint arXiv:1804.06868, 2018.\n\n[8] Heung-Yeung Shum, Xiaodong He, and Di Li. From eliza to xiaoice: Challenges and opportunities with\n\nsocial chatbots. arXiv preprint arXiv:1801.01957, 2018.\n\n[9] Wei Lu, Hwee Tou Ng, Wee Sun Lee, and Luke S Zettlemoyer. A generative model for parsing natural\nlanguage to meaning representations. In Proceedings of the Conference on Empirical Methods in Natural\nLanguage Processing, pages 783\u2013792. Association for Computational Linguistics, 2008.\n\n[10] Dipendra Kumar Misra and Yoav Artzi. Neural shift-reduce ccg semantic parsing. In Proceedings of the\n\n2016 Conference on Empirical Methods in Natural Language Processing, pages 1775\u20131786, 2016.\n\n[11] Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. Neural semantic parsing with type constraints\nfor semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural\nLanguage Processing, pages 1516\u20131526. Association for Computational Linguistics, September 2017.\n\n[12] Bo Chen, Le Sun, and Xianpei Han. Sequence-to-action: End-to-end semantic graph generation for\nIn Proceedings of the 56th Annual Meeting of the Association for Computational\n\nsemantic parsing.\nLinguistics, 2018.\n\n[13] Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In\nProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long\nPapers), pages 440\u2013450, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL\nhttp://aclweb.org/anthology/P17-1041.\n\n[14] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in\n\nprogrammatic context. arXiv preprint arXiv:1808.09588, 2018.\n\n[15] Pengcheng Yin and Graham Neubig. Tranx: A transition-based neural abstract syntax parser for semantic\nparsing and code generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural\nLanguage Processing: System Demonstrations, pages 7\u201312. Association for Computational Linguistics,\n2018. URL http://aclweb.org/anthology/D18-2002.\n\n[16] Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L Gaunt.\n\nLearning to represent edits. arXiv preprint arXiv:1810.13337, 2018.\n\n9\n\n\f[17] Amrita Saha, Vardaan Pahuja, Mitesh M Khapra, Karthik Sankaranarayanan, and Sarath Chandar. Complex\nsequential question answering: Towards learning to converse over linked question answer pairs with a\nknowledge graph. arXiv preprint arXiv:1801.10314, 2018.\n\n[18] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In\n\nAdvances in neural information processing systems, pages 3104\u20133112, 2014.\n\n[19] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning\n\nto align and translate. Proceeding of ICLR, 2015.\n\n[20] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger\nSchwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder\u2013decoder for statistical\nmachine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language\nProcessing (EMNLP), pages 1724\u20131734, 2014.\n\n[21] Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural\n\nmachine translation. empirical methods in natural language processing, pages 1412\u20131421, 2015.\n\n[22] Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston.\n\nKey-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126, 2016.\n\n[23] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translat-\ning embeddings for modeling multi-relational data. In Advances in neural information processing systems,\npages 2787\u20132795, 2013.\n\n[24] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor\nnetworks for knowledge base completion. In Advances in neural information processing systems, pages\n926\u2013934, 2013.\n\n[25] Alane Suhr and Yoav Artzi. Situated mapping of sequential instructions to actions with single-step reward\nobservation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics,\npages 2072\u20132082. Association for Computational Linguistics, 2018. URL http://aclweb.org/\nanthology/P18-1193.\n\n[26] Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Mem2seq: Effectively incorporating knowledge\n\nbases into end-to-end task-oriented dialog systems. arXiv preprint arXiv:1804.08217, 2018.\n\n[27] Liangchen Luo, Wenhao Huang, Qi Zeng, Zaiqing Nie, and Xu Sun. Learning personalized end-to-end\ngoal-oriented dialog. In Proceedings of the 33rd AAAI Conference on Arti\ufb01cial Intelligence, Honolulu,\nHawaii, January 2019.\n\n[28] Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur\nSzlam, and Jason Weston. Evaluating prerequisite qualities for learning end-to-end dialog systems. arXiv\npreprint arXiv:1511.06931, 2015.\n\n[29] Jason E Weston. Dialog-based language learning. In Advances in Neural Information Processing Systems,\n\npages 829\u2013837, 2016.\n\n10\n\n\f", "award": [], "sourceid": 1534, "authors": [{"given_name": "Daya", "family_name": "Guo", "institution": "Sun Yat-Sen University"}, {"given_name": "Duyu", "family_name": "Tang", "institution": "Microsoft Research"}, {"given_name": "Nan", "family_name": "Duan", "institution": "Microsoft Research"}, {"given_name": "Ming", "family_name": "Zhou", "institution": "Microsoft Research"}, {"given_name": "Jian", "family_name": "Yin", "institution": "Sun Yat-Sen University"}]}