{"title": "Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering", "book": "Advances in Neural Information Processing Systems", "page_first": 2654, "page_last": 2665, "abstract": "Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction a novel `fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, deep network techniques have been employed to successively reduce the large set of facts until one of the two entities of the final remaining fact is predicted as the answer. We observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. Instead, we develop an entity graph and use a graph convolutional network to `reason' about the correct answer by jointly considering all entities. We show on the challenging FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state-of-the-art.", "full_text": "Out of the Box: Reasoning with Graph Convolution\n\nNets for Factual Visual Question Answering\n\nMedhini Narasimhan, Svetlana Lazebnik, Alexander G. Schwing\n\nUniversity of Illinois Urbana-Champaign\n\n{medhini2, slazebni, aschwing}@illinois.edu\n\nAbstract\n\nAccurately answering a question about a given image requires combining obser-\nvations with general knowledge. While this is effortless for humans, reasoning\nwith general knowledge remains an algorithmic challenge. To advance research\nin this direction a novel \u2018fact-based\u2019 visual question answering (FVQA) task has\nbeen introduced recently along with a large set of curated facts which link two\nentities, i.e., two possible answers, via a relation. Given a question-image pair,\ndeep network techniques have been employed to successively reduce the large set\nof facts until one of the two entities of the \ufb01nal remaining fact is predicted as the\nanswer. We observe that a successive process which considers one fact at a time\nto form a local decision is sub-optimal. Instead, we develop an entity graph and\nuse a graph convolutional network to \u2018reason\u2019 about the correct answer by jointly\nconsidering all entities. We show on the challenging FVQA dataset that this leads\nto an improvement in accuracy of around 7% compared to the state of the art.\n\n1\n\nIntroduction\n\nWhen answering questions about images, we easily combine the visualized situation with general\nknowledge that is available to us. However, for algorithms, an effortless combination of general\nknowledge with observations remains challenging, despite signi\ufb01cant work which aims to leverage\nthese mechanisms for autonomous agents and virtual assistants.\nIn recent years, a signi\ufb01cant amount of research has investigated algorithms for visual question\nanswering (VQA) [2, 19, 32, 43, 44, 63], visual question generation (VQG) [21, 30, 36, 49], and visual\ndialog [12,13,20], paving the way to autonomy for arti\ufb01cial agents operating in the real world. Images\nand questions in these datasets cover a wide range of perceptual abilities such as counting, object\nrecognition, object localization, and even logical reasoning. However, for many of these datasets\nthe questions can be answered solely based on the visualized content, i.e., no general knowledge is\nrequired. Therefore, numerous approaches address VQA, VQG and dialog tasks by extracting visual\ncues using deep network architectures [1,2,11,16,18,19,24,32,33,35,41,44,53,56,57,59,60,63,65,\n66, 68], while general knowledge remains unavailable.\nTo bridge this discrepancy between human behavior and present day algorithmic design, Wang et\nal. [50] introduced a novel \u2018fact-based\u2019 VQA (FVQA) task, and an accompanying dataset containing\nimages, questions with corresponding answers and a knowledge base (KB) of facts extracted from\nthree different sources: WebChild [47], DBPedia [3] and ConceptNet [45]. Unlike classical VQA\ndatasets, a question in the FVQA dataset is answered by a collective analysis of the information in\nthe image and the KB of facts. Each question is mapped to a single supporting fact which contains\nthe answer to the question. Thus, answering a question requires analyzing the image and choosing\nthe right supporting fact, for which Wang et al. [50] propose a keyword-matching technique. This\napproach suffers when the question doesn\u2019t focus on the most obvious visual concept and when there\nare synonyms and homographs. Moreover, special information about the visual concept type and\nthe answer source make it hard to generalize their approach to other datasets. We addressed these\nissues in our previous work [38], where we proposed a learning-based approach which embeds the\nimage question pairs and the facts to the same space and ranks the facts according to their relevance.\nWe observed a signi\ufb01cant improvement in performance which motivated us to explore other learning\nbased methods, particularly those which exploit the graphical structure of the facts.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: Results of our graph convolutional net based approach on the recently introduced FVQA dataset.\n\nIn this work, our main motivation is to develop a technique which uses the information from multiple\nfacts before arriving at an answer and relies less on retrieving the single \u2018correct\u2019 fact needed to\nanswer a question. To this end, we develop a model which \u2018thinks out of the box,\u2019 i.e., it \u2018reasons\u2019\nabout the right answer by taking into account a list of facts via a Graph Convolution Network\n(GCN) [25]. The GCN enables joint selection of the answer from a list of candidate answers, which\nsets our approach apart from the previous methods that assess one fact at a time. Moreover, we select\na list of supporting facts in the KB by ranking GloVe embeddings. This handles challenges due to\nsynonyms and homographs and also works well with questions that don\u2019t focus on the main object.\nWe demonstrate the proposed algorithm on the FVQA dataset [50], outperforming the state of the art\nby around 7%. Fig. 1 shows results obtained by our model. Unlike the models proposed in [50], our\nmethod does not require any information about the ground truth fact (visual concept type and answer\nsource). In contrast to our approach in [38], which focuses on learning a joint image-question-fact\nembedding for retrieving the right fact, our current work uses a simpler method for retrieving multiple\ncandidate facts (while still ensuring that the recall of the ground truth fact is high), followed by a\nnovel GCN inference step that collectively assesses all the relevant facts before arriving at an answer.\nUsing an ablation analysis we \ufb01nd improvements due to the GCN component, which exploits the\ngraphical structure of the knowledge base and allows for sharing of information between possible\nanswers, thus improving the explainability of our model.\n\n2 Related Work\nWe develop a visual question answering algorithm based on graph convolutional nets which bene\ufb01ts\nfrom general knowledge encoded in the form of a knowledge base. We therefore brie\ufb02y review\nexisting work in the areas of visual question answering, fact-based visual question answering and\ngraph convolutional networks.\nVisual Question Answering: Recently, there has been signi\ufb01cant progress in creating large VQA\ndatasets [2, 17, 23, 34, 41, 66] and deep network models which correctly answer a question about\nan image. The initial VQA models [1, 2, 4, 11, 16, 17, 19, 24, 32, 33, 35, 41, 44, 53, 57, 59, 60, 65, 68]\ncombined the LSTM encoding of the question and the CNN encoding of the image using a deep\nnetwork which \ufb01nally predicted the answer. Results can be improved with attention-based multi-\nmodal networks [1, 11, 16, 32, 43, 44, 57, 59] and dynamic memory networks [22, 56]. All of these\nmethods were tested on standard VQA datasets where the questions can solely be answered by\nobserving the image. No out of the box thinking was required. For example, given an image of a cat,\nand the question, \u201cCan the animal in the image be domesticated?,\u201d we want our method to combine\nfeatures from the image with common sense knowledge (a cat can be domesticated). This calls for\nthe development of a model which leverages external knowledge.\nFact-based Visual Question Answering: Recent research in using external knowledge for natural\nlanguage comprehension led to the development of semantic parsing [5, 6, 10, 15, 28, 31, 38, 40, 46, 55,\n61, 62, 64] and information retrieval [7\u20139, 14, 26, 48, 58] methods. However, knowledge based visual\nquestion answering is fairly new. Notable examples in this direction are works by Zhu et al. [67], Wu\net al. [54], Wang et al. [51], Narasimhan et al. [37], Krishnamurthy and Kollar [27], and our previous\nwork, Narasimhan and Schwing [38].\nAsk Me Anything (AMA) by Wu et al. [54], AHAB by Wang et al. [51], and FVQA by Wang et\nal. [50] are closely related to our work. In AMA, attribute information extracted from the image is\nused to query the external knowledge base DBpedia [3], to retrieve paragraphs which are summarized\n\n2\n\nQuestion:What is the area in the image used for?Relation:UsedForVisual Concept: FieldFact:(Field, UsedFor, Grazing Animals) Answer:Grazing AnimalsQuestion:Which object in the image is more similar to a tiger?Relation:RelatedToVisual Concept: CatFact:(Cat, RelatedTo, Tiger) Answer:CatQuestion:What can be found on the ground in this image?Relation:AtLocationVisual Concept: BeachFact:(Sand, AtLocation, Beach) Answer:Sand\fto form a knowledge vector. The knowledge vector is combined with the attribute vector and multiple\ncaptions generated for the image, before being passed as input to an LSTM which predicts the answer.\nThe main drawback of AMA is that it does not perform any explicit reasoning and ignores the possible\nstructure in the KB. To address this, AHAB and FVQA attempt to perform explicit reasoning. In\nAHAB, the question is converted to a database query via a multistep process, and the response to\nthe query is processed to obtain the \ufb01nal answer. FVQA also learns a mapping from questions to\ndatabase queries through classifying questions into categories and extracting parts from the question\ndeemed to be important. A matching score is computed between the facts retrieved from the database\nand the question, to determine the most relevant fact which forms the basis of the answer for the\nquestion. Both these methods use databases with a particular structure: facts are represented as tuples,\nfor example, (Apple, IsA, Fruit), and (Cheetah, FasterThan, Lion).\nThe present work follows up on our earlier method, Straight to the Facts (STTF) [38]. STTF uses\nobject, scene, and action predictors to represent an image and an LSTM to represent a question and\ncombines the two using a deep network. The facts are scored based on the cosine similarity of the\nimage-question embedding and fact embedding. The answer is extracted from the highest scoring\nfact.\nWe evaluate our method on the dataset released as part of the FVQA work, referred to as the FVQA\ndataset [50], which is a subset of three structured databases \u2013 DBpedia [3], ConceptNet [45], and\nWebChild [47].\nGraph Convolutional Nets: Kipf and Welling [25] introduced Graph Convolutional Networks\n(GCN) to extend Conv nets (CNNs) [29] to arbitrarily connected undirected graphs. GCNs learn\nrepresentations for every node in the graph that encodes both the local structure of the graph\nsurrounding the node of interest, as well as the features of the node itself. At a graph convolutional\nlayer, features are aggregated from neighboring nodes and the node itself to produce new output\nfeatures. By stacking multiple layers, we are able to gather information from nodes further away.\nGCNs have been applied successfully for graph node classi\ufb01cation [25], graph link prediction [42],\nand zero-shot prediction [52]. Knowledge graphs naturally lend themselves to applications of GCNs\nowing to the underlying structured interactions between nodes connected by relationships of various\ntypes. In this work, given an image and a question about the image, we \ufb01rst identify useful sub-graphs\nof a large knowledge graph such as DBpedia [3] and then use GCNs to produce representations\nencoding node and neighborhood features that can be used for answering the question.\nSpeci\ufb01cally, we propose a model that retrieves the most relevant facts to a question-answer pair based\non GloVe features. The sub-graph of facts is passed through a graph convolution network which\npredicts an answer from these facts. Our approach has the following advantages: 1) Unlike FVQA\nand AHAB, we avoid the step of query construction and do not use the ground truth visual concept or\nanswer type information which makes it possible to incorporate any fact space into our model. 2) We\nuse GloVe embeddings for retrieving and representing facts which works well with synonyms and\nhomographs. 3) In contrast to STTF, which uses a deep network to arrive at the right fact, we use a\nGCN which operates on a subgraph of relevant facts while retaining the graphical structure of the\nknowledge base which allows for reasoning using message passing. 4) Unlike previous works, we\nhave reduced the reliance on the knowledge of the ground truth fact at training time.\n\n3 Visual Question Answering with Knowledge Bases\n\nTo jointly \u2018reason\u2019 about a set of answers for a given question-image pair, we develop a graph\nconvolution net (GCN) based approach for visual question answering with knowledge bases. In the\nfollowing we \ufb01rst provide an overview of the proposed approach before delving into details of the\nindividual components.\nOverview: Our proposed approach is outlined in Fig. 2. Given an image I and a corresponding\nquestion Q, the task is to predict an answer A while using an external knowledge base KB which\nconsists of facts, fi, i.e., KB = {f1, f2, . . . , f|KB|}. A fact is represented as a Resource Distribution\nFramework (RDF) triplet of the form f = (x, r, y), where x is a visual concept grounded in the image,\ny is an attribute or phrase, and r \u2208 R is a relation between the two entities, x and y. The relations in\nthe knowledge base are part of a set of 13 possible relations R = {Category, Comparative, HasA,\nIsA, HasProperty, CapableOf, Desires, RelatedTo, AtLocation, PartOf, ReceivesAction, UsedFor,\nCreatedBy}. Subsequently we use x(f ), y(f ), or rel(f ) to extract the visual concept x, the attribute\nphrase y, or the relation r in fact f = (x, r, y) respectively.\n\n3\n\n\fFigure 2: Outline of the proposed approach: Given an image and a question, we use a similarity scoring\ntechnique (1) to obtain relevant facts from the fact space. An LSTM (2) predicts the relation from the question to\nfurther reduce the set of relevant facts and its entities. An entity embedding is obtained by concatenating the\nvisual concepts embedding of the image (3), the LSTM embedding of the question (4), and the LSTM embedding\nof the entity (5). Each entity forms a single node in the graph and the relations constitute the edges (6). A GCN\nfollowed by an MLP performs joint assessment (7) to predict the answer. Our approach is trained end-to-end.\nEvery question Q is associated with a single fact, f\u2217, that helps answer the question. More speci\ufb01cally,\nthe answer A is one of the two entities of that fact, i.e., either A = x\u2217 or A = y\u2217, both of which can\nbe extracted from f\u2217 = (x\u2217, r\u2217, y\u2217).\nWang et al. [50] formulate the task as prediction of a fact \u02c6f = (\u02c6x, \u02c6r, \u02c6y) for a given question-image pair,\nand subsequently extract either \u02c6x or \u02c6y, depending on the result of an answer source classi\ufb01er. As there\nare over 190, 000 facts, retrieving the correct supporting fact f\u2217 is challenging and computationally\ninef\ufb01cient. Usage of question properties like \u2018visual concept type\u2019 makes the proposed approach hard\nto extend.\nGuided by the observation that the correct supporting fact f\u2217 is within the top-100 of a retrieval\nmodel 84.8% of the time, we develop a two step solution: (1) retrieving the most relevant facts\nfor a given question-image pair. To do this, we extract the top-100 facts, i.e., f100 based on word\nsimilarity between the question and the fact. Further, we obtain the set of relevant facts frel by\nreducing f100 based on consistency of the fact relation r with a predicted relation \u02c6r. (2) predicting\nthe answer as one of the entities in this reduced fact space frel. To predict the answer we use a GCN\nto compute representations of nodes in a graph, where the nodes correspond to the unique entities\ne \u2208 E = {x(f ) : f \u2208 frel}\u222a{y(f ) : f \u2208 frel}, i.e., either x or y in the fact space frel. Two entities in\nthe graph are connected if a fact relates the two. Using a GCN permits to jointly assess the suitability\nof all entities which makes our proposed approach different from classi\ufb01cation based techniques.\nFor example, consider the image and the question shown in Fig. 2. The relation for this question is\n\u201cIsA\u201d and the fact associated with this question-image pair is (Orange, IsA, Citric). The answer is\nOrange. In the following we \ufb01rst discuss retrieval of the most relevant facts for a given question-image\npair before detailing our GCN approach for extracting the answer from this reduced fact space.\n\n3.1 Retrieval of Relevant Facts\nTo retrieve a set of relevant facts frel for a given question-image pair, we pursue a score based\napproach. We \ufb01rst compute the cosine similarity of the GloVe embeddings of the words in the fact\nwith the words in the question and the words of the visual concepts detected in the image. Because\nsome words may differ between question and fact, we obtain a fact score by averaging the Top-K\nword similarity scores. We rank the facts based on their similarity and retrieve the top-100 facts for\neach question, which we denote f100. We chose 100 facts as this gives the best downstream accuracy\nas shown in Tab. 1. As indicated in Tab. 1, we observe a high recall of the ground truth fact in the\nretrieved facts while using this technique. This motivates us to avoid a complex model which \ufb01nds the\nright fact, as used in [50] and [38], and instead use the retrieved facts to directly predict the answer.\n\n4\n\nObject, Scene, Action PredictionsLSTMQuestion Emb.LSTMQuestion:Which object is a citric fruit?Top 100 FactsRelevantEntitiesRelation: IsALSTMEntity Emb.1FilterFact \u00a0Space2354Graph NodeNodeEmbedding6Sec 3.2: Answer PredictionSec 3.1: Fact Retrieval0010hidden layersOutputsInputGraph7H(0)1H(0)3H(0)2H(0)4fruitorangeCitricfruitappleH(L)1H(L)3H(L)2H(L)4Answer:orangeGraphConvLayerMLP\fFact Recall\n\n76.5\nDownstream Accuracy 22.6 58.93\n\n@1 @50 @100 @150 @200 @500\n93.1\n22.6\n60.22\n\n88.4\n68.23\n\n91.6\n65.61\n\n84.8\n69.35\n\nTable 1: Recall and downstream accuracy for different number of facts.\n\nWe further reduce this set of 100 facts by assessing their relation attribute. To predict the relation\nfrom a given question, we use the approach described in [38]. We retain the facts among the top-100\nonly if their relation agrees with the predicted relation \u02c6r, i.e., frel = {f \u2208 f100 : rel(f ) = \u02c6r}.\nFor every question, unique entities in the facts frel are grouped into a set of candidate entities,\nE = {x(f ) : f \u2208 frel} \u222a {y(f ) : f \u2208 frel}, with |E| \u2264 200 (2 entities/fact and at most 100 facts).\nCurrently, we train the relation predictor\u2019s parameters independently of the remaining model. In\nfuture work we aim for an end-to-end model which includes this step.\n3.2 Answer Prediction\nGiven the set of candidate entities E, we want to \u2018reason\u2019 about the answer, i.e., we want to predict\nan entity \u02c6e \u2208 E. To jointly assess the suitability of all candidate entities in E, we develop a Graph-\nConvolution Net (GCN) based approach which is augmented by a multi-layer perceptron (MLP). The\nnodes in the employed graph correspond to the available entities e \u2208 E and their node representation\nis given as an input to the GCN. The GCN combines entity representations in multiple iterative steps.\nThe \ufb01nal transformed entity representations learned by the GCN are then used as input in an MLP\nwhich predicts a binary label, i.e., {1, 0}, for each entity e \u2208 E, indicating if e is or isn\u2019t the answer.\nMore formally, the goal of the GCN is to learn how to combine representations for the nodes e \u2208 E\nof a graph, G = (E,E). Its output feature representations depend on: (1) learnable weights; (2) an\nadjacency matrix Aadj describing the graph structure E. We consider two entities to be connected if\nthey belong to the same fact; (3) a parametric input representation gw(e) for every node e \u2208 E of\nthe graph. We subsume the original feature representations of all nodes in an |E| \u00d7 D-dimensional\nfeature matrix H (0) \u2208 R|E|\u00d7D, where D is the number of features. In our case, each node e \u2208 E is\nrepresented by the concatenation of the corresponding image, question and entity representation, i.e.,\nw (e)). Combining the three representations ensures that each node/entity\ngw(e) = (gV\ndepends on the image and the question. The node representation is discussed in detail below.\nThe GCN consists of L hidden layers where each layer is a non-linear function f (\u00b7,\u00b7). Speci\ufb01cally,\n(1)\n\nH (l) = f (H (l\u22121), A) = \u03c3( \u02dcD\u22121/2 \u02dcA \u02dcD\u22121/2H (l\u22121)W (l\u22121)) \u2200l \u2208 {1, . . . , L},\n\nw (Q), gC\n\nw (I), gQ\n\nwhere the input to the GCN is H (0), \u02dcA = Aadj + I (I is an identity matrix), \u02dcD is the diagonal node\ndegree matrix of \u02dcA, W (l) is the matrix of trainable weights at the l-th layer of the GCN, and \u03c3(\u00b7) is a\nnon-linear activation function. We let the K-dimensional vector \u02c6g(e) \u2208 RK refer to the output of the\nGCN, extracted from H (L) \u2208 R|E|\u00d7K. Hereby, K is the number of output features.\nThe output of the GCN, \u02c6g(e) is passed through an MLP to obtain the probability pNN\ne \u2208 E is the answer for the given question-image pair. We obtain our predicted answer \u02c6A via\n\nw (\u02c6g(e)) that\n\npNN\nw (\u02c6g(e)).\n\nw (I), gQ\n\nw (Q), gC\n\n\u02c6A = arg max\ne\u2208E\n\n(2)\nAs mentioned before, each node e \u2208 E is represented by the concatenation of the corresponding\nimage, question and entity representation, i.e., gw(e) = (gV\nw (e)). We discuss those\nthree representations subsequently.\nw (I) \u2208 {0, 1}1176 is a multi-hot vector of\n1. Image Representation: The image representation, gV\nsize 1176, indicating the visual concepts which are grounded in the image. Three types of visual\nconcepts are detected in the image: actions, scenes and objects. These are detected using the same\npre-trained networks described in [38].\n2. Question Representation: An LSTM net is used to encode each question into the representation\nw (Q) \u2208 R128. The LSTM is initialized with GloVe embeddings [39] for each word in the question,\ngQ\nwhich is \ufb01ne-tuned during training. The hidden representation of the LSTM constitutes the question\nencoding.\n3. Entity Representation: For each question, the entity encoding gC\nw (e) is computed for every entity\ne in the entity set E. Note that an entity e is generally composed of multiple words. Therefore, similar\n\n5\n\n\fMethod\n\nLSTM-Question+Image+Pre-VQA [50]\nHie-Question+Image+Pre-VQA [50]\nFVQA [50]\nEnsemble [50]\nStraight to the Facts (STTF) [38]\nOurs\nEntity\n\nAccuracy\n\n@1\n24.98\n43.41\n56.91\n58.76\n62.20\n\n@3\n40.40\n59.44\n64.65\n\n-\n\n75.60\n\nMLP\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n-\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n\nGCN Layers\n\n-\n-\n2\n-\n-\n2\n1\n1\n3\n3\n2\n2\n\n2\n\nRel\n-\n@1\n@1\n-\n@1\n@1\n@1\n@3\n@1\n@3\n@1\n@3\n\ngt\n\n10.32\n13.89\n14.12\n29.72\n50.36\n48.43\n54.60\n57.89\n56.90\n60.78\n65.80\n69.35\n\n72.97\n\n77.99\n\n13.15\n16.40\n17.75\n35.38\n56.21\n53.87\n60.91\n65.14\n62.32\n68.65\n77.32\n80.25\n\n83.01\n\n-\n\nQ\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n\n13\n\nHuman\n\nVC\n-\n-\n-\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n(cid:88)\n\nTable 2: Answer accuracy over the FVQA dataset.\n\nto the question encoding, the hidden representation of an LSTM net is used. It is also initialized with\nthe GloVe embeddings [39] of each word in the entity, which is \ufb01ne-tuned during training.\nThe answer prediction model parameters consists of weights from the question embedding, entity\nembedding, GCN, and MLP. These are trained end-to-end.\n3.3 Learning\nWe note that the answer prediction and relation prediction model parameters are trained separately.\nThe dataset, D = {(I, Q, f\u2217, A\u2217)}, to train both these parameters is obtained from [50]. It contains\ntuples (I, Q, f\u2217, A\u2217) each composed of an image I, a question Q, as well as the ground-truth fact f\u2217\nand answer A\u2217.\nTo train the relation predictor\u2019s parameters we use the subset D1 = {(Q, r\u2217)}, containing pairs\nof questions and the corresponding relations r\u2217 = rel(f\u2217) extracted from the ground-truth fact f\u2217.\nStochastic gradient descent and classical cross-entropy loss are used to train the classi\ufb01er.\nThe answer predictor\u2019s parameters, consist of the question and entity embeddings, the two hidden\nlayers of the GCN, and the layers of the MLP. The model operates on question-image pairs and\nextracts the entity label from the ground-truth answer A\u2217 of the dataset D, i.e., 0 if it isn\u2019t the answer\nand 1 if it is. Again we use stochastic gradient descent and binary cross-entropy loss.\n4 Experimental Evaluation\nBefore assessing the proposed approach subsequently, we \ufb01rst review properties of the FVQA dataset.\nWe then present quantitative results to compare our proposed approach with existing baselines before\nillustrating qualitative results.\nFactual visual question answering dataset: To evaluate our model, We use the publicly available\nFVQA [50] knowledge base and dataset. This dataset consists of 2,190 images, 5,286 questions, and\n4,126 unique facts corresponding to the questions. The knowledge base consists of 193,449 facts,\nwhich were constructed by extracting top visual concepts for all images in the dataset and querying\nfor those concepts in the knowledge bases, WebChild [47], ConceptNet [45], and DBPedia [3].\nRetrieval of Relevant Facts: As described in Sec. 3.1, a similarity scoring technique is used to\nretrieve the top-100 facts f100 for every question. GloVe 100d embeddings are used to represent each\nword in the fact and question. An initial stop-word removal is performed to remove stop words (such\nas \u201cwhat,\u201d \u201cwhere,\u201d \u201cthe\u201d) from the question. To assign a similarity score to each fact, we compute\nthe word-wise cosine similarity of the GloVe embedding of every word in the fact with the words in\nthe question and the detected visual concepts. We choose the top K% of the words in the fact with\nthe highest similarity and average these values to assign a similarity score to the fact. Empirically we\n\n6\n\n\fSub-component\nFact-retrieval\nRelation prediction\nAnswer prediction(GCN)\nTotal error\n\nError % @1\n\n15.20\n9.4\n6.05\n30.65\n\nTable 3: Error contribution of the sub-components of the model to the total Top-1 error (30.65%).\n\nfound K = 80 to give the best result. The facts are sorted based on the similarity and the 100 highest\nscoring facts are \ufb01ltered. Tab. 1 shows that the ground truth fact is present in the top-100 retrieved\nfacts 84.8% of the time and is retrieved as the top-1 fact 22.5% of the time. The numbers reported are\nan average over the \ufb01ve test sets. We also varied the number of facts retrieved in the \ufb01rst stage and\nreport the recall and downstream accuracy in Tab. 1. The recall @50 (76.5%) is lower than the recall\n@100 (84.8%), which causes the \ufb01nal accuracy of the model to drop to 58.93%. When we retrieve\n150 facts, recall is 88.4% and \ufb01nal accuracy is 68.23%, which is slightly below the \ufb01nal accuracy\nwhen retrieving 100 facts (69.35%). The \ufb01nal accuracy further drops as we increase the number of\nretrieved facts to 200 and 500.\nPredicting the relation: As described earlier, we use the network proposed in [38] to determine the\nrelation given a question. Using this approach, the Top-1 and Top-3 accuracy for relation prediction\nare 75.4% and 91.97% respectively.\nPredicting the Correct Answer: Sec. 3.2 explains in detail the model used to predict an answer\nfrom the set of candidate entities E. Each node of the graph G is represented by the concatenation of\nthe image, question, and entity embeddings. The image embedding gV\nw (I) is a multi-hot vector of size\n1176, indicating the presence of a visual concept in the image. The LSTM to compute the question\nw (Q) is initialized with GloVe 100d embeddings for each of the words in the question.\nembedding gQ\nBatch normalization and a dropout of 0.5 is applied after both the embedding layer and the LSTM\nlayer. The question embedding is given by the hidden layer of the LSTM and is of size 128. Each\nentity e \u2208 E is also represented by a 128 dimensional vector gC\nw (e) which is computed by an LSTM\noperating on the words of the entity e. The concatenated vector gw(e) = (gV\nw (e)) has\na dimension of 1429 (i.e., 1176+128+128).\nFor each question, the feature matrix H (0) is constructed from the node representations gw(e). The\nadjacency matrix Aadj denotes the edges between the nodes. It is constructed by using the Top-1 or\nTop-3 relations predicted in Sec. 3.1. The adjacency matrix Aadj \u2208 {0, 1}200\u00d7200 is of size 200\u00d7 200\nas the set E has at most 200 unique entities (i.e., 2 entities per fact and 100 facts per question). The\nGCN consists of 2 hidden layers, each operating on 200 nodes, and each node is represented by a\nfeature vector of size 512. The representations of each node from the second hidden layer, i.e., H (2)\nare used as input for a multi-layer perceptron which has 512 input nodes and 128 hidden nodes. The\noutput of the hidden nodes is passed to a binary classi\ufb01er that predicts 0 if the entity is not the answer\nand 1 if it is. The model is trained end-to-end over 100 epochs with batch gradient descent (Adam\noptimizer) using cross-entropy loss for each node. Batch normalization and a dropout of 0.5 was\napplied after each layer. The activation function used throughout is ReLU.\nTo prove the effectiveness of our model, we show six ablation studies in Tab. 2. Q, VC, Entity denote\nquestion, visual concept, and entity embeddings respectively. \u201811\u2019 is the model discussed in Sec. 3\nwhere the entities are \ufb01rst \ufb01ltered by the predicted relation and each node of the graph is represented\nby a concatenation of the question, visual concept, and entity embeddings. \u201812\u2019 uses the top three\nrelations predicted by the question-relation LSTM net and retains all the entities which are connected\nby these three relations. \u201813\u2019 uses the ground truth relation for every question.\nTo validate the approach we construct some additional baselines. In \u20181,\u2019 each node is represented using\nonly the question and the entity embeddings and the entities are not \ufb01ltered by relation. Instead, all\nthe entities in E are fed to the MLP. \u20182\u2019 additionally \ufb01lters based on relation. \u20183\u2019 introduces a 2-layer\nGCN before the MLP. \u20184\u2019 is the same as \u20181\u2019 except each node is now represented using question,\nentity and visual concept embeddings. \u20185\u2019 \ufb01lters by relation and skips the GCN by feeding the entity\nrepresentations directly to the MLP. \u20186\u2019 skips the MLP and the output nodes of the GCN are directly\nclassi\ufb01ed using a binary classi\ufb01er. We observe that there is a signi\ufb01cant improvement in performance\nwhen we include the visual concept features in addition to question and entity embeddings, thus\nhighlighting the importance of the visual concepts. Without visual concepts, the facts retrieved in the\n\ufb01rst step have low recall which in turn reduces the downstream test accuracy.\n\nw (Q), gC\n\nw (I), gQ\n\n7\n\n\fFigure 3: Visual Concepts (VCs) detected by our model. For each image we detect objects, scenes, and\nactions. We observe the supporting facts to have strong alignment with the VCs which proves the effectiveness\nof including VCs in our model.\n\nWe also report the top-1 and top-3 accuracy obtained by varying the number of layers in the GCN.\nWith 3 layers (\u20189\u2019 and \u201810\u2019), our model over\ufb01ts, causing the test accuracy to drop to 60.78%. With\n1 layer (\u20187\u2019 and \u20188\u2019), the accuracy is 57.89% and we hypothesize that this is due to the limited\ninformation exchange that occurs with one GCN layer. We observe a correlation between the sparsity\nof the adjacency matrix and the performance of the 1 layer GCN model. When the number of facts\nretrieved is large and the matrix is less sparse, the 1 layer GCN model makes a wrong prediction.\nThis indicates that the 2nd layer of the GCN allows for more message passing and provides a stronger\nsignal when there are many facts to analyze.\nWe compare the accuracy of our model with the FVQA baselines and our previous work, STTF in\nTab. 2. The accuracy reported here is averaged over all \ufb01ve train-test splits. As shown, our best\nmodel \u201813\u2019 outperforms the state-of-the-art STTF technique by more than 7% and the FVQA baseline\nwithout ensemble by over 12%. Note that combining GCN and MLP clearly outperforms usage of\nonly one part. FVQA and STTF both try to predict the ground truth fact. If the fact is predicted\nincorrectly, the answer will also be wrong, thus causing the model to fail. Our method circumvents\npredicting the fact and instead uses multiple relevant facts to predict the answer. This approach clearly\nworks better.\nSynonyms and homographs: Here we show the improvements of our model compared to the\nbaseline with respect to synonyms and homographs. To retrieve the top 100 facts, we use trainable\nword embeddings which are known to group synonyms and separate homographs.\nWe ran additional tests using Wordnet to determine the number of question-fact pairs which contain\nsynonyms. The test data contains 1105 such pairs out of which our model predicts 95.38% correctly,\nwhereas the FVQA and STTF models predict 78% and the 91.6% correctly. In addition, we manually\ngenerated 100 synonymous questions by replacing words in the questions with synonyms (e.g. \u201cWhat\nin the bowl can you eat?\", is rephrased as, \u201cWhat in the bowl is edible?\"). Tests on these 100 new\nsamples \ufb01nd that our model predicts 91 of these correctly, whereas the key-word matching FVQA\ntechnique gets only 61 of these right. As STTF also uses GloVe embeddings, it gets 89 correct.\nWith regards to homographs, the test set has 998 questions which contain words that have multiple\nmeanings across facts. Our model predicts correct answers for 81.16%, whereas the FVQA model\nand STTF model get 66.33% and 79.4% correct, respectively.\nQualitative results: As described in Sec. 3.2, the image embedding is constructed based on the\nvisual concepts detected in the image. Fig. 3 shows the object, scene, and action detection for two\nexamples in our dataset. We also indicate the question corresponding to the image, the supporting\nfact, relation, and answer detected by our model. Using the high-level features helps summarize the\nsalient content in the image as the facts are closely related to the visual concepts. We observe our\nmodel to work well even when the question does not focus on the main visual concept in the image.\nTab. 2 shows that including the visual concept improves the accuracy of our model by nearly 20%.\nFig. 4 depicts a few success and failure examples of our method. In our model, predicting the correct\nanswer involves three main steps: (1) Selecting the right supporting fact in the Top-100 facts, f100;\n(2) Predicting the right relation; (3) Selecting the right entity in the GCN. In the top two rows of\nexamples, our model correctly executes all the three steps. As shown, our model works for visual\nconcepts of all three types, i.e., actions, scenes and objects. Examples in the second row indicates\nthat our model works well with synonyms and homographs as we use GloVe embeddings of words.\nThe second example in the second row shows that our method obtains the right answer even when the\n\n8\n\nQuestion:Which object in this image can fly?Predicted/GT Relation:CapableOfRelevant Visual Concept: FrisbeeSupporting Fact:(Frisbee, CapableOf, Flying)Predicted/GT Answer:FrisbeeQuestion:Which is the yellow fruit in this image?Predicted/GT Relation:IsARelevant Visual Concept: BananaSupporting Fact:(Banana, IsA, Yellow fruit)Predicted/GT Answer:Banana\fFigure 4: Success and failure cases: Success cases are shown in the top two rows. Our method correctly predicts\nthe relation, visual concept, and the answer. The bottom row shows three different failure cases.\n\nquestion and the fact do not have many words in common. This is due to the comparison with visual\nconcepts while retrieving the facts.\nThe last row shows failure cases. Our method fails if any of the three steps produce incorrect output.\nIn the \ufb01rst example the ground-truth fact (Airplane, UsedFor, Flying) isn\u2019t part of the top-100. This\nhappens when words in the fact are neither related to the words in the question nor the list of visual\nconcepts. A second failure mode is due to wrong node/entity predictions (selecting laptop instead\nof keyboard), e.g., because a similar fact, (Laptop, UsedFor, Data processing) exists. These type of\nerrors are rare (Tab. 3) and happen only when the fact space contains a fact similar to the ground truth\none. The third failure mode is due to relation prediction accuracies which are around 75%, and 92%\nfor Top-1 and Top-3 respectively, as shown in [38].\n\n5 Conclusions\nWe developed a method for \u2018reasoning\u2019 in factual visual question answering using graph convolution\nnets. We showed that our proposed algorithm outperforms existing baselines by a large margin of\n7%. We attribute these improvements to \u2018joint reasoning about answers,\u2019 which facilitates sharing\nof information before making an informed decision. Further, we achieve this high increase in\nperformance by using only the ground truth relation and answer information, with no reliance on the\nground truth fact. Currently, all the components of our model except for fact retrieval are trainable\nend-to-end. In the future, we plan to extend our network to incorporate this step into a uni\ufb01ed\nframework.\nAcknowledgments: This material is based upon work supported in part by the National Science\nFoundation under Grant No. 1718221 and Grant No. 1563727, Samsung, 3M, IBM-ILLINOIS Center\nfor Cognitive Computing Systems Research (C3SR), Amazon Research Award, and AWS Machine\nLearning Research Award. We thank NVIDIA for providing the GPUs used for this research. We\nalso thank Arun Mallya and Aditya Deshpande for their help.\n\n9\n\nQuestion: What in this image is made by baking? Pred. Relation: CategoryPred. Visual Concept: Donut (object) Supporting Fact: (Donut, Category, Cooking) Pred./GT Answer: DonutQuestion: What object in this image is spiky? Pred. Relation: RelatedToPred. Visual Concept: Pineapple (object) Supporting Fact: (Pineapple, RelatedTo, Spiky)Pred./GT Answer: PineappleQuestion: Which object in this image is venomous?Pred. Relation: HasPropertyPred. Visual Concept: Snake (object) Supporting Fact: (Snake, HasProperty, Venomous)Pred./GT Answer: SnakeQuestion: Which action shown here is faster than walking?Pred. Relation: Comparative (faster)Pred. Visual Concept: Cycling (action) Supporting Fact: (Cycling, Faster, Walking)Pred./GT Answer: CyclingQuestion: Which vehicle shown here can float? Pred. Relation: CapableOfPred. Visual Concept: Boat (object) Supporting Fact: (Boat, CapableOf, Sailing) Pred./GT Answer: BoatQuestion: What is the place in this image used for? Pred. Relation: UsedForPred. Visual Concept: Kitchen (scene) Supporting Fact: (Kitchen, UsedFor, Cooking) Pred./GT Answer: KitchenQuestion: What does the animal in the image like to chase? Pred. Relation: CapableOfPred. Visual Concept: Cat (object) Supporting Fact: (Cat, CapableOf, Hunting mice) Pred./GT Answer: CatQuestion:What is the plant-eating animal shown here? Pred. Relation: CategoryPred. Visual Concept: Giraffe (object) Supporting Fact: (Giraffe, Category, Herbivore) Pred./GT Answer: GiraffeQuestion:What is the object that the picture is taken from used for?Pred. Relation:UsedForGT Supporting Fact:(Airplane, UsedFor, Flying)Pred. Answer:Printing pictures GT Answer:Flying Error: GT Fact not retrieved in Top\u2013100. Question: What object in this image is used for entering data? Pred. Relation: UsedForGT Supporting Fact: (Keyboard, UsedFor, Data entry) Pred. Answer: LaptopGT Answer: KeyboardError: GCN predicted the wrong node.Question: What object in this image is used to play polka music? Pred. Relation: UsedForGT Relation: ReceivesActionGT Supporting Fact: (Accordion, ReceivesAction, Polka Music) Pred. Answer: GuitarGT Answer: AccordionError: Incorrect annotation / Wrong relation predicted.Question: What game is shown here? Pred. Relation: IsAGT Supporting Fact: (Rock-paper-scissors, IsA, Game) Pred. Answer: SoccerGT Answer: Rock-paper-scissors Error: GT Fact not retrieved in Top\u2013100. \fReferences\n[1] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Deep compositional question answering with neural\n\nmodule networks. In CVPR, 2016.\n\n[2] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual Question\n\nAnswering. In ICCV, 2015.\n\n[3] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives. Dbpedia: A nucleus for a web of\n\nopen data. In ISWC/ASWC, 2007.\n\n[4] H. Ben-younes, R. Cadene, M. Cord, and N. Thome. Mutan: Multimodal tucker fusion for visual question\n\nanswering. In ICCV, 2017.\n\n[5] J. Berant, A. Chou, R. Frostig, and P. Liang. Semantic Parsing on Freebase from Question-Answer Pairs.\n\nIn EMNLP, 2013.\n\n[6] J. Berant and P. Liang. Semantic parsing via paraphrasing. In ACL, 2014.\n[7] A. Bordes, S. Chopra, and J. Weston. Question answering with sub-graph embeddings. In EMNLP, 2014.\n[8] A. Bordes, N. Usunier, S. Chopra, and J. Weston. Large-scale simple question answering with memory\n\n[9] A. Bordes, J. Weston, and N. Usunier. Open question answering with weakly supervised embedding\n\nnetworks. In ICLR, 2015.\n\nmodels. In ECML, 2014.\n\n2013.\n\n[10] Q. Cai and A. Yates. Large-scale Semantic Parsing via Schema Matching and Lexicon Extension. In ACL,\n\n[11] A. Das, H. Agrawal, C. L. Zitnick, D. Parikh, and D. Batra. Human attention in visual question answering:\n\nDo humans and deep networks look at the same regions? In EMNLP, 2016.\n\n[12] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra. Visual Dialog. In\n\nCVPR, 2017.\n\n[13] A. Das, S. Kottur, J. M. Moura, S. Lee, and D. Batra. Learning cooperative visual dialog agents with deep\n\nreinforcement learning. arXiv:1703.06585, 2017.\n\n[14] L. Dong, F. Wei, M. Zhou, and K. Xu. Question answering over freebase with multi-column convolutional\n\nneural networks. In ACL, 2015.\n\n[15] A. Fader, L. Zettlemoyer, and O. Etzioni. Open question answering over curated and extracted knowledge\n\nbases. In KDD, 2014.\n\n[16] A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear\n\npooling for visual question answering and visual grounding. In EMNLP, 2016.\n\n[17] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? Dataset and\n\nMethods for Multilingual Image Question Answering. In NeurIPS, 2015.\n\n[18] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. Iqa: Visual question answering\n\nin interactive environments. In CVPR, 2018.\n\n[19] A. Jabri, A. Joulin, and L. van der Maaten. Revisiting Visual Question Answering Baselines. In ECCV,\n\n2016.\n\n[20] U. Jain, S. Lazebnik, and A. G. Schwing. Two can play this Game: Visual Dialog with Discriminative\n\nQuestion Generation and Answering. In CVPR, 2018.\n\n[21] U. Jain, Z. Zhang, and A. G. Schwing. Creativity: Generating Diverse Questions using Variational\n\nAutoencoders. In CVPR, 2017.\n\n[22] A. Jiang, F. Wang, F. Porikli, and Y. Li. Compositional memory for visual question answering.\n\narXiv:1511.05676, 2015.\n\n[23] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. Clevr: A diagnostic\n\ndataset for compositional language and elementary visual reasoning. In CVPR, 2017.\n\n[24] J.-H. Kim, S.-W. L. D.-H. Kwak, M.-O. Heo, J. Kim, J.-W. Ha, and B.-T. Zhang. Multimodal residual\n\nlearning for visual qa. In NeurIPS, 2016.\n\n[25] T. N. Kipf and M. Welling.\n\narXiv:1609.02907, 2016.\n\nSemi-supervised classi\ufb01cation with graph convolutional networks.\n\n[26] O. Kolomiyets and M.-F. Moens. A survey on question answering technology from an information retrieval\n\nperspective. In Information Sciences, 2011.\n\n[27] J. Krishnamurthy and T. Kollar. Jointly learning to parse and perceive: Connecting natural language to the\n\nphysical world. In ACL, 2013.\n\n[28] T. Kwiatkowski, E. Choi, Y. Artzi, and L. Zettlemoyer. Scaling semantic parsers with on-the-\ufb02y ontology\n\nmatching. In EMNLP, 2013.\n\n[29] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition.\n\nIEEE, 1998.\n\n[30] Y. Li, N. Duan, B. Zhou, X. Chu, W. Ouyang, and X. Wang. Visual question generation as dual task of\n\nvisual question answering. arXiv:1709.07192, 2017.\n\n[31] P. Liang, M. I. Jordan, and D. Klein. Learning dependency-based compositional semantics. In Computa-\n\ntional Linguistics, 2013.\n\n[32] J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question\n\nanswering. In NeurIPS, 2016.\n\n10\n\n\f[33] L. Ma, Z. Lu, and H. Li. Learning to answer questions from image using convolutional neural network. In\n\nAAAI, 2016.\n\n[34] M. Malinowski and M. Fritz. A Multi-World Approach to Question Answering about Real-World Scenes\n\nbased on Uncertain Input. In NeurIPS, 2014.\n\n[35] M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering\n\nquestions about images. In ICCV, 2015.\n\n[36] N. Mostafazadeh, I. Misra, J. Devlin, M. Mitchell, X. He, and L. Vanderwende. Generating natural\n\nquestions about an image. arXiv:1603.06059, 2016.\n\n[37] K. Narasimhan, A. Yala, and R. Barzilay. Improving information extraction by acquiring external evidence\n\nwith reinforcement learning. In EMNLP, 2016.\n\n[38] M. Narasimhan and A. G. Schwing. Straight to the facts: Learning knowledge base retrieval for factual\n\nvisual question answering. In ECCV, 2018.\n\n[39] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP,\n\n[40] S. Reddy, O. T\u00e4ckstr\u00f6m, M. Collins, T. Kwiatkowski, D. Das, M. Steedman, and M. Lapata. Transforming\n\ndependency structures to logical forms for semantic parsing. In ACL, 2016.\n\n[41] M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In NeurIPS,\n\n2014.\n\n2015.\n\n2016.\n\n[42] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. v. d. Berg, I. Titov, and M. Welling. Modeling relational data\n\nwith graph convolutional networks. In ESWC, 2018.\n\n[43] I. Schwartz, A. G. Schwing, and T. Hazan. High-Order Attention Models for Visual Question Answering.\n\nIn NeurIPS, 2017.\n\n[44] K. J. Shih, S. Singh, and D. Hoiem. Where to look: Focus regions for visual question answering. In CVPR,\n\n[45] R. Speer, J. Chin, and C. Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In\n\nAAAI, 2017.\n\n[46] S. W. t. Yih, M.-W. Chang, X. He, and J. Gao. Semantic parsing via staged query graph generation:\n\nQuestion answering with knowledge base. In ACL-IJCNLP, 2015.\n\n[47] N. Tandon, G. de Melo, F. Suchanek, and G. Weikum. Webchild: Harvesting and organizing commonsense\n\nknowledge from the web. In WSDM, 2014.\n\n[48] C. Unger, L. B\u00fchmann, J. Lehmann, A.-C. N. Ngomo, D. Gerber, and P. Cimiano. Template-based question\n\nanswering over RDF data. In WWW, 2012.\n\n[49] A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. Crandall, and D. Batra. Diverse\n\nbeam search: Decoding diverse solutions from neural sequence models. arXiv:1610.02424, 2016.\n\n[50] P. Wang, Q. Wu, C. Shen, A. Dick, and A. v. d. Hengel. Fvqa: Fact-based visual question answering.\n\nTPAMI, 2018.\n\n[51] P. Wang, Q. Wu, C. Shen, A. Dick, and A. Van Den Henge. Explicit knowledge-based reasoning for visual\n\nquestion answering. In IJCAI, 2017.\n\n[52] X. Wang, Y. Ye, and A. Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In\n\nCVPR, 2018.\n\n[53] Q. Wu, C. Shen, A. van den Hengel, P. Wang, and A. Dick. Image captioning and visual question answering\n\nbased on attributes and their related external knowledge. arXiv:1603.02814, 2016.\n\n[54] Q. Wu, P. Wang, C. Shen, A. Dick, and A. van den Hengel. Ask me anything: Free-form visual question\n\nanswering based on knowledge from external sources. In CVPR, 2016.\n\n[55] C. Xiao, M. Dymetman, and C. Gardent. Sequence-based structured prediction for semantic parsing. In\n\nACL, 2016.\n\n[56] C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering.\n\nIn ICML, 2016.\n\n[57] H. Xu and K. Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual\n\nquestion answering. In ECCV, 2016.\n\n[58] X.Yao and B. V. Durme. Information extraction over structured data: Question answering with Freebase.\n\nIn ACL, 2014.\n\n[59] Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering.\n\nIn CVPR, 2016.\n\n[60] L. Yu, E. Park, A. Berg, and T. Berg. Visual madlibs: Fill in the blank image generation and question\n\nanswering. In ICCV, 2015.\n\n[61] L. S. Zettlemoyer and M. Collins. Learning context-dependent mappings from sentences to logical form.\n\nIn ACL, 2005.\n\n[62] L. S. Zettlemoyer and M. Collins. Learning to map sentences to logical form: Structured classi\ufb01cation\n\nwith probabilistic categorial grammars. In UAI, 2005.\n\n[63] P. Zhang, Y. Goyal, D. Summers-Stay, D. Batra, and D. Parikh. Yin and yang: Balancing and answering\n\nbinary visual questions. arXiv:1511.05099, 2015.\n\n[64] Y. Zhang, K. Liu, S. He, G. Ji, Z. Liu, H. Wu, and J. Zhao. Question answering over knowledge base with\n\nneural attention combining global knowledge information. arXiv:1606.00979, 2016.\n\n11\n\n\f[65] B. Zhou, Y. Tian, S. Sukhbataar, A. Szlam, and R. Fergus. Simple baseline for visual question answering.\n\n[66] Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7W: Grounded Question Answering in Images. In\n\narXiv:1512.02167, 2015.\n\nCVPR, 2016.\n\n[67] Y. Zhu, C. Zhang, C. R\u00e9, and L. Fei-Fei. Building a large-scale multimodal Knowledge Base for Visual\n\nQuestion Answering. In CoRR, 2015.\n\n[68] C. L. Zitnick, A. Agrawal, S. Antol, M. Mitchell, D. Batra, and D. Parikh. Measuring machine intelligence\n\nthrough visual question answering. AI Magazine, 2016.\n\n12\n\n\f", "award": [], "sourceid": 1357, "authors": [{"given_name": "Medhini", "family_name": "Narasimhan", "institution": "UIUC"}, {"given_name": "Svetlana", "family_name": "Lazebnik", "institution": "UIUC"}, {"given_name": "Alexander", "family_name": "Schwing", "institution": "University of Illinois at Urbana-Champaign"}]}