{"title": "Recurrent Models of Visual Attention", "book": "Advances in Neural Information Processing Systems", "page_first": 2204, "page_last": 2212, "abstract": "Applying convolutional neural networks to large images is computationally expensive because the amount of computation scales linearly with the number of image pixels. We present a novel recurrent neural network model that is capable of extracting information from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution. Like convolutional neural networks, the proposed model has a degree of translation invariance built-in, but the amount of computation it performs can be controlled independently of the input image size. While the model is non-differentiable, it can be trained using reinforcement learning methods to learn task-specific policies. We evaluate our model on several image classification tasks, where it significantly outperforms a convolutional neural network baseline on cluttered images, and on a dynamic visual control problem, where it learns to track a simple object without an explicit training signal for doing so.", "full_text": "Recurrent Models of Visual Attention\n\nVolodymyr Mnih Nicolas Heess Alex Graves Koray Kavukcuoglu\n\nGoogle DeepMind\n\n{vmnih,heess,gravesa,korayk} @ google.com\n\nAbstract\n\nApplying convolutional neural networks to large images is computationally ex-\npensive because the amount of computation scales linearly with the number of\nimage pixels. We present a novel recurrent neural network model that is ca-\npable of extracting information from an image or video by adaptively selecting\na sequence of regions or locations and only processing the selected regions at\nhigh resolution. Like convolutional neural networks, the proposed model has a\ndegree of translation invariance built-in, but the amount of computation it per-\nforms can be controlled independently of the input image size. While the model\nis non-differentiable, it can be trained using reinforcement learning methods to\nlearn task-speci\ufb01c policies. We evaluate our model on several image classi\ufb01cation\ntasks, where it signi\ufb01cantly outperforms a convolutional neural network baseline\non cluttered images, and on a dynamic visual control problem, where it learns to\ntrack a simple object without an explicit training signal for doing so.\n\n1\n\nIntroduction\n\nNeural network-based architectures have recently had great success in signi\ufb01cantly advancing the\nstate of the art on challenging image classi\ufb01cation and object detection datasets [8, 12, 19]. Their\nexcellent recognition accuracy, however, comes at a high computational cost both at training and\ntesting time. The large convolutional neural networks typically used currently take days to train on\nmultiple GPUs even though the input images are downsampled to reduce computation [12]. In the\ncase of object detection processing a single image at test time currently takes seconds when running\non a single GPU [8, 19] as these approaches effectively follow the classical sliding window paradigm\nfrom the computer vision literature where a classi\ufb01er, trained to detect an object in a tightly cropped\nbounding box, is applied independently to thousands of candidate windows from the test image at\ndifferent positions and scales. Although some computations can be shared, the main computational\nexpense for these models comes from convolving \ufb01lter maps with the entire input image, therefore\ntheir computational complexity is at least linear in the number of pixels.\nOne important property of human perception is that one does not tend to process a whole scene\nin its entirety at once. Instead humans focus attention selectively on parts of the visual space to\nacquire information when and where it is needed, and combine information from different \ufb01xations\nover time to build up an internal representation of the scene [18], guiding future eye movements\nand decision making. Focusing the computational resources on parts of a scene saves \u201cbandwidth\u201d\nas fewer \u201cpixels\u201d need to be processed. But it also substantially reduces the task complexity as\nthe object of interest can be placed in the center of the \ufb01xation and irrelevant features of the visual\nenvironment (\u201cclutter\u201d) outside the \ufb01xated region are naturally ignored.\nIn line with its fundamental role, the guidance of human eye movements has been extensively studied\nin neuroscience and cognitive science literature. While low-level scene properties and bottom up\nprocesses (e.g. in the form of saliency; [11]) play an important role, the locations on which humans\n\ufb01xate have also been shown to be strongly task speci\ufb01c (see [9] for a review and also e.g. [15, 22]). In\nthis paper we take inspiration from these results and develop a novel framework for attention-based\ntask-driven visual processing with neural networks. Our model considers attention-based processing\n\n1\n\n\fof a visual scene as a control problem and is general enough to be applied to static images, videos,\nor as a perceptual module of an agent that interacts with a dynamic visual environment (e.g. robots,\ncomputer game playing agents).\nThe model is a recurrent neural network (RNN) which processes inputs sequentially, attending to\ndifferent locations within the images (or video frames) one at a time, and incrementally combines\ninformation from these \ufb01xations to build up a dynamic internal representation of the scene or envi-\nronment. Instead of processing an entire image or even bounding box at once, at each step, the model\nselects the next location to attend to based on past information and the demands of the task. Both\nthe number of parameters in our model and the amount of computation it performs can be controlled\nindependently of the size of the input image, which is in contrast to convolutional networks whose\ncomputational demands scale linearly with the number of image pixels. We describe an end-to-end\noptimization procedure that allows the model to be trained directly with respect to a given task and\nto maximize a performance measure which may depend on the entire sequence of decisions made by\nthe model. This procedure uses backpropagation to train the neural-network components and policy\ngradient to address the non-differentiabilities due to the control problem.\nWe show that our model can learn effective task-speci\ufb01c strategies for where to look on several\nimage classi\ufb01cation tasks as well as a dynamic visual control problem. Our results also suggest that\nan attention-based model may be better than a convolutional neural network at both dealing with\nclutter and scaling up to large input images.\n2 Previous Work\nComputational limitations have received much attention in the computer vision literature. For in-\nstance, for object detection, much work has been dedicated to reducing the cost of the widespread\nsliding window paradigm, focusing primarily on reducing the number of windows for which the\nfull classi\ufb01er is evaluated, e.g. via classi\ufb01er cascades (e.g. [7, 24]), removing image regions from\nconsideration via a branch and bound approach on the classi\ufb01er output (e.g. [13]), or by proposing\ncandidate windows that are likely to contain objects (e.g. [1, 23]). Even though substantial speedups\nmay be obtained with such approaches, and some of these can be combined with or used as an add-on\nto CNN classi\ufb01ers [8], they remain \ufb01rmly rooted in the window classi\ufb01er design for object detection\nand only exploit past information to inform future processing of the image in a very limited way.\nA second class of approaches that has a long history in computer vision and is strongly motivated\nby human perception are saliency detectors (e.g. [11]). These approaches prioritize the processing\nof potentially interesting (\u201csalient\u201d) image regions which are typically identi\ufb01ed based on some\nmeasure of local low-level feature contrast. Saliency detectors indeed capture some of the properties\nof human eye movements, but they typically do not to integrate information across \ufb01xations, their\nsaliency computations are mostly hardwired, and they are based on low-level image properties only,\nusually ignoring other factors such as semantic content of a scene and task demands (but see [22]).\nSome works in the computer vision literature and elsewhere e.g. [2, 4, 6, 14, 16, 17, 20] have em-\nbraced vision as a sequential decision task as we do here. There, as in our work, information about\nthe image is gathered sequentially and the decision where to attend next is based on previous \ufb01xa-\ntions of the image. [4] employs the learned Bayesian observer model from [5] to the task of object\ndetection. The learning framework of [5] is related to ours as they also employ a policy gradient\nformulation (cf. section 3) but their overall setup is considerably more restrictive than ours and only\nsome parts of the system are learned.\nOur work is perhaps the most similar to the other attempts to implement attentional processing in a\ndeep learning framework [6, 14, 17]. Our formulation which employs an RNN to integrate visual\ninformation over time and to decide how to act is, however, more general, and our learning procedure\nallows for end-to-end optimization of the sequential decision process instead of relying on greedy\naction selection. We further demonstrate how the same general architecture can be used for ef\ufb01cient\nobject recognition in still images as well as to interact with a dynamic visual environment in a\ntask-driven way.\n\n3 The Recurrent Attention Model (RAM)\nIn this paper we consider the attention problem as the sequential decision process of a goal-directed\nagent interacting with a visual environment. At each point in time, the agent observes the environ-\nment only via a bandwidth-limited sensor, i.e. it never senses the environment in full. It may extract\n\n2\n\n\fFigure 1: A) Glimpse Sensor: Given the coordinates of the glimpse and an input image, the sen-\nsor extracts a retina-like representation \u03c1(xt, lt\u22121) centered at lt\u22121 that contains multiple resolution\npatches. B) Glimpse Network: Given the location (lt\u22121) and input image (xt), uses the glimpse\nsensor to extract retina representation \u03c1(xt, lt\u22121). The retina representation and glimpse location is\nthen mapped into a hidden space using independent linear layers parameterized by \u03b80\ng respec-\ng to combine the information from both\ntively using recti\ufb01ed units followed by another linear layer \u03b82\ng}) de\ufb01nes a trainable bandwidth limited sensor\ncomponents. The glimpse network fg(.;{\u03b80\nfor the attention network producing the glimpse representation gt. C) Model Architecture: Overall,\nthe model is an RNN. The core network of the model fh(.; \u03b8h) takes the glimpse representation gt as\ninput and combining with the internal representation at previous time step ht\u22121, produces the new\ninternal state of the model ht. The location network fl(.; \u03b8l) and the action network fa(.; \u03b8a) use the\ninternal state ht of the model to produce the next location to attend to lt and the action/classi\ufb01cation\nat respectively. This basic RNN iteration is repeated for a variable number of steps.\n\ng and \u03b81\n\ng, \u03b81\n\ng, \u03b82\n\ninformation only in a local region or in a narrow frequency band. The agent can, however, actively\ncontrol how to deploy its sensor resources (e.g. choose the sensor location). The agent can also\naffect the true state of the environment by executing actions. Since the environment is only partially\nobserved the agent needs to integrate information over time in order to determine how to act and\nhow to deploy its sensor most effectively. At each step, the agent receives a scalar reward (which\ndepends on the actions the agent has executed and can be delayed), and the goal of the agent is to\nmaximize the total sum of such rewards.\nThis formulation encompasses tasks as diverse as object detection in static images and control prob-\nlems like playing a computer game from the image stream visible on the screen. For a game, the\nenvironment state would be the true state of the game engine and the agent\u2019s sensor would operate\non the video frame shown on the screen. (Note that for most games, a single frame would not fully\nspecify the game state). The environment actions here would correspond to joystick controls, and\nthe reward would re\ufb02ect points scored. For object detection in static images the state of the envi-\nronment would be \ufb01xed and correspond to the true contents of the image. The environmental action\nwould correspond to the classi\ufb01cation decision (which may be executed only after a \ufb01xed number\nof \ufb01xations), and the reward would re\ufb02ect if the decision is correct.\n3.1 Model\nThe agent is built around a recurrent neural network as shown in Fig. 1. At each time step, it\nprocesses the sensor data, integrates information over time, and chooses how to act and how to\ndeploy its sensor at next time step:\nSensor: At each step t the agent receives a (partial) observation of the environment in the form of\nan image xt. The agent does not have full access to this image but rather can extract information\nfrom xt via its bandwidth limited sensor \u03c1, e.g. by focusing the sensor on some region or frequency\nband of interest.\nIn this paper we assume that the bandwidth-limited sensor extracts a retina-like representation\n\u03c1(xt, lt\u22121) around location lt\u22121 from image xt. It encodes the region around l at a high-resolution\nbut uses a progressively lower resolution for pixels further from l, resulting in a vector of much\n\n3\n\nlt-1gtGlimpseSensorxt\u03c1(xt , lt-1)\u03b8g0\u03b8g1\u03b8g2Glimpse Network : fg(\u03b8g )lt-1gtltatltgt+1lt+1at+1htht+1fg(\u03b8g)ht-1fl(\u03b8l)fa(\u03b8a)fh(\u03b8h)fg(\u03b8g)fl(\u03b8l)fa(\u03b8a)fh(\u03b8h)xt\u03c1(xt , lt-1)lt-1Glimpse SensorA)B)C)\fg, \u03b82\n\ng, \u03b81\n\ng} (Fig. 1B).\n\nlower dimensionality than the original image x. We will refer to this low-resolution representation\nas a glimpse [14]. The glimpse sensor is used inside what we call the glimpse network fg to produce\nthe glimpse feature vector gt = fg(xt, lt\u22121; \u03b8g) where \u03b8g = {\u03b80\nInternal state: The agent maintains an interal state which summarizes information extracted from\nthe history of past observations; it encodes the agent\u2019s knowledge of the environment and is in-\nstrumental to deciding how to act and where to deploy the sensor. This internal state is formed\nby the hidden units ht of the recurrent neural network and updated over time by the core network:\nht = fh(ht\u22121, gt; \u03b8h). The external input to the network is the glimpse feature vector gt.\nActions: At each step, the agent performs two actions: it decides how to deploy its sensor via the\nsensor control lt, and an environment action at which might affect the state of the environment.\nThe nature of the environment action depends on the task. In this work, the location actions are\nchosen stochastically from a distribution parameterized by the location network fl(ht; \u03b8l) at time t:\nlt \u223c p(\u00b7|fl(ht; \u03b8l)). The environment action at is similarly drawn from a distribution conditioned\non a second network output at \u223c p(\u00b7|fa(ht; \u03b8a)). For classi\ufb01cation it is formulated using a softmax\noutput and for dynamic environments, its exact formulation depends on the action set de\ufb01ned for\nthat particular environment (e.g.\njoystick movements, motor control, ...). Finally, our model can\nalso be augmented with an additional action that decides when it will stop taking glimpses. This\ncould, for example, be used to learn a cost-sensitive classi\ufb01er by giving the agent a negative reward\nfor each glimpse it takes, forcing it to trade off making correct classi\ufb01cations with the cost of taking\nmore glimpses.\nReward: After executing an action the agent receives a new visual observation of the environment\nxt+1 and a reward signal rt+1. The goal of the agent is to maximize the sum of the reward signal1\nt=1 rt. In the case of object recognition, for\n\nwhich is usually very sparse and delayed: R = (cid:80)T\n\nexample, rT = 1 if the object is classi\ufb01ed correctly after T steps and 0 otherwise.\nThe above setup is a special instance of what is known in the RL community as a Partially Observ-\nable Markov Decision Process (POMDP). The true state of the environment (which can be static or\ndynamic) is unobserved. In this view, the agent needs to learn a (stochastic) policy \u03c0((lt, at)|s1:t; \u03b8)\nwith parameters \u03b8 that, at each step t, maps the history of past interactions with the environment\ns1:t = x1, l1, a1, . . . xt\u22121, lt\u22121, at\u22121, xt to a distribution over actions for the current time step, sub-\nject to the constraint of the sensor. In our case, the policy \u03c0 is de\ufb01ned by the RNN outlined above,\nand the history st is summarized in the state of the hidden units ht. We will describe the speci\ufb01c\nchoices for the above components in Section 4.\n\n3.2 Training\nThe parameters of our agent are given by the parameters of the glimpse network, the core network\n(Fig. 1C), and the action network \u03b8 = {\u03b8g, \u03b8h, \u03b8a} and we learn these to maximize the total reward\nthe agent can expect when interacting with the environment.\nMore formally, the policy of the agent, possibly in combination with the dynamics of the environ-\nment (e.g. for game-playing), induces a distribution over possible interaction sequences s1:N and we\n= Ep(s1:T ;\u03b8) [R],\naim to maximize the reward under this distribution: J(\u03b8) = Ep(s1:T ;\u03b8)\nwhere p(s1:T ; \u03b8) depends on the policy\nMaximizing J exactly is non-trivial since it involves an expectation over the high-dimensional inter-\naction sequences which may in turn involve unknown environment dynamics. Viewing the problem\nas a POMDP, however, allows us to bring techniques from the RL literature to bear: As shown by\nWilliams [26] a sample approximation to the gradient is given by\n\n(cid:104)(cid:80)T\n\nt=1 rt\n\n(cid:105)\n\nT(cid:88)\n\nt=1\n\nM(cid:88)\n\nT(cid:88)\n\ni=1\n\nt=1\n\n\u2207\u03b8J =\n\nEp(s1:T ;\u03b8) [\u2207\u03b8 log \u03c0(ut|s1:t; \u03b8)R] \u2248 1\nM\n\n\u2207\u03b8 log \u03c0(ui\n\nt|si\n\n1:t; \u03b8)Ri,\n\n(1)\n\nwhere si\u2019s are interaction sequences obtained by running the current agent \u03c0\u03b8 for i = 1 . . . M\nepisodes.\n\nrewards obtained in the distant future contribute less: R =(cid:80)T\n\n1Depending on the scenario it may be more appropriate to consider a sum of discounted rewards, where\nt=1 \u03b3t\u22121rt. In this case we can have T \u2192 \u221e.\n\n4\n\n\fM(cid:88)\n\nT(cid:88)\n\ni=1\n\nt=1\n\n1\nM\n\n\u2207\u03b8 log \u03c0(ui\n\nt|si\n\n1:t; \u03b8)(cid:0)Ri\n\nt \u2212 bt\n\n(cid:1) ,\n\n(2)\n\nThe learning rule (1) is also known as the REINFORCE rule, and it involves running the agent with\nits current policy to obtain samples of interaction sequences s1:T and then adjusting the parameters\n\u03b8 of our agent such that the log-probability of chosen actions that have led to high cumulative reward\nis increased, while that of actions having produced low reward is decreased.\nEq. (1) requires us to compute \u2207\u03b8 log \u03c0(ui\nde\ufb01nes our agent evaluated at time step t and can be computed by standard backpropagation [25].\nVariance Reduction : Equation (1) provides us with an unbiased estimate of the gradient but it may\nhave high variance. It is therefore common to consider a gradient estimate of the form\n\n1:t; \u03b8). But this is just the gradient of the RNN that\n\nt|si\n\nt =(cid:80)T\n\nt(cid:48)=1 ri\n\n1:t (e.g. via hi\n\nt) but not on the action ui\n\nt(cid:48) is the cumulative reward obtained following the execution of action ui\n\nt, and\nwhere Ri\nbt is a baseline that may depend on si\nt itself. This estimate\nis equal to (1) in expectation but may have lower variance. It is natural to select bt = E\u03c0 [Rt] [21],\nand this form of baseline known as the value function in the reinforcement learning literature. The\nresulting algorithm increases the log-probability of an action that was followed by a larger than\nexpected cumulative reward, and decreases the probability if the obtained cumulative reward was\nt\u2019s and bt.\nsmaller. We use this type of baseline and learn it by reducing the squared error between Ri\nUsing a Hybrid Supervised Loss: The algorithm described above allows us to train the agent when\nthe \u201cbest\u201d actions are unknown, and the learning signal is only provided via the reward. For instance,\nwe may not know a priori which sequence of \ufb01xations provides most information about an unknown\nimage, but the total reward at the end of an episode will give us an indication whether the tried\nsequence was good or bad.\nHowever, in some situations we do know the correct action to take: For instance, in an object\ndetection task the agent has to output the label of the object as the \ufb01nal action. For the training\nimages this label will be known and we can directly optimize the policy to output the correct label\nassociated with a training image at the end of an observation sequence. This can be achieved, as is\ncommon in supervised learning, by maximizing the conditional probability of the true label given\nthe observations from the image, i.e. by maximizing log \u03c0(a\u2217\nT corresponds to the\nground-truth label(-action) associated with the image from which observations s1:T were obtained.\nWe follow this approach for classi\ufb01cation problems where we optimize the cross entropy loss to\ntrain the action network fa and backpropagate the gradients through the core and glimpse networks.\nThe location network fl is always trained with REINFORCE.\n4 Experiments\nWe evaluated our approach on several image classi\ufb01cation tasks as well as a simple game. We \ufb01rst\ndescribe the design choices that were common to all our experiments:\nRetina and location encodings: The retina encoding \u03c1(x, l) extracts k square patches centered at\nlocation l, with the \ufb01rst patch being gw \u00d7 gw pixels in size, and each successive patch having twice\nthe width of the previous. The k patches are then all resized to gw \u00d7 gw and concatenated. Glimpse\nlocations l were encoded as real-valued (x, y) coordinates2 with (0, 0) being the center of the image\nx and (\u22121,\u22121) being the top left corner of x.\nGlimpse network: The glimpse network fg(x, l) had two fully connected layers. Let Linear(x) de-\nnote a linear transformation of the vector x, i.e. Linear(x) = W x+b for some weight matrix W and\nbias vector b, and let Rect(x) = max(x, 0) be the recti\ufb01er nonlinearity. The output g of the glimpse\nnetwork was de\ufb01ned as g = Rect(Linear(hg) + Linear(hl)) where hg = Rect(Linear(\u03c1(x, l)))\nand hl = Rect(Linear(l)). The dimensionality of hg and hl was 128 while the dimensionality of\ng was 256 for all attention models trained in this paper.\nLocation network: The policy for the locations l was de\ufb01ned by a two-component Gaussian with a\n\ufb01xed variance. The location network outputs the mean of the location policy at time t and is de\ufb01ned\nas fl(h) = Linear(h) where h is the state of the core network/RNN.\n\nT|s1:T ; \u03b8), where a\u2217\n\n2We also experimented with using a discrete representation for the locations l but found that it was dif\ufb01cult\n\nto learn policies over more than 25 possible discrete locations.\n\n5\n\n\f(a) 28x28 MNIST\n\nModel\nFC, 2 layers (256 hiddens each)\nConvolutional, 2 layers\nRAM, 2 glimpses, 8 \u00d7 8, 1 scale\nRAM, 3 glimpses, 8 \u00d7 8, 1 scale\nRAM, 4 glimpses, 8 \u00d7 8, 1 scale\nRAM, 5 glimpses, 8 \u00d7 8, 1 scale\nRAM, 6 glimpses, 8 \u00d7 8, 1 scale\nRAM, 7 glimpses, 8 \u00d7 8, 1 scale\n\nError\n1.69%\n1.21%\n3.79%\n1.51%\n1.54%\n1.34%\n1.12%\n1.07%\n\n(b) 60x60 Translated MNIST\n\nModel\nFC, 2 layers (64 hiddens each)\nFC, 2 layers (256 hiddens each)\nConvolutional, 2 layers\nRAM, 4 glimpses, 12 \u00d7 12, 3 scales\nRAM, 6 glimpses, 12 \u00d7 12, 3 scales\nRAM, 8 glimpses, 12 \u00d7 12, 3 scales\n\nError\n6.42%\n2.63%\n1.62%\n1.54%\n1.22%\n1.2%\n\nTable 1: Classi\ufb01cation results on the MNIST and Translated MNIST datasets. FC denotes a fully-\nconnected network with two layers of recti\ufb01er units. The convolutional network had one layer of 8\n10 \u00d7 10 \ufb01lters with stride 5, followed by a fully connected layer with 256 units with recti\ufb01ers after\neach layer. Instances of the attention model are labeled with the number of glimpses, the number of\nscales in the retina, and the size of the retina.\n\n(a) Translated MNIST inputs.\n\n(b) Cluttered Translated MNIST inputs.\n\nFigure 2: Examples of test cases for the Translated and Cluttered Translated MNIST tasks.\n\nCore network: For the classi\ufb01cation experiments that follow the core fh was a network of recti\ufb01er\nunits de\ufb01ned as ht = fh(ht\u22121) = Rect(Linear(ht\u22121) + Linear(gt)). The experiment done on a\ndynamic environment used a core of LSTM units [10].\n\nImage Classi\ufb01cation\n\n4.1\nThe attention network used in the following classi\ufb01cation experiments made a classi\ufb01cation decision\nonly at the last timestep t = N. The action network fa was simply a linear softmax classi\ufb01er de\ufb01ned\nas fa(h) = exp (Linear(h)) /Z, where Z is a normalizing constant. The RNN state vector h had\ndimensionality 256. All methods were trained using stochastic gradient descent with minibatches\nof size 20 and momentum of 0.9. We annealed the learning rate linearly from its initial value to 0\nover the course of training. Hyperparameters such as the initial learning rate and the variance of the\nlocation policy were selected using random search [3]. The reward at the last time step was 1 if the\nagent classi\ufb01ed correctly and 0 otherwise. The rewards for all other timesteps were 0.\nCentered Digits: We \ufb01rst tested the ability of our training method to learn successful glimpse\npolicies by using it to train RAM models with up to 7 glimpses on the MNIST digits dataset. The\n\u201cretina\u201d for this experiment was simply an 8\u00d7 8 patch, which is only big enough to capture a part of\na digit, hence the experiment also tested the ability of RAM to combine information from multiple\nglimpses. We also trained standard feedforward and convolutional neural networks with two hidden\nlayers as a baselines. The error rates achieved by the different models on the test set are shown in\nTable 1a. We see that the performance of RAM generally improves with more glimpses, and that\nit eventually outperforms a the baseline models trained on the full 28 \u00d7 28 centered digits. This\ndemonstrates the model can successfully learn to combine information from multiple glimpses.\nNon-Centered Digits: The second problem we considered was classifying non-centered digits. We\ncreated a new task called Translated MNIST, for which data was generated by placing an MNIST\ndigit in a random location of a larger blank patch. Training cases were generated on the \ufb02y so the\neffective training set size was 50000 (the size of the MNIST training set) multiplied by the possible\nnumber of locations. Figure 2a contains a random sample of test cases for the 60 by 60 Translated\nMNIST task. Table 1b shows the results for several different models trained on the Translated\nMNIST task with 60 by 60 patches. In addition to RAM and two fully-connected networks we\nalso trained a network with one convolutional layer of 16 10 \u00d7 10 \ufb01lters with stride 5 followed\nby a recti\ufb01er nonlinearity and then a fully-connected layer of 256 recti\ufb01er units. The convolutional\nnetwork, the RAM networks, and the smaller fully connected model all had roughly the same number\nof parameters. Since the convolutional network has some degree of translation invariance built in, it\n\n6\n\n\f(a) 60x60 Cluttered Translated MNIST\nError\n28.58%\n11.96%\n8.09%\n4.96%\n4.08%\n4.04%\n14.4%\n\nModel\nFC, 2 layers (64 hiddens each)\nFC, 2 layers (256 hiddens each)\nConvolutional, 2 layers\nRAM, 4 glimpses, 12 \u00d7 12, 3 scales\nRAM, 6 glimpses, 12 \u00d7 12, 3 scales\nRAM, 8 glimpses, 12 \u00d7 12, 3 scales\nRAM, 8 random glimpses\n\n(b) 100x100 Cluttered Translated MNIST\nError\nModel\n14.35%\nConvolutional, 2 layers\nRAM, 4 glimpses, 12 \u00d7 12, 4 scales\n9.41%\nRAM, 6 glimpses, 12 \u00d7 12, 4 scales\n8.31%\nRAM, 8 glimpses, 12 \u00d7 12, 4 scales\n8.11%\nRAM, 8 random glimpses\n28.4%\n\nTable 2: Classi\ufb01cation on the Cluttered Translated MNIST dataset. FC denotes a fully-connected\nnetwork with two layers of recti\ufb01er units. The convolutional network had one layer of 8 10 \u00d7 10\n\ufb01lters with stride 5, followed by a fully connected layer with 256 units in the 60 \u00d7 60 case and\n86 units in the 100 \u00d7 100 case with recti\ufb01ers after each layer. Instances of the attention model are\nlabeled with the number of glimpses, the size of the retina, and the number of scales in the retina.\nAll models except for the big fully connected network had roughly the same number of parameters.\n\nFigure 3: Examples of the learned policy on 60 \u00d7 60 cluttered-translated MNIST task. Column 1:\nThe input image with glimpse path overlaid in green. Columns 2-7: The six glimpses the network\nchooses. The center of each image shows the full resolution glimpse, the outer low resolution areas\nare obtained by upscaling the low resolution glimpses back to full image size. The glimpse paths\nclearly show that the learned policy avoids computation in empty or noisy parts of the input space\nand directly explores the area around the object of interest.\n\nattains a signi\ufb01cantly lower error rate of 1.62% than the fully connected networks. However, RAM\nwith 4 glimpses gets slightly better performance than the convolutional network and outperforms\nit further for 6 and 8 glimpses, reaching 1.2% error. This is possible because the attention model\ncan focus its retina on the digit and hence learn a translation invariant policy. This experiment also\nshows that the attention model is able to successfully search for an object in a big image when the\nobject is not centered.\nCluttered Non-Centered Digits: One of the most challenging aspects of classifying real-world\nimages is the presence of a wide range clutter. Systems that operate on the entire image at full\nresolution are particularly susceptible to clutter and must learn to be invariant to it. One possible\nadvantage of an attention mechanism is that it may make it easier to learn in the presence of clutter\nby focusing on the relevant part of the image and ignoring the irrelevant part. We test this hypothesis\nwith several experiments on a new task we call Cluttered Translated MNIST. Data for this task was\ngenerated by \ufb01rst placing an MNIST digit in a random location of a larger blank image and then\nadding random 8 by 8 subpatches from other random MNIST digits to random locations of the\nimage. The goal is to classify the complete digit present in the image. Figure 2b shows a random\nsample of test cases for the 60 by 60 Cluttered Translated MNIST task.\nTable 2a shows the classi\ufb01cation results for the models we trained on 60 by 60 Cluttered Translated\nMNIST with 4 pieces of clutter. The presence of clutter makes the task much more dif\ufb01cult but the\nperformance of the attention model is affected less than the performance of the other models. RAM\nwith 4 glimpses reaches 4.96% error, which outperforms fully-connected models by a wide margin\nand the convolutional neural network by over 3%, and RAM trained with 6 and 8 glimpses achieves\neven lower error. Since RAM achieves larger relative error improvements over a convolutional\nnetwork in the presence of clutter these results suggest the attention-based models may be better at\ndealing with clutter than convolutional networks because they can simply ignore it by not looking at\nit. Two samples of learned policy is shown in Figure 3 and more are included in the supplementary\nmaterials. The \ufb01rst column shows the original data point with the glimpse path overlaid. The\n\n7\n\n\flocation of the \ufb01rst glimpse is marked with a \ufb01lled circle and the location of the \ufb01nal glimpse is\nmarked with an empty circle. The intermediate points on the path are traced with solid straight lines.\nEach consecutive image to the right shows a representation of the glimpse that the network sees. It\ncan be seen that the learned policy can reliably \ufb01nd and explore around the object of interest while\navoiding clutter at the same time. Finally, Table 2a also includes results for an 8-glimpse RAM\nmodel that selects glimpse locations uniformly at random. RAM models that learn the glimpse\npolicy achieve much lower error rates even with half as many glimpses.\nTo further test this hypothesis we also performed experiments on 100 by 100 Cluttered Translated\nMNIST with 8 pieces of clutter. The test errors achieved by the models we compared are shown\nin Table 2b. The results show similar improvements of RAM over a convolutional network. It has\nto be noted that the overall capacity and the amount of computation of our model does not change\nfrom 60 \u00d7 60 images to 100 \u00d7 100, whereas the hidden layer of the convolutional network that is\nconnected to the linear layer grows linearly with the number of pixels in the input.\n4.2 Dynamic Environments\nOne appealing property of the recurrent attention model is that it can be applied to videos or inter-\nactive problems with a visual input just as easily as to static image tasks. We test the ability of our\napproach to learn a control policy in a dynamic visual environment while perceiving the environment\nthrough a bandwidth-limited retina by training it to play a simple game. The game is played on a 24\nby 24 screen of binary pixels and involves two objects: a single pixel that represents a ball falling\nfrom the top of the screen while bouncing off the sides of the screen and a two-pixel paddle posi-\ntioned at the bottom of the screen which the agent controls with the aim of catching the ball. When\nthe falling pixel reaches the bottom of the screen the agent either gets a reward of 1 if the paddle\noverlaps with the ball and a reward of 0 otherwise. The game then restarts from the beginning.\nWe trained the recurrent attention model to play the game of \u201cCatch\u201d using only the \ufb01nal reward\nas input. The network had a 6 by 6 retina at three scales as its input, which means that the agent\nhad to capture the ball in the 6 by 6 highest resolution region in order to know its precise position.\nIn addition to the two location actions, the attention model had three game actions (left, right, and\ndo nothing) and the action network fa used a linear softmax to model a distribution over the game\nactions. We used a core network of 256 LSTM units.\nWe performed random search to \ufb01nd suitable hyper-parameters and trained each agent for 20 mil-\nlion frames. A video of the best agent, which catches the ball roughly 85% of the time, can\nbe downloaded from http://www.cs.toronto.edu/\u02dcvmnih/docs/attention.mov.\nThe video shows that the recurrent attention model learned to play the game by tracking the ball\nnear the bottom of the screen. Since the agent was not in any way told to track the ball and was\nonly rewarded for catching it, this result demonstrates the ability of the model to learn effective\ntask-speci\ufb01c attention policies.\n5 Discussion\nThis paper introduced a novel visual attention model that is formulated as a single recurrent neural\nnetwork which takes a glimpse window as its input and uses the internal state of the network to\nselect the next location to focus on as well as to generate control signals in a dynamic environment.\nAlthough the model is not differentiable, the proposed uni\ufb01ed architecture is trained end-to-end\nfrom pixel inputs to actions using a policy gradient method. The model has several appealing prop-\nerties. First, both the number of parameters and the amount of computation RAM performs can\nbe controlled independently of the size of the input images. Second, the model is able to ignore\nclutter present in an image by centering its retina on the relevant regions. Our experiments show that\nRAM signi\ufb01cantly outperforms a convolutional architecture with a comparable number of parame-\nters on a cluttered object classi\ufb01cation task. Additionally, the \ufb02exibility of our approach allows for\na number of interesting extensions. For example, the network can be augmented with another action\nthat allows it terminate at any time point and make a \ufb01nal classi\ufb01cation decision. Our preliminary\nexperiments show that this allows the network to learn to stop taking glimpses once it has enough in-\nformation to make a con\ufb01dent classi\ufb01cation. The network can also be allowed to control the scale at\nwhich the retina samples the image allowing it to \ufb01t objects of different size in the \ufb01xed size retina.\nIn both cases, the extra actions can be simply added to the action network fa and trained using the\npolicy gradient procedure we have described. Given the encouraging results achieved by RAM, ap-\nplying the model to large scale object recognition and video classi\ufb01cation is a natural direction for\nfuture work.\n\n8\n\n\fReferences\n[1] Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari. What is an object? In CVPR, 2010.\n[2] Bogdan Alexe, Nicolas Heess, Yee Whye Teh, and Vittorio Ferrari. Searching for objects driven by\n\ncontext. In NIPS, 2012.\n\n[3] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of\n\nMachine Learning Research, 13:281\u2013305, 2012.\n\n[4] Nicholas J. Butko and Javier R. Movellan. Optimal scanning for faster object detection. In CVPR, 2009.\n[5] N.J. Butko and J.R. Movellan. I-pomdp: An infomax model of eye movement. In Proceedings of the 7th\n\nIEEE International Conference on Development and Learning, ICDL \u201908, pages 139 \u2013144, 2008.\n\n[6] Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep\n\narchitectures for image tracking. Neural Computation, 24(8):2151\u20132184, 2012.\n\n[7] Pedro F. Felzenszwalb, Ross B. Girshick, and David A. McAllester. Cascade object detection with de-\n\nformable part models. In CVPR, 2010.\n\n[8] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate\n\nobject detection and semantic segmentation. CoRR, abs/1311.2524, 2013.\n\n[9] Mary Hayhoe and Dana Ballard. Eye movements in natural behavior. Trends in Cognitive Sciences,\n\n9(4):188 \u2013 194, 2005.\n\n[10] Sepp Hochreiter and J\u00a8urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735\u2013\n\n1780, 1997.\n\n[11] L. Itti, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE\n\nTransactions on Pattern Analysis and Machine Intelligence, 20(11):1254\u20131259, 1998.\n\n[12] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton.\n\nImagenet classi\ufb01cation with deep convolutional\n\nneural networks. In Advances in Neural Information Processing Systems 25, pages 1106\u20131114, 2012.\n\n[13] Christoph H. Lampert, Matthew B. Blaschko, and Thomas Hofmann. Beyond sliding windows: Object\n\nlocalization by ef\ufb01cient subwindow search. In CVPR, 2008.\n\n[14] Hugo Larochelle and Geoffrey E. Hinton. Learning to combine foveal glimpses with a third-order boltz-\n\nmann machine. In NIPS, 2010.\n\n[15] Stefan Mathe and Cristian Sminchisescu. Action from still image dataset and inverse optimal control to\n\nlearn task speci\ufb01c visual scanpaths. In NIPS, 2013.\n\n[16] Lucas Paletta, Gerald Fritz, and Christin Seifert. Q-learning of sequential attention for visual object\n\nrecognition from informative local descriptors. In CVPR, 2005.\n[17] M. Ranzato. On Learning Where To Look. ArXiv e-prints, 2014.\n[18] Ronald A. Rensink. The dynamic representation of scenes. Visual Cognition, 7(1-3):17\u201342, 2000.\n[19] Pierre Sermanet, David Eigen, Xiang Zhang, Micha\u00a8el Mathieu, Rob Fergus, and Yann LeCun. Overfeat:\nIntegrated recognition, localization and detection using convolutional networks. CoRR, abs/1312.6229,\n2013.\n\n[20] Kenneth O. Stanley and Risto Miikkulainen. Evolving a roving eye for go. In GECCO, 2004.\n[21] Richard S. Sutton, David Mcallester, Satinder Singh, and Yishay Mansour. Policy gradient methods for\n\nreinforcement learning with function approximation. In NIPS, pages 1057\u20131063. MIT Press, 2000.\n\n[22] Antonio Torralba, Aude Oliva, Monica S Castelhano, and John M Henderson. Contextual guidance of eye\nmovements and attention in real-world scenes: the role of global features in object search. Psychol Rev,\npages 766\u2013786, 2006.\n\n[23] K E A van de Sande, J.R.R. Uijlings, T Gevers, and A.W.M. Smeulders. Segmentation as Selective Search\n\nfor Object Recognition. In ICCV, 2011.\n\n[24] Paul A. Viola and Michael J. Jones. Rapid object detection using a boosted cascade of simple features. In\n\nCVPR, 2001.\n\n[25] Daan Wierstra, Alexander Foerster, Jan Peters, and Juergen Schmidhuber. Solving deep memory pomdps\n\nwith recurrent policy gradients. In ICANN. 2007.\n\n[26] R.J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.\n\nMachine Learning, 8(3):229\u2013256, 1992.\n\n9\n\n\f", "award": [], "sourceid": 1149, "authors": [{"given_name": "Volodymyr", "family_name": "Mnih", "institution": "Google DeepMind"}, {"given_name": "Nicolas", "family_name": "Heess", "institution": "Google DeepMind"}, {"given_name": "Alex", "family_name": "Graves", "institution": "Google DeepMind"}, {"given_name": "koray", "family_name": "kavukcuoglu", "institution": "Google DeepMind"}]}