{"title": "Strategic Attentive Writer for Learning Macro-Actions", "book": "Advances in Neural Information Processing Systems", "page_first": 3486, "page_last": 3494, "abstract": "We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner purely by interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub-sequences by learning for how long the plan can be committed to -- i.e. followed without replaning. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro-actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.", "full_text": "Strategic Attentive Writer for Learning\n\nMacro-Actions\n\nAlexander (Sasha) Vezhnevets, Volodymyr Mnih, John Agapiou,\nSimon Osindero, Alex Graves, Oriol Vinyals, Koray Kavukcuoglu\n\nGoogle DeepMind\n\n{vezhnick,vmnih,jagapiou,osindero,gravesa,vinyals,korayk}@google.com\n\nAbstract\n\nWe present a novel deep recurrent neural network architecture that learns to build\nimplicit plans in an end-to-end manner purely by interacting with an environment\nin reinforcement learning setting. The network builds an internal plan, which is\ncontinuously updated upon observation of the next input from the environment.\nIt can also partition this internal representation into contiguous sub-sequences\nby learning for how long the plan can be committed to \u2013 i.e. followed without\nreplaning. Combining these properties, the proposed model, dubbed STRategic At-\ntentive Writer (STRAW) can learn high-level, temporally abstracted macro-actions\nof varying lengths that are solely learnt from data without any prior information.\nThese macro-actions enable both structured exploration and economic computation.\nWe experimentally demonstrate that STRAW delivers strong improvements on\nseveral ATARI games by employing temporally extended planning strategies (e.g.\nMs. Pacman and Frostbite). It is at the same time a general algorithm that can\nbe applied on any sequence data. To that end, we also show that when trained\non text prediction task, STRAW naturally predicts frequent n-grams (instead of\nmacro-actions), demonstrating the generality of the approach.\n\nIntroduction\n\n1\nUsing reinforcement learning to train neural network controllers has recently led to rapid progress\non a number of challenging control tasks [15, 17, 26]. Much of the success of these methods has\nbeen attributed to the ability of neural networks to learn useful abstractions or representations of\nthe stream of observations, allowing the agents to generalize between similar states. Notably, these\nagents do not exploit another type of structure \u2013 the one present in the space of controls or policies.\nIndeed, not all sequences of low-level controls lead to interesting high-level behaviour and an agent\nthat can automatically discover useful macro-actions should be capable of more ef\ufb01cient exploration\nand learning. The discovery of such temporal abstractions has been a long-standing problem in\nboth reinforcement learning and sequence prediction in general, yet no truly scalable and successful\narchitectures exist.\nWe propose a new deep recurrent neural network architecture, dubbed STRategic Attentive Writer\n(STRAW), that is capable of learning macro-actions in a reinforcement learning setting. Unlike the\nvast majority of reinforcement learning approaches [15, 17, 26], which output a single action after\neach observation, STRAW maintains a multi-step action plan. STRAW periodically updates the plan\nbased on observations and commits to the plan between the replanning decision points. The replanning\ndecisions as well as the commonly occurring sequences of actions, i.e. macro-actions, are learned\nfrom rewards. To encourage exploration with macro-actions we introduce a noisy communication\nchannel between a feature extractor (e.g. convolutional neural network) and the planning modules,\ntaking inspiration from recent developments in variational auto-encoders [11, 13, 24]. Injecting noise\nat this level of the network generates randomness in plans updates that cover multiple time steps and\nthereby creates the desired effect.\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fOur proposed architecture is a step towards more natural decision making, wherein one observation\ncan generate a whole sequence of outputs if it is informative enough. This provides several important\nbene\ufb01ts. First and foremost, it facilitates structured exploration in reinforcement learning \u2013 as the\nnetwork learns meaningful action patterns it can use them to make longer exploratory steps in the\nstate space [4]. Second, since the model does not need to process observations while it is committed\nto its action plan, it learns to allocate computation to key moments thereby freeing up resources\nwhen the plan is being followed. Additionally, the acquisition of macro-actions can aid transfer and\ngeneralization to other related problems in the same domain (assuming that other problems from the\ndomain share similar structure in terms of action-effects).\nWe evaluate STRAW on a subset of Atari games that require longer term planning and show that\nit leads to substantial improvements in scores. We also demonstrate the generality of the STRAW\narchitecture by training it on a text prediction task and show that it learns to use frequent n-grams as\nthe macro-actions on this task.\nThe following section reviews the related work. Section 3 de\ufb01nes the STRAW model formally.\nSection 4 describes the training procedure for both supervised and reinforcement learning cases.\nSection 5 presents the experimental evaluation of STRAW on 8 ATARI games, 2D maze navigation\nand next character prediction tasks. Section 6 concludes.\n\n2 Related Work\nLearning temporally extended actions and temporal abstraction in general are long standing problems\nin reinforcement learning [5, 6, 7, 8, 12, 20, 21, 22, 23, 25, 27, 28, 30]. The options framework [21,\n28] provides a general formulation. An option is a sub-policy with a termination condition, which\ntakes in environment observations and outputs actions until the termination condition is met. An agent\npicks an option using its policy-over-options and subsequently follows it until termination, at which\npoint the policy-over-options is queried again and the process continues. Notice, that macro-action is\na particular, simpler instance of options, where the action sequence (or a distribution over them) is\ndecided at the time the macro-action is initiated. Options are typically learned using subgoals and\n\u2019pseudo-rewards\u2019 that are provided explicitly [7, 8, 28]. For a simple, tabular case [30], each state\ncan be used as a subgoal. Given the options, a policy-over-options can be learned using standard\ntechniques by treating options as actions. Recently [14, 29] have demonstrated that combining\ndeep learning with pre-de\ufb01ned subgoals delivers promising results in challenging environments like\nMinecraft and Atari, however, subgoal discovery remains an unsolved problem. Another recent\nwork by [1] shows a theoretical possibility of learning options jointly with a policy-over-options by\nextending the policy gradient theorem to options, but the approach was only tested on a toy problem.\nIn contrast, STRAW learns macro-actions and a policy over them in an end-to-end fashion from only\nthe environment\u2019s reward signal and without resorting to explicit pseudo-rewards or hand-crafted\nsubgoals. The macro-actions are represented implicitly inside the model, arising naturally from the\ninterplay between action and commitment plans within the network. Our experiments demonstrate\nthat the model scales to a variety of tasks from next character prediction in text to ATARI games.\n3 The model\nSTRAW is a deep recurrent neural network with two modules. The \ufb01rst module translates environment\nobservations into an action-plan \u2013 a state variable which represents an explicit stochastic plan of\nfuture actions. STRAW generates macro-actions by committing to the action-plan and following\nit without updating for a number of steps. The second module maintains commitment-plan \u2013 a\nstate variable that determines at which step the network terminates a macro-action and updates the\naction-plan. The action-plan is a matrix where one dimension corresponds to time and the other to\nthe set of possible discrete actions. The elements of this matrix are proportional to the probability\nof taking the corresponding action at the corresponding time step. Similarly, the commitment plan\nrepresents the probabilities of terminating a macro-action at the particular step. For updating both\nplans we use attentive writing technique [11], which allows the network to focus on parts of a plan\nwhere the current observation is informative of desired outputs. This section formally de\ufb01nes the\nmodel, we describe the way it is trained later in section 4.\nThe state of the network at time t is comprised of matrices At \u2208 RA\u00d7T and ct \u2208 R1\u00d7T . Matrix\nAt is the action-plan. Each element At\na,\u03c4 is proportional to the probability of outputting a at time\nt + \u03c4. Here A is a total number of possible actions and T is a maximum time horizon of the plan.\nTo generate an action at time t, the \ufb01rst column of At (i.e. At\u20220) is transformed into a distribution\n\n2\n\n\f1\n\nt1\n\nFigure 1: Schematic illustration of STRAW playing a maze navigation game. The input frames\nindicate maze geometry (black = corridor, blue = wall), red dot corresponds to the position of the\nagent and green to the goal, which it tries to reach. A frame is \ufb01rst passed through a convolutional\nnetwork, acting as a feature extractor and then into STRAW. The top two rows depict the plans A\nand c. Given a gate gt, STRAW either updates the plans (steps 1 and 4) or commits to them.\nover possible outputs by a SoftMax function. This distribution is then sampled to generate the action\nat. Thereby the content of At corresponds to the plan of future actions as conceived at time t. The\nsingle row matrix ct represents the commitment-plan of the network. Let gt be a binary random\nvariable distributed as follows: gt \u223c ct\u22121\n. If gt = 1 then at this step the plans will be updated,\notherwise gt = 0 means they will be committed to. Macro-actions are de\ufb01ned as a sequence of\noutputs {at}t2\u22121\nproduced by the network between steps where gt is \u2018on\u2019: i.e gt1 = gt2 = 1 and\ngt(cid:48) = 0,\u2200t1 < t(cid:48) < t2. During commitment the plans are rolled over to the next step using the\nmatrix time-shift operator \u03c1, which shifts a matrix by removing the \ufb01rst column and appending a\ncolumn \ufb01lled with zeros to its rear. Applying \u03c1 to At or ct re\ufb02ects the advancement of time. Figure 1\nillustrates the work\ufb02ow. Notice that during commitment (step 2 and 3) the network doesn\u2019t compute\nthe forward pass, thereby saving computation.\nAttentive planning. An important assumption that underpins the usage of macro-actions is that\none observation reveals enough information to generate a sequence of actions. The complexity of the\nsequence and its length can vary dramatically, even within one environment. Therefore the network\nhas to focus on the part of the plan where the current observation is informative of desired actions.\nTo achieve this, we apply differentiable attentive reading and writing operations [11], where attention\nis de\ufb01ned over the temporal dimension. This technique was originally proposed for image generation,\nhere instead it is used to update the plans At and ct. In the image domain, the attention operates over\nthe spatial extent of an image, reading and writing pixel values. Here it operates over the temporal\nextent of a plan, and is used to read and write action probabilities. The differentiability of the attention\nmodel [11] makes it possible to train with standard backpropagation.\nAn array of Gaussian \ufb01lters is applied to the plan, yielding a \u2018patch\u2019 of smoothly varying location\nand zoom. Let A be the total number of possible actions and K be a parameter that determines the\ntemporal resolution of the patch. A grid of A \u00d7 K one-dimensional Gaussian \ufb01lters is positioned\non the plan by specifying the coordinates of the grid center and the stride distance between adjacent\n\ufb01lters. The stride controls the \u2018zoom\u2019 of the patch; that is, the larger the stride, the larger an area of\nthe original plan will be visible in the attention patch, but the lower the effective resolution of the\npatch will be. The \ufb01ltering is performed along the temporal dimension only. Let \u03c8 be a vector of\nattention parameters, i.e.: grid position, stride, and standard deviation of Gaussian \ufb01lters. We de\ufb01ne\nthe attention operations as follows:\n\nD = write(p, \u03c8A\n\nt ); \u03b2t = read(At, \u03c8A\nt )\n\n(1)\n\nThe write operation takes in a patch p \u2208 RA\u00d7K and attention parameters \u03c8A\nt . It produces a matrix D\nof the same size as At, which contains the patch p scaled and positioned according to \u03c8A\nt with the\nrest set to 0. Analogously the read operation takes the full plan At together with attention parameters\n\n3\n\n**00=11Input frameSTRAWcommitment-planaction-plan...action space:ConvNettimereplancommitcommitreplan\fFigure 2: Schematic illustration of an action-plan update. Given zt, the network produces an action-\nt . The write operation creates an update to the action-plan by\npatch and attention parameters \u03c8A\nscaling and shifting the action-patch according to \u03c8A\nt . The update is then added to \u03c1(At).\nt and outputs a read patch \u03b2 \u2208 RA\u00d7K, which is extracted from At according to \u03c8A\n\u03c8A\nreaders to [11] for details.\nAction-plan update. Let zt be a feature representation (e.g. the output of a deep convolutional\nnetwork) of an observation xt. Given zt, gt and the previous state At\u22121 STRAW computes an update\nto the action-plan using the Algorithm 1. Here f \u03c8 and f A are linear functions and h is two layer\nperceptron. Figure 2 gives an illustration of an update to At.\n\nt . We direct\n\nAlgorithm 1 Action-plan update\n\nInput: zt, gt, At\u22121\nOutput: At, at\nif gt = 1 then\n\nt = f \u03c8(zt)\n\nCompute attention parameters \u03c8A\nAttentively read the current state of the action-plan \u03b2t = read(At\u22121, \u03c8A\nt );\nCompute intermediate representation \u03bet = h([\u03b2t, zt])\nUpdate At = \u03c1(At\u22121) + gt \u00b7 write(f A(\u03bet), \u03c8A\nt )\nUpdate At = \u03c1(At\u22121) //just advance time\n\nelse // gt = 0\n\nend if\nSample an action at \u223c SoftMax(At\n0)\n\nCommitment-plan update. Now we introduce a module that partitions the action-plan into macro-\nactions by de\ufb01ning the temporal extent to which the current action-plan At can be followed without\nre-planning. The commitment-plan ct is updated at the same time as the action-plan, i.e. when gt = 1\nor otherwise it is rolled over to the next time step by \u03c1 operator. Unlike the planning module, where\nAt is updated additively, ct is overwritten completely using the following equations:\n\ngt \u223c ct\u22121\n1\nt = f c([\u03c8A\n\nelse \u03c8c\n\nif gt = 0 then ct = \u03c1(ct\u22121)\n\nt , \u03bet]) ct = Sigmoid(b + write(e, \u03c8c\n\nt ));\n\n(2)\n\nHere the same attentive writing operation is used, but with only one Gaussian \ufb01lter for attention over\nc. The patch e is therefore just a scalar, which we \ufb01x to a high value (40 in our experiments). This\nhigh value of e is chosen so that the attention parameters \u03c8c\nt de\ufb01ne a time step when re-planning is\nguaranteed to happen. Vector b is of the same size as dt \ufb01lled with a shared, learnable bias b, which\nde\ufb01nes the probability of re-planning earlier than the step implied by \u03c8c\nt .\nNotice that gt is used as a multiplicative gate in algorithm 1. This allows for retroactive credit\nassignment during training, as gradients from write operation at time t + \u03c4 directly \ufb02ow into the\ncommitment module through state ct \u2013 we set \u2207ct\u22121\n1 \u2261 \u2207gt as proposed in [3]. Moreover, when\ngt = 0 only the computationally cheap operator \u03c1 is invoked. Thereby more commitment signi\ufb01cantly\nsaves computation.\n\n3.1 Structured exploration with macro-actions\nThe architecture de\ufb01ned above is capable of generating macro-actions. This section describes\nhow to use macro-actions for structured exploration. We introduce STRAW-explorer (STRAWe),\na version of STRAW with a noisy communication channel between the feature extractor (e.g. a\nconvolutional neural network) and STRAW planning modules. Let \u03b6t be the activations of the\nlast layer of the feature extractor. We use \u03b6t to regress the parameters of a Gaussian distribution\n\n4\n\n\fQ(zt|\u03b6t) = N (\u00b5(\u03b6t), I \u00b7 \u03c3(\u03b6t)) from which zt is sampled. Here \u00b5 is a vector and \u03c3 is a scalar.\nInjecting noise at this level of the network generates randomness on the level of plan updates that\ncover multiple time steps. This effect is reinforced by commitment, which forces STRAWe to execute\nthe plan and experience the outcome instead of rolling the update back on the next step. In section 5\nwe demonstrate how this signi\ufb01cantly improves score on games like Frostbite and Ms. Pacman.\n4 Learning\nThe training loss of the model is de\ufb01ned as follows:\n\n(cid:17)\n\nT(cid:88)\n\n(cid:16)\n\nt=1\n\n,\n\n1\n\nL =\n\nLout(At) + gt \u00b7 \u03b1KL(Q(zt|\u03b6t)|P (zt)) + \u03bbct\n\nit restarts. The goal of the agent is to maximize the discounted return Rt =(cid:80)\u221e\n\n(3)\nwhere Lout is a domain speci\ufb01c differentiable loss function de\ufb01ned over the network\u2019s output. For\nsupervised problems, like next character prediction in text, Lout can be de\ufb01ned as negative log\nlikelihood of the correct output. We discuss the reinforcement learning case later in this section. The\ntwo extra terms are regularisers. The \ufb01rst is a cost of communication through the noisy channel,\nwhich is de\ufb01ned as KL divergence between latent distributions Q(zt|\u03b6t) and some prior P (zt). Since\nthe latent distribution is a Gaussian (sec. 3.1), a natural choice for the prior is a Gaussian with a zero\nmean and standard deviation of one. The last term penalizes re-planning and encourages commitment.\nFor reinforcement learning we consider the standard setting where an agent is interacting with an\nenvironment in discrete time. At each step t, the agent observes the state of the environment xt and\nselects an action at from a \ufb01nite set of possible actions. The environment responds with a new state\nxt+1 and a scalar reward rt. The process continues until the terminal state is reached, after which\nk=0 \u03b1krt+k+1. The\nagent\u2019s behaviour is de\ufb01ned by its policy \u03c0 \u2013 mapping from state space into action space. STRAW\nproduces a distribution over possible actions (a stochastic policy) by passing the \ufb01rst column of\nthe action-plan At through a SoftMax function: \u03c0(at|xt; \u03b8) = SoftMax(At\u20220). An action is then\nproduced by sampling the output distribution.\nWe use a recently proposed Asynchronous Advantage Actor-Critic (A3C) method [18], which directly\noptimizes the policy of an agent. A3C requires a value function estimator V (xt) for variance\nreduction. This estimator can be produced in a number of ways, for example by a separate neural\nnetwork. The most natural solution for our architecture is to create value-plan containing the estimates.\nTo keep the architecture simple and ef\ufb01cient, we simply add an auxiliary row to the action plan which\ncorresponds to the value function estimation. It participates in attentive reading and writing during\nthe update, thereby sharing the temporal attention with action-plan. The plan is then split into action\npart and the estimator before the SoftMax is applied and an action is sampled. The policy gradient\nupdate for Lout in L is de\ufb01ned as follows:\n\n\u2207Lout = \u2207\u03b8 log \u03c0(at|xt; \u03b8)(Rt \u2212 V (xt; \u03b8)) + \u03b2\u2207\u03b8H(\u03c0(at|xt; \u03b8))\n\n(4)\nHere H(\u03c0(at|xt; \u03b8)) is entropy of the policy, which stimulates the exploration of primitive actions.\nThe network contains two random variables \u2013 zt and gt, which we have to pass gradients through. For\nzt we employ the re-parametrization trick [13, 24]. For gt, we set \u2207ct\u22121\n1 \u2261 \u2207gt as proposed in [3].\n5 Experiments\nThe goal of our experiments was to demonstrate that STRAW learns meaningful and useful macro-\nactions. We use three domains of increasing complexity: supervised next character prediction in\ntext [10], 2D maze navigation and ATARI games [2].\n5.1 Experimental setup\nArchitecture. The read and write patches are A \u00d7 10 dimensional, and h is a 2 layer perceptron\nwith 64 hidden units. The time horizon T = 500. For STRAWe (sec. 3.1) the Gaussian distribution\nfor structured exploration is 128-dimensional. Ablative analysis for some of these choices is provided\nin section 5.5. Feature representation of the state space is particular for each domain. For 2D mazes\nand ATARI it is a convolutional neural net (CNN) and it is an LSTM [9] for text. We provide more\ndetails in the corresponding sections.\nBaselines. The experiments employ two baselines: a simple feed forward network (FF) and a\nrecurrent LSTM network, both on top of a representation learned by CNN. FF directly regresses\nthe action probabilities and value function estimate from feature representation. The LSTM [9]\n\n5\n\n\farchitecture is a widely used recurrent network and it was demonstrated to perform well on a suite\nof reinforcement learning problems [18]. It has 128 hidden units and its inputs are the feature\nrepresentation of an observation and the previous action of the agent. Action probabilities and the\nvalue function estimate are regressed from its hidden state.\n\nOptimization. We use the A3C method [18] for all reinforcement learning experiments. It was\nshown to achieve state-of-the-art results on several challenging benchmarks [18]. We cut the trajectory\nand run backpropagation through time [19] after 40 forward passes of a network or if a terminal signal\nis received. The optimization process runs 32 asynchronous threads using shared RMSProp. There\nare 4 hyper-parameters in STRAW and 2 in the LSTM and FF baselines. For each method, we ran 200\nexperiments, each using randomly sampled hyperparameters. Learning rate and entropy penalty were\nsampled from a LogUniform(10\u22124, 10\u22123) interval. Learning rate is linearly annealed from a sampled\nvalue to 0. To explore STRAW behaviour, we sample coding cost \u03b1 \u223c LogUniform(10\u22127, 10\u22124) and\nreplanning penalty \u03bb \u223c LogUniform(10\u22126, 10\u22122). For stability, we clip the advantage Rt\u2212V (xt; \u03b8)\n(eq. 4) to [\u22121, 1] for all methods and for STRAW(e) we do not propagate gradients from commitment\nmodule into planning module through \u03c8A\nt and \u03bet. We de\ufb01ne a training epoch as one million\nobservations.\n\nFigure 3: Detailed analysis. a) re-planning in random maze b) transfer to farther goals c) learned\nmacro-actions for next character prediction d) commitment on ATARI.\n\n5.2 Text\nSTRAW is a general sequence prediction architecture. To demonstrate that it is capable of learning\noutput patterns with complex structure we present a qualitative experiment on next character prediction\nusing Penn Treebank dataset [16]. Actions in this case correspond to emitting characters and macro-\nactions to their sequences. For this experiment we use an LSTM, which receives a one-hot-encoding\nof 50 characters as input. A STRAW module is connected on top. We omit the noisy Gaussian\nchannel, as this task is fully supervised and does not require exploration. The actions now correspond\nto emitting characters. The network is trained with stochastic gradient descent using supervised\nnegative log-likelihood loss (sec. 4). At each step we feed a character to the model which updates the\nLSTM representation, but only update the STRAW plans according to the commitment plan ct. For\na trained model we record macro-actions \u2013 the sequences of characters produced when STRAW is\ncommitted to the plan. If STRAW adequately learns the structure of the data, then its macro-actions\nshould correspond to common n-grams. Figure 3.c plots the 20 most frequent macro-action of length\n4 produced by STRAW. On x-axis is the rank of the frequency at which macro-action is used by\nSTRAW, on y-axis is it\u2019s actual frequency rank as a 4-gram. Notice how STRAW correctly learned to\npredict frequent 4-grams such as \u2019the\u2019, \u2019ing\u2019, \u2019and\u2019 and so on.\n\n2D mazes\n\n5.3\nTo investigate what macro-actions our model learns and whether they are useful for reinforcement\nlearning we conduct an experiment on a random 2D mazes domain. A maze is a grid-world with\ntwo types of cells \u2013 walls and corridors. One of the corridor cells is marked as a goal. The agent\nreceives a small negative reward of \u2212r every step, and double that if it tries to move through the wall.\nIt receives as small positive reward r when it steps on the goal and the episode terminates. Otherwise\nepisode terminates after 100 steps. Therefore to maximize return an agent has to reach the goal as\nfast as possible. In every training episode the position of walls, goal and the starting position of the\nagent are randomized. An agent fully observes the state of the maze. The remainder of this section\npresents two experiments in this domain. Here we use STRAWe version with structured exploration.\n\n6\n\nSTRAW rankN-gram rankTextMaze replanningTransfer in mazes_the_