{"title": "LIIR: Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 4403, "page_last": 4414, "abstract": "A great challenge in cooperative decentralized multi-agent reinforcement learning (MARL) is generating diversified behaviors for each individual agent when receiving only a team reward. Prior studies have paid much effort on reward shaping or designing a centralized critic that can discriminatively credit the agents.\nIn this paper, we propose to merge the two directions and learn each agent an intrinsic reward function which diversely stimulates the agents at each time step. Specifically, the intrinsic reward for a specific agent will be involved in computing a distinct proxy critic for the agent to direct the updating of its individual policy. Meanwhile, the parameterized intrinsic reward function will be updated towards maximizing the expected accumulated team reward from the environment so that the objective is consistent with the original MARL problem. The proposed method is referred to as learning individual intrinsic reward (LIIR) in MARL. We compare LIIR with a number of state-of-the-art MARL methods on battle games in StarCraft II. The results demonstrate the effectiveness of LIIR, and we show LIIR can assign each individual agent an insightful intrinsic reward per time step.", "full_text": "LIIR: Learning Individual Intrinsic Reward in\n\nMulti-Agent Reinforcement Learning\n\nYali Du\u2217\n\nUniversity College London\n\nLondon, UK\n\nLei Han\u2217\n\nTencent AI Lab\n\nShenzhen, Guangdong, China\n\nyali.du@ucl.ac.uk\n\nleihan.cs@gmail.com\n\nMeng Fang\n\nTencent Robotics X\n\nShenzhen, Guangdong, China\n\nTianhong Dai\n\nImperial College London\n\nLondon, UK\n\nmfang@tencent.com\n\ntianhong.dai15@imperial.ac.uk\n\nJi Liu\n\nKwai Inc.\n\nSeattle, U.S.A.\n\nji.liu.uwisc@gmail.com\n\nDacheng Tao\n\nUBTECH Sydney AI Centre,\n\nThe University of Sydney\n\nNSW, Australia\n\ndacheng.tao@sydney.edu.au\n\nAbstract\n\nA great challenge in cooperative decentralized multi-agent reinforcement learning\n(MARL) is generating diversi\ufb01ed behaviors for each individual agent when receiv-\ning only a team reward. Prior studies have paid many efforts on reward shaping\nor designing a centralized critic that can discriminatively credit the agents. In this\npaper, we propose to merge the two directions and learn each agent an intrinsic re-\nward function which diversely stimulates the agents at each time step. Speci\ufb01cally,\nthe intrinsic reward for a speci\ufb01c agent will be involved in computing a distinct\nproxy critic for the agent to direct the updating of its individual policy. Meanwhile,\nthe parameterized intrinsic reward function will be updated towards maximizing\nthe expected accumulated team reward from the environment so that the objective\nis consistent with the original MARL problem. The proposed method is referred to\nas learning individual intrinsic reward (LIIR) in MARL. We compare LIIR with\na number of state-of-the-art MARL methods on battle games in StarCraft II. The\nresults demonstrate the effectiveness of LIIR, and we show LIIR can assign each\nindividual agent an insightful intrinsic reward per time step.\n\n1\n\nIntroduction\n\nMany real-world problems, such as traf\ufb01c light control [1], coordination of autonomous vehicles\n[2], resources management [3] and multi-player video games [4, 5], can be naturally formulated into\ncooperative multi-agent systems, where the objective is to maximize the return in the perspective of a\nteam of agents. When the agents are manipulated with a centralized controller which could access\nthe joint or global state of all the agents, coordination among the agents is easier and the main effort\nof the controller is usually paid on \ufb01nding an effective communication scheme among the agents.\nExamples include a wide range of approaches on designing effective centralized MARL architectures\n[5, 6, 7, 8].\n\n\u2217Equal contribution. Correspondence to the \ufb01rst two authors.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fUnfortunately, when the agents are independently deployed and communications are disabled or\nprohibitive, each agent has to predict its own action conditioning on its partial observation trajectory.\nWithout a centralized controller, each agent is responsible to collaborate with others on its own\ndecision. This pushes much burden on the capability of each agent\u2019s policy. Worse still, in most of\nthe real-world MARL applications, the agents only receive a team reward, from which it is dif\ufb01cult to\ndeduce each agent\u2019s contribution to the team\u2019s success, making the learning and collaboration among\nagents nontrivial. There have been many efforts paid on distinguishing the agents\u2019 credit, known as\nthe credit assignment problem in MARL [9, 10]. A general way is reward shaping [5, 11, 12], which,\nhowever, requires abundant human labor to assign precise rewards to each individual agent. Under\nsome real-world tasks, such as reducing the latency in a traf\ufb01c network, there might even not exist any\nclear choice of the reward functions for an individual agent (vehicle in the example). Another branch\nof commonly adopted methods try to design a centralized critic that is capable to distinguish the\nstate-action values of the agents during training [9, 10], and then perform decentralized executions\nduring testing.\nOur approach builds a connection between reward shaping and critic learning. That is, we propose to\nlearn each agent a parameterized individual intrinsic reward function by maximizing a centralized\ncritic. The optimal intrinsic reward problem has been introduced in [13] for single agent learning\nscenarios and studied in some recent RL approaches [14, 15, 16]. Inspired by the concept, we propose\nto introduce the intrinsic reward design into multi-agent systems to distinguish the contributions of\nthe agents when the environment only returns a team reward. Speci\ufb01cally, we learn each agent a\nparameterized intrinsic reward function that outputs an intrinsic reward for that agent at each time\nstep to induce diversi\ufb01ed behaviors. With these intrinsic rewards, we de\ufb01ne each agent a distinct\nproxy expected discounted return that is a combination of the real team reward from the environment\nand the learned intrinsic reward. Using the actor-critic method, the individual policy of each agent\nis updated under the direction of the corresponding proxy critic. The parameters of the intrinsic\nreward functions are updated to maximize the standard accumulated discounted team return from the\nenvironment. Therefore, the objective of the entire procedure is consistent with that of the original\nMARL problem.\nInsightfully, from an optimization perspective, the proposed method can be categorized to the bilevel\noptimization, where the problem of solving individual proxy objectives is nested within the outer\noptimization task which maximizes the standard multi-agent return. The parameters of the policy\nand the intrinsic reward function are treated as the parameters of the inner and outer optimization\nproblems, respectively. We refer the proposed method to as learning individual intrinsic reward\n(LIIR) in MARL. Empirically, we show that LIIR outperforms a number of state-of-the-art MARL\napproaches on extensive settings in the battle game of StarCraft II. We also conduct insightful case\nstudies to visualize the learned intrinsic reward, and the results demonstrate that the learned intrinsic\nreward function can generate diverse reward signals for the agents and the agents can also act diversely\nin a collaborative way.\n\n2 Related Work\n\nWhen considering a centralized controller in MARL, the controller takes the joint or global observation\nof the agents as input and outputs multiple actions for the agents in one step. Many studies have been\nproposed on pursuing effective communication architecture among the agents within a centralized\ncontroller. For example, densely connected communication layers or modules have been embedded\nin a centralized controller that directly outputs multi-head predictions for the agents [6, 5]. Recurrent\nneural networks (RNN) have also been introduced to enable a sequence of agents to communicate\nthrough the recurrent module [7]. However, in many MARL applications, the agents have to\nbe separately deployed that each agent has to make its own decision conditioning on its partial\nobservation.\nDecentralized methods naturally deal with the above situation. The simplest approach is learning\nan individual policy or Q-function for each agent. This was \ufb01rst attempted with Q-learning [17],\nwhich was then extended with deep neural networks applied [18, 19]. Fully decentralized methods\nare limited under the case where only a team reward is given, since distinguishing the agents\u2019\ncontributions is dif\ufb01cult. To address the credit assignment problem in decentralized MARL, many\nexisting methods utilize the framework with a centralized critic and decentralized policy. That is, the\npolicies are deployed independently by taking individual observation as input, while the centralized\n\n2\n\n\fcritic focuses on quantifying the differences among the agents. For example, the counterfactual\nmulti-agent policy gradient [9] uses a counterfactual baseline to assign credits for the agents; the\nvalue decomposition network [20] decomposes the centralized value into a sum of individual agent\nvalues to discriminate their contributions; the QMIX [10] method adopts a similar idea that assumes\nthe centralized Q-value function is monotonically increasing with the individual Q-values. Most of\nthe existing methods focus on the architecture design of the critic, even strong assumptions on the\nvalue functions are unavoidable. Our method differs from these approaches that rather than working\non the value functions, we propose to learn each agent an intrinsic reward at each time step. The\nbene\ufb01ts are that no assumptions are attached on the value functions and the agents are allocated an\nexplicit immediate intrinsic reward at each time step to assign their credits.\nOur work is also related to the optimal intrinsic reward design problem in single agent setting\n[21, 22, 23, 16, 24]. Some prior works have used heuristic metrics to de\ufb01ne the intrinsic reward. For\nexample, in [22] the intrinsic reward is de\ufb01ned as the squared difference between two consecutive\nstates, and in [23] a metric named curiosity is used as the intrinsic reward. In [24] the learning\nof intrinsic reward is integrated with the update of the policy. A recent approach [16] proposes\nto parameterize the intrinsic reward function and alternatively updates the policy parameters and\nthe intrinsic reward parameters. In this paper, we extend the setting to multi-agent system and use\nindividual intrinsic reward function to distinguish the credits of the agents.\n\n3 Background\n\n3.1 Cooperative Multi-Agent Reinforcement Learning\n\nt}n\ni=1 with each si\n\nt}n\ni=1 with each ui\n\nWe consider a fully cooperative multi-agent system, in which the agents need to be independently\ndeployed without a central controller. The system can be described as a tuple as (cid:104)A, S, U, P, r, \u03b3, \u03c10(cid:105).\nLet A = {1, 2,\u00b7\u00b7\u00b7 , n} denote the set of n agents. Denote observation space of the agents as\nS = {S1, S2,\u00b7\u00b7\u00b7 , Sn} and the action space of the agents as U = {U1, U2,\u00b7\u00b7\u00b7 , Un} respectively. At\nt \u2208 Si being the partial observation from agent i. Accordingly,\ntime step t, let st = {si\nlet ut = {ui\nt \u2208 Ui indicating the action taken by the agent i. We overload notations\nand use st \u2208 S to refer to the true state of the environment. P (st+1|st, ut) : S \u00d7 U \u00d7 S \u2192 [0, 1]\nis the state transition function. r(st, ut) : S \u00d7 U \u2192 R indicates the team reward function from the\nenvironment. In order to differentiate the team reward from the environment and the intrinsic reward\nthat will be learned, we refer the team reward to as the extrinsic team reward rex(st, ut), following\nthe usage in [16]. \u03b3 \u2208 [0, 1) is a discount factor and \u03c10 : S \u2192 R is the distribution of the initial state\ns0. Let \u03c0i(ui\ni=1. Let\nJ ex(\u03c0) = Es0,u0,\u00b7\u00b7\u00b7 [Rex\nt+l denoting the expected discounted extrinsic reward,\nt) for i \u2208 A, and st+1 \u223c P (st+1|st, ut). De\ufb01ne the extrinsic value\nwhere s0 \u223c \u03c10(s0), ui\nfunction as V ex\ni=1 that achieve\nthe maximum expected extrinsic team reward J ex(\u03c0\u2217).\n\nt) : Si \u00d7 Ui \u2192 [0, 1] be a stochastic policy for agent i and denote \u03c0 = {\u03c0i}n\n\nt ]. We aim to \ufb01nd optimal policies \u03c0\u2217 = {\u03c0\u2217\n\n\u03c0 (st) = Eut,st+1,\u00b7\u00b7\u00b7 [Rex\n\ni }n\n\nt|si\n\nt =(cid:80)\u221e\n\nl=0 \u03b3lrex\n\n0 ] with Rex\nt|si\nt \u223c \u03c0i(ui\n\n3.2 Centralized Learning with Decentralized Execution\n\nCentralized learning with decentralized execution (CLDE) is a commonly used architecture to learn\na centralized critic to update the decentralized policies during training. In CLDE, actor-critic (AC)\nstyle methods [25, 26, 27, 28, 29] are often selected. In our case, AC algorithms use n independent\nparameterized policies \u03c0\u03b8i for i \u2208 A and update \u03b8i by maximizing the expected extrinsic reward\nJ ex(\u03b81, \u03b82,\u00b7\u00b7\u00b7 , \u03b8n) = Es,u [Rex] using the policy gradient\n\n\u2207\u03b8iJ ex(\u03b81, \u03b82,\u00b7\u00b7\u00b7 , \u03b8n) = Es,u [\u2207\u03b8i log \u03c0\u03b8i(ui|si)A\u03c0(s, u)] ,\n\n(1)\nwhere A\u03c0(s, u) is the centralized critic. There are several ways to estimate A\u03c0(s, u). For example,\nA\u03c0(s, u) = rex(s, u) + V ex(s(cid:48)) \u2212 V ex(s) is the standard advantage function [27, 28], where s(cid:48) is the\nsuccessive state of the agents. In [9], A\u03c0(s, u) is de\ufb01ned as an estimated state-action value function\nminus a counterfactual baseline.\n\n3.3 Parameterized Intrinsic Reward\n\nA recent study [16] has investigated learning a parameterized intrinsic reward function in single agent\nsetting. The idea is to explicitly de\ufb01ne the intrinsic reward function as rin\n\u03b7 (s, a) for a state-action pair\n\n3\n\n\f(s, a) of the agent, and it is summed up with the extrinsic reward rex(s, a) from the environment to\nserve as the return signal for updating the policy. The intrinsic reward parameter \u03b7 is updated towards\nmaximizing the expected extrinsic reward J ex. The intuition for updating \u03b7 is to \ufb01nd the effect that\nthe change on \u03b7 would in\ufb02uence the extrinsic value through the change in the policy parameters. This\ntechnique can be viewed as an instance of meta learning [30, 31, 32]; the intrinsic reward function\nserves as a meta-learner that learns to improve the agents objective. In our case, we extend the\nintrinsic reward learning method to deal with decentralized MARL problem and we use the intrinsic\nrewards to diversely stimulate the agents to learn from the environment.\n\n4 Method\n\nIn this section, we formally propose the LIIR method. We \ufb01rst provide a formal de\ufb01nition of the\nconsidered problem based on what have been introduced in Section 3, then we introduce a bilevel\noptimization algorithm for solving the proposed objective.\n\n4.1 The Objective\n\nBy de\ufb01ning an intrinsic reward function rin\n(si, ui) which is parameterized by \u03b7i and takes a state-\n\u03b7i\naction pair (si, ui) of an individual agent i as input, we propose to assign agent i a distinct proxy\nreward\n\nrproxy\ni,t = rex\n\n(2)\nat time step t. In (2), we have omitted the arguments of the reward functions for simplicity, and \u03bb is a\nhyper-parameter that balances the extrinsic team reward and the distinct intrinsic reward. Note that in\nthe standard MARL problem with a team reward, there does not exist any distinct reward for each\nagent. Now, after creating each agent a proxy reward rproxy\nat time step t, we accordingly de\ufb01ne a\ndiscounted proxy reward for each agent i as\n\nt + \u03bbrin\ni,t,\n\ni,t\n\nRproxy\n\ni,t =\n\n\u03b3l(rex\n\nt+l + \u03bbrin\n\ni,t+l),\n\nand the proxy value function for agent i as\n\nl=0\n\nV proxy\ni\n\n(si,t) = Eui,t,si,t+1,\u00b7\u00b7\u00b7[Rproxy\n\ni,t\n\n].\n\n(3)\n\n(4)\n\n\u221e(cid:88)\n\nmax\n\u03b7,\u03b8\ns.t.\n\nDifferent from the extrinsic (standard) value V ex, these proxy value functions V proxy\n\u2019s do not have\nany physical meanings and they will be only used for updating the individual policy parameters \u03b8i\u2019s.\nNow, the considered overall objective is de\ufb01ned as\n\ni\n\nJ ex(\u03b7),\n\n(5)\n\n:= Esi,0,ui,0,\u00b7\u00b7\u00b7(cid:2)Rproxy\n\nJ proxy\ni\n\n(cid:3) depending on \u03b8i and \u03b7, \u03b7 indicates the intrinsic reward parame-\n\n\u2200i \u2208 [1, 2,\u00b7\u00b7\u00b7 , n]\n\n(\u03b8, \u03b7),\n\n\u03b8\n\n\u03b8i = arg max\n\ni\n\ni,0\n\nwhere J proxy\nter set {\u03b71, \u03b72,\u00b7\u00b7\u00b7 , \u03b7n} and \u03b8 indicates the policy parameter set {\u03b81, \u03b82,\u00b7\u00b7\u00b7 , \u03b8n}.\nIn problem (5), the goal is to maximize J ex through optimizing \u03b7, while the policy parameter \u03b8i is\noptimized by maximizing the proxy expected discounted return J proxy\nfor agent i. The advantage is\nthat by learning a distinct intrinsic reward for each agent per time step, the agents will be diversely\nstimulated and this will accumulatively in\ufb02uence the policy learning via the policy gradient. Moreover,\nfrom an optimization perspective, problem (5) can be viewed as a bilevel optimization problem,\nsince the problem of maximizing the individual proxy expected returns is nested within the outer\noptimization task, which is maximizing the extrinsic expected return. In the next subsection, we will\ndiscuss how J ex is connected with the intrinsic reward parameter \u03b7.\n\ni\n\n4.2 Algorithm\n\nAs a bilevel optimization problem, at each iteration, the policy parameters are updated with respect to\nthe inner proxy tasks, while the intrinsic reward parameters are updated to maximize the extrinsic\nexpected return.\n\n4\n\n\fFigure 1: Architecture of the LIIR method. The architecture contains four parameter components:\n\u03b8i\u2019s for policies, \u03b7i\u2019s for intrinsic reward, and \u03d5i\u2019s and \u03c6\u2019s for extrinsic and proxy values respectively.\nSpeci\ufb01cally, the policy parameter of each agent is updated by the policy gradient with its proxy critic.\nGiven a trajectory generated by the policy \u03c0\u03b8i, \u03b8i can be updated by applying the policy gradient\nde\ufb01ned in (1):\n\n\u2207\u03b8i log \u03c0\u03b8i(ui|si)Aproxy\n\ni\n\ni\n\ni\n\ni\n\ni\n\n(si, ui),\n\n(si, ui) = Rproxy\n\ni) \u2212 V proxy\n(s(cid:48)\n(si, ui) = rproxy\nis the proxy value parameterized by \u03d5i and s(cid:48)\n\n(6)\n(si, ui) is the proxy critic that can be chosen in a variety of ways [25, 26, 27, 28]. For\nleads to the REINFORCE algorithm [26]. In this paper, we choose\n(si) as the advantage function [27, 28], where\n\u03d5i\ni is the next state of agent i in the trajectory.\ni can be represented as\n\nwhere Aproxy\nexample, Aproxy\nAproxy\ni\nV proxy\n\u03d5i\nGiven (6) and a policy learning rate \u03b1, the updated policy parameter \u03b8(cid:48)\ni = \u03b8i + \u03b1\u2207\u03b8i log \u03c0\u03b8i(ui|si)Aproxy\n\u03b8(cid:48)\nThen, we build the connection between \u03b7 and J ex and specify the updating procedure for \u03b7. Given\nthe updated policy parameters \u03b8(cid:48)\n\ni\u2019s, using the chain rule, we have\n\n(si, ui) + V proxy\n\n(si, ui).\n\n\u03d5i\n\n\u2207\u03b7iJ ex = \u2207\u03b8(cid:48)\n\nJ ex\u2207\u03b7i\u03b8(cid:48)\ni.\n\n(7)\nThe spirit of (7) is to formulate the effect of the change of \u03b7i on in\ufb02uencing J ex through its in\ufb02uence\nin the updated policy parameter \u03b8(cid:48)\ni. This is a commonly adopted technique in meta-gradient learning\n[30, 31, 32, 33]. Computing the meta-gradient \u2207\u03b7iJ ex requires new samples generated by the updated\npolicy parameter \u03b8(cid:48)\ni, while this can be avoid by reusing the samples generated by \u03b8i with importance\nsampling [16]. In (7), \u2207\u03b8(cid:48)\n\nJ ex can be estimated by stochastic gradient as\n\ni\n\ni\n\ni\n\n\u2207\u03b8(cid:48)\n\n(ui|si)Aex(s, u),\n\nlog \u03c0\u03b8(cid:48)\n\n(8)\nwhere Aex(s, u) is the centralized extrinsic critic. Similar to proxy critics, we choose Aex(s, u) =\n\u03c6 (s(cid:48)) \u2212 V ex\nrex(s, u) + V ex\n\u03c6 (s), where V ex\n\u03c6 (s) is the extrinsic value parameterized by \u03c6. The second\nterm in (7) can be derived as\n\u2207\u03b7i\u03b8(cid:48)\n\n(si, ui)]\n\ni\n\ni\n\ni = \u2207\u03b7i[\u03b8i + \u03b1\u2207\u03b8i log \u03c0\u03b8i(ui|si)Aproxy\n= \u03b1\u03bb\u2207\u03b8i log \u03c0\u03b8i (ai|si)\u2207\u03b7i rproxy\n\ni\n(si, ui).\n\ni\n\n(9)\n\nFig. 1 gives an illustration of the entire architecture of the LIIR method. A sketch of the optimization\nalgorithm is presented in Algorithm 1.\n\n5 Experiments\n\nIn this section, we \ufb01rst evaluate LIIR on a simple 1D pursuit game speci\ufb01cally designed for the\nconsidered settings to see whether LIIR can learn reasonable distinct intrinsic rewards. Then, we\n\n5\n\n\fAlgorithm 1 The optimization algorithm for LIIR.\nInput: policy learning rate \u03b1 and intrinsic reward learning rate \u03b2.\nOutput: policy parameters \u03b8 and intrinsic reward parameters \u03b7.\n1: Init: initialize \u03b8 and \u03b7;\n2: while termination is not reached do\n3:\n\nSample a trajectory D = {s0, u0, s1, u1,\u00b7\u00b7\u00b7} by executing actions with the decentralized\npolicies {\u03c0\u03b81,\u00b7\u00b7\u00b7 , \u03c0\u03b8n};\nUpdate \u03b8 according to (6) with learning rate \u03b1;\nCompute (8) using new samples from {\u03c0\u03b8(cid:48)\n\u2207\u03b8(cid:48)\n\u03c0\u03b8(cid:48)\n\u03c0\u03b8i (ui|si) Aex(s, u);\nUpdate \u03b7 according to (7), step 5 and (9) with learning rate \u03b2;\n\n} or reuse D to replace (8) with\n\n,\u00b7\u00b7\u00b7 , \u03c0\u03b8(cid:48)\n\n(ui|si)\n\n, \u03c0\u03b8(cid:48)\n\n2\n\n1\n\nn\n\n4:\n5:\n\ni\n\ni\n\n6:\n7: end while\n\ncomprehensively study LIIR in several challenging micromanagement games in the game of StarCraft\nII, and compare LIIR with a number of state-of-the-art MARL methods.2\n\n5.1 A Simple 1D Pursuit Study\n\nWe design a simple game named 1D Pursuit\nto provide a fast veri\ufb01cation for the quality of\nthe intrinsic reward learned by LIIR. In 1D pur-\nsuit, a team of two agents are initially assigned\nwith some random integers denoted by x and y\nrespectively, and each agent could take actions\nfrom {+1,\u22121, 0} to either increase, decrease or\nkeep its value to approach a target value z that\nis unknown to the agents. For a collaborative\nsetting, the team reward for the two agents is\nset to be inversely proportional to the summa-\ntion of their absolute differences between their\nvalues and the target value. That is, both the\ntwo agents should adjust their values towards\nthe target value. The observation of each agent\nis a two-dimension vector containing its current\ninteger value and another agent\u2019s integer value.\nThe team reward is set to be +0.01 if both agents take actions that approaching the target value,\n\u22120.01 if both agents take actions that moving away from the target value, and 0 otherwise. The target\nvalue is set to be 0. The initial integers for the two agents are randomly generated from {\u221210, ..., 10}.\nWe implement LIIR based on the architecture depicted in Fig. 1. The detailed network structure is\nprovided in the supplementary material. In Fig. 2, we plot the histogram of the distributions of the\nintrinsic reward averaged from 1000 episodes. We denote actions approaching the target as \u201cGood\u201d\nactions and actions moving away from the target as \u201cBad\u201d actions. The result shows that LIIR can\nassign reasonable intrinsic reward to the agents.\n\nFigure 2: The distribution of the learned intrinsic re-\nwards v.s. frequencies (counts) of taking \u201cGood\u201d and\n\u201cBad\u201d actions from 1000 1D pursuit games.\n\n5.2 StarCraft II Micromanagement\n\nIn this subsection, we comprehensively evaluate the proposed LIIR method in the game of StarCraft\nII based on the learning environment SC2LE [34] and mini-game settings in SMAC [35]. We compare\nthe LIIR method with a number of state-of-the-art MARL methods that use the CLDE architecture.\nWe also provide some insightful case studies to visualize the learned intrinsic rewards.\nStarCraft II is a popular real-time strategy game and it has been studied under MARL settings\n[9, 10, 7, 36, 37]. In the experiments, we consider symmetric battle games in StarCraft II , where\nboth single type agents and mixed type agents are considered.\nSpeci\ufb01cally, the considered scenarios contain 3 Marines vs. 3 Marines (3M), 8 Marines vs. 8 Marines\n(8M), 2 Stalkers & 3 Zealots vs. 2 Stalkers & 3 Zealots (2S3Z), and 3 Stalkers & 5 Zealots vs. 3\n\n2The source codes of LIIR are available through https://github.com/yalidu/liir.\n\n6\n\n0.00.20.40.60.81.0Intrinsic reward rin0250500750100012501500Action frequenciesGood actionsBad actions\fStalkers & 5 Zealots (3S5Z). In these settings, Marine and Stalker are units of Terran and Protoss,\nrespectively, and both of them can attack enemies at a distance, while Zealot is a melee unit of Protoss\nand it can only attack enemies who stand close to it. In all these games, only the units from self side\nare treated as agents.\nEach agent is described by several attributes including the health point (HP), weapon cooling down\n(CD), shield (for 2S3Z and 3S5Z), unit type, last action and the relative distance of the observed units.\nThe enemy unit is described in the same way except that CD is excluded. The partial observation of\nan agent is composed by the attributes of the units, including both the agents and the enemy units,\nshown up within its view range that is a circle with a certain radius. The action space contains 4 move\ndirections, k attack actions where k is the \ufb01xed maximum number of the enemy units in a map, stop\nand none-operation. The input dimension and the output action dimension are \ufb01xed with a certain\nordering over the agents and enemy units. Dead enemy units will be masked out from the action\nspace to ensure the executed action is valid. At each time step, the agents receive a joint team reward\nwhich is de\ufb01ned by the total damage of the agents and the total damage from the enemy side. In\nall the scenarios, following the con\ufb01gurations in [9, 10], we train the agents against the build-in AI\nopponent. More detailed settings can be acquired from the SMAC environment [35].\n\n5.2.1 Compared Methods and Training Details\n\nThe considered methods for evaluation include\n\n\u2022 independent Q-learning (IQL) [17]: IQL trains decentralized Q-functions for each agent.\nSince the observation and action spaces of the agents are the same within a speci\ufb01c environ-\nmental setting, a policy will be shared across all the agents;\n\n\u2022 independent actor-critic (IAC) [9]: IAC is similar to IQL except that it adopts the actor-critic\n\nmethod;\n\n\u2022 Central-V [9]: the method learns a centralized critic with decentralized policies. Similarly,\n\nall agents share the same policy network;\n\n\u2022 COMA [9]: the method learns a centralized critic that is the state-action value minus a\n\ncounterfactual baseline;\n\n\u2022 QMIX [10]: the method learns decentralized Q-function for each agent with the assumption\nthat the centralized Q-value is monotonically increasing with the individual Q-values. In\nthe implementations, the agents share the same Q-function;\n\n\u2022 LIIR: the proposed method. In the experiments, the agents share the same policy, intrinsic\nreward function and proxy critic. Since each agent has its own partial observation, sharing\npolicy parameters does not imply that they act the same.\n\nFor COMA and QMIX, we use their original implementations, in which the main policy network or Q-\nnetwork consist of some fully connected (FC) layers and a GRU module.3 All the other methods adopt\nsimilar network structures compared to COMA and QMIX. As depicted in Fig. 1, the parameters\nof LIIR contain 4 components corresponding to the shared policy parameter \u03b8, intrinsic reward\nparameter \u03b7, proxy value parameter \u03d5 and extrinsic value parameter \u03c6. To achieve fair comparison,\nwe set the policy network structure, i.e., \u03b8, as what is exactly used for COMA\u2019s policy network. Then,\nwe compress the other parameters \u03b7, \u03d5 and \u03c6 to let their total size equal to the parameter size of the\nremaining part in COMA. More details can be found in the supplementary material. All the methods\nare trained with 3 millions of steps in 3M and 8M, and with 10 millions of steps for 2S3Z and 3S5Z.\nThe hyper-parameter \u03bb in (2) is set to 0.01 throughout the experiments (we tried different choices of\n\u03bb while we found that the results did not differ much). We use a \ufb01xed learning rate of 5e-4 and use\nbatches of 32 episodes for all the methods. We use 32 actors to generate the trajectories in parallel,\nand use one NVIDIA Tesla M40 GPU for training.\n\n5.2.2 Results\n\nTo evaluate the performance of each method, we freeze the training every 100 episodes and test the\nmodel over 20 episodes to compute an average test winning rate. The entire training procedure is\n\n3https://github.com/oxwhirl/pymarl\n\n7\n\n\f(a) 3M\n\n(b) 8M\n\n(c) 2S3Z\n\n(d) 3S5Z\n\nFigure 3: Test winning rates vs. training steps of various methods on all the scenarios.\n\nrepeated for 5 times to plot the winning rate curve with standard deviation. The results are reported\nin Fig. 3, where the averaged winning rates vs. the training steps on all the battle scenarios are given.\nIn 3M which is the simplest game, all the test winning rates keep increasing as the training steps\nincrease. In 8M, 2S3Z and 3S5Z, the independent learning methods, i.e., IQL and IAC, fail to learn\na good policy for the agents and the methods using a CLDE architecture always outperform the\nindependent learning methods. In 3M and 8M, COMA and Central-V show comparable performance,\nwhile in 2S3Z and 3S5Z, Central-V outperforms QMIX and COMA. For all these scenarios, the LIIR\nmethod consistently shows the best performance, and it achieves around 90% winning rate in all the\nscenarios. This demonstrates that learning the intrinsic reward function can ultimately induce better\ntrained policies.\n\n5.2.3 Visualizing the Learned Intrinsic Reward\n\nIn addition to evaluate the performance of the trained policy in Section 5.2.2, we are more curious\nabout how much effect the learned intrinsic reward function actually contributes to the policy learning.\nIn order to \ufb01gure out what has been learned in the intrinsic reward function, we propose to explicitly\nvisualize these rewards. That is, we plot the learned intrinsic reward of each agent at each time step\nin a complete trajectory during testing. It is worth mentioning that during testing the intrinsic rewards\nare independent with the learned policy, and these rewards will not be used at all when generating the\ntrajectory. For clarity, we randomly choose two test replays in 3M and 2S3Z which contain fewer\nagents to plot all the agents\u2019 intrinsic rewards. Figs. 4 and 5 show the intrinsic rewards in 3M and\n2S3Z, respectively. We also attach some auxiliary snapshots to explain some interesting segments in\nthe curves. In all the snapshots, the red colored units indicate the agents controlled by LIIR.\nIn Fig. 4(a), agent 1 is dead at time step 9, and we can observe that its intrinsic reward turns to be\nvery low after time step 6 compared to the other two agents. As revealed by Figs. 4(b) and (c), at\ntime step 6, all the three agents focus \ufb01re on one of the enemy Marine, while agent 1 has the lowest\nHP; after that, agent 1 still keeps \ufb01ring instead of running away from the enemies and the intrinsic\nreward function predicts a low rin\n1 , indicating that u1 = attack is not a good action at that time;\n\ufb01nally, agent 1 dies at time step 9 and the corresponding intrinsic reward is very low.\nIn Fig. 5(a), after time step 27, we see that agent 2\u2019s intrinsic reward increases a lot compared to\nthe other agents. Figs. 5(b) and (c) provides a clear explanation that at time step 27, agent 2 (with\n\n8\n\n0.00.51.01.52.02.53.0#Steps1e6020406080100Average Win Rate(%)LIIRCOMAQMIXCentral-VIQLIAC0.00.51.01.52.02.53.0#Steps1e6020406080100Average Win Rate(%)LIIRCOMAQMIXCentral-VIQLIAC0.00.20.40.60.81.0#Steps1e7020406080100Average Win Rate(%)LIIRCOMAQMIXCentral-VIQLIAC0.00.20.40.60.81.0#Steps1e7020406080100Average Win Rate(%)LIIRCOMAQMIXCentral-VIQLIAC\f(a) Intrinsic reward\n\n(b) t = 6\n\n(c) t = 9\n\nFigure 4: An example of the intrinsic reward curves and auxiliary snapshots on 3M.\n\n(a) Intrinsic reward\n\n(b) t = 27\n\n(c) t = 32\n\nFigure 5: An example of the intrinsic reward curves and auxiliary snapshots on 2S3Z.\n\nlow HP) stops \ufb01ring and runs along the red arrows (the move actions only take 4 directions here) to\navoid the attack from the enemy Zealot; until reaching an enemy Stalker at time step 32, agent 2\nstarts attacking the Stalker which is \ufb01nally killed. Moreover, the overall trend of both the curves in\nFigs. 4(a) and 5(a) keeps increasing, indicating that the controlled team \ufb01nally wins the game.\nBesides visualizing the two episodes illustrated above, we also provide overall statistics of the learned\nintrinsic reward. We collect the intrinsic reward for the action \u201cattack\u201d when the corresponding health\npoints are lower than 50% from 100 test episodes. We then compute the cosine similarity (a value\nin [-1, 1]) between the health point and the intrinsic reward. The averaged cosine similarity is 0.55\nfor 2S3Z and 0.67 for 3M. The results show that the health point and intrinsic reward are positively\ncorrelated. That is, when the health point is low, the intrinsic reward is generally low for taking the\n\u201cattack\u201d action as well, which is reasonable in this scenario.\nThe above case studies demonstrate that the learned intrinsic reward can indeed provide diverse\nfeedback signals for the agents and these signals are very informative in evaluating the agents\u2019\nimmediate behaviors.\n\n6 Conclusion\n\nWe have proposed a novel multi-agent reinforcement learning algorithm, which learns an individual\nintrinsic reward for each agent. The method can assign each agent a distinct intrinsic reward so\nthat the agents are stimulated differently, even when the environment only feedbacks a team reward.\nGiven the intrinsic reward for each agent, we de\ufb01ne each of them a proxy critic to direct their policy\nlearning via actor-critic algorithms. We show that the formulated multi-agent learning problem can\nbe viewed as a bilevel optimization problem. Our empirical results carried on the battle games in\nStarCraft II demonstrate that learning the intrinsic reward function could eventually induce better\ntrained policy compared with a number of state-of-the-art competitors. We further perform two case\nstudies to visualize the learned intrinsic reward values, and the results provide clear explanations on\nthe effects of the learned intrinsic rewards.\nFor future work, we are interested in applying the LIIR method to more challenging scenarios, such\nas real-world traf\ufb01c control with many agents and competitive multi-agent systems. Moreover, in\naddition to the simple summation form in (2), it is also interesting to investigate the optimal form of\nthe proxy reward function.\n\nAcknowledgments\n\nThe authors would like to thank anonymous reviewers for their constructive comments. Yali Du is\nduring an internship at Tencent AI Lab when working on this project.\n\n9\n\n\ud835\udc61=7Agent 1Agent 2Agent 3Attack\ud835\udc61=27Agent 2Agent 2AttackMove\fReferences\n[1] Marco Wiering. Multi-agent reinforcement learning for traf\ufb01c light control. In International\n\nConference on Machine Learning (ICML), pages 1151\u20131158, 2000.\n\n[2] Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep rein-\nforcement learning framework for autonomous driving. Electronic Imaging, 2017(19):70\u201376,\n2017.\n\n[3] Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. Resource manage-\nment with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot\nTopics in Networks, pages 50\u201356, 2016.\n\n[4] Micha\u0142 Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Ja\u00b4skowski.\nViZDoom: A Doom-based AI research platform for visual reinforcement learning. In IEEE\nConference on Computational Intelligence and Games, pages 341\u2013348, Sep 2016.\n\n[5] OpenAI. Openai \ufb01ve. https://blog.openai.com/openai-\ufb01ve/, 2018.\n\n[6] Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropa-\ngation. In Advances in Neural Information Processing Systems (NeurIPS), pages 2244\u20132252,\n2016.\n\n[7] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang.\nMultiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv\npreprint arXiv:1703.10069, 2017.\n\n[8] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooper-\nation. In Advances in Neural Information Processing Systems (NeurIPS), pages 7254\u20137264,\n2018.\n\n[9] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon\nWhiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on\nArti\ufb01cial Intelligence (AAAI), 2018.\n\n[10] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder Witt, Gregory Farquhar, Jakob Foerster,\nand Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent\nreinforcement learning. In International Conference on Machine Learning (ICML), pages\n4292\u20134301, 2018.\n\n[11] Marek Grze\u00b4s. Reward shaping in episodic reinforcement learning. In Proceedings of the 16th\nConference on Autonomous Agents and MultiAgent Systems, pages 565\u2013573. International\nFoundation for Autonomous Agents and Multiagent Systems, 2017.\n\n[12] Patrick Mannion, Sam Devlin, Jim Duggan, and Enda Howley. Reward shaping for knowledge-\nbased multi-objective multi-agent reinforcement learning. The Knowledge Engineering Review,\n33, 2018.\n\n[13] Satinder Singh, Richard L Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated\nreinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental\nDevelopment, 2(2):70\u201382, 2010.\n\n[14] Jonathan Sorg, Richard L Lewis, and Satinder P Singh. Reward design via online gradient\nascent. In Advances in Neural Information Processing Systems (NeurIPS), pages 2190\u20132198,\n2010.\n\n[15] Xiaoxiao Guo, Satinder Singh, Richard Lewis, and Honglak Lee. Deep learning for reward\ndesign to improve monte carlo tree search in atari games. In Proceedings of the Twenty-Fifth\nInternational Joint Conference on Arti\ufb01cial Intelligence (IJCAI), pages 1519\u20131525, 2016.\n\n[16] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient\nmethods. In Advances in Neural Information Processing Systems (NeurIPS), pages 4644\u20134654,\n2018.\n\n10\n\n\f[17] Ming Tan. Multi-agent reinforcement learning: independent versus cooperative agents. In\n\nInternational Conference on Machine Learning (ICML), pages 330\u2013337, 1993.\n\n[18] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King\u2019s\n\nCollege, Cambridge, 1989.\n\n[19] Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru,\nJaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement\nlearning. PloS one, 12(4), 2017.\n\n[20] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi,\nMax Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-\ndecomposition networks for cooperative multi-agent learning based on team reward. In Pro-\nceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems\n(AAMAS), pages 2085\u20132087, 2018.\n\n[21] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical\ndeep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In\nAdvances in Neural Information Processing Systems (NeurIPS), pages 3675\u20133683, 2016.\n\n[22] Nat Dilokthanakul, Christos Kaplanis, Nick Pawlowski, and Murray Shanahan. Feature control\nas intrinsic motivation for hierarchical reinforcement learning. IEEE Transactions on Neural\nNetworks and Learning Systems, 2019.\n\n[23] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration\nIn International Conference on Machine Learning (ICML),\n\nby self-supervised prediction.\nvolume 2017, 2017.\n\n[24] Bingyao Liu, Satinder Singh, Richard L Lewis, and Shiyin Qin. Optimal rewards for cooperative\n\nagents. IEEE Transactions on Autonomous Mental Development, 6(4):286\u2013297, 2014.\n\n[25] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce-\n\nment learning. Machine Learning, 8(3-4):229\u2013256, 1992.\n\n[26] Richard S Sutton, Andrew G Barto, and Francis Bach. Reinforcement learning: An introduction.\n\nMIT press, 1998.\n\n[27] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust\nregion policy optimization. In International Conference on Machine Learning (ICML), pages\n1889\u20131897, 2015.\n\n[28] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal\n\npolicy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\n\n[29] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted\nimitation learning for batched historical data. In Advances in Neural Information Processing\nSystems (NeurIPS), pages 6288\u20136297, 2018.\n\n[30] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom\nSchaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent\nby gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), pages\n3981\u20133989, 2016.\n\n[31] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.\nIn International Conference on\n\nMeta-learning with memory-augmented neural networks.\nMachine Learning (ICML), pages 1842\u20131850, 2016.\n\n[32] Alex Nichol, Joshua Achiam, and John Schulman. On \ufb01rst-order meta-learning algorithms.\n\narXiv preprint arXiv:1803.02999, 2018.\n\n[33] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In\n\nAdvances in Neural Information Processing Systems (NeurIPS), pages 2396\u20132407, 2018.\n\n11\n\n\f[34] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets,\nMichelle Yeo, Alireza Makhzani, Heinrich K\u00a8uttler, John Agapiou, and Julian Schrittwieser.\nStarcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782,\n2017.\n\n[35] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas\nNardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon\nWhiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019.\n\n[36] Lei Han, Peng Sun, Yali Du, Jiechao Xiong, Qing Wang, Xinghai Sun, Han Liu, and Tong Zhang.\nGrid-wise control for multi-agent reinforcement learning in video game ai. In International\nConference on Machine Learning (ICML), pages 2576\u20132585, 2019.\n\n[37] Peng Sun, Xinghai Sun, Lei Han, Jiechao Xiong, Qing Wang, Bo Li, Yang Zheng, Ji Liu,\nYongsheng Liu, Han Liu, et al. Tstarbots: Defeating the cheating level builtin ai in starcraft ii in\nthe full game. arXiv preprint arXiv:1809.07193, 2018.\n\n12\n\n\f", "award": [], "sourceid": 2448, "authors": [{"given_name": "Yali", "family_name": "Du", "institution": "University College London"}, {"given_name": "Lei", "family_name": "Han", "institution": "Tencent AI Lab"}, {"given_name": "Meng", "family_name": "Fang", "institution": "Tencent"}, {"given_name": "Ji", "family_name": "Liu", "institution": "University of Rochester, Tencent AI lab"}, {"given_name": "Tianhong", "family_name": "Dai", "institution": "Imperial College London"}, {"given_name": "Dacheng", "family_name": "Tao", "institution": "University of Sydney"}]}