{"title": "ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games", "book": "Advances in Neural Information Processing Systems", "page_first": 2659, "page_last": 2669, "abstract": "In this paper, we propose ELF, an Extensive, Lightweight and Flexible platform for fundamental reinforcement learning research. Using ELF, we implement a highly customizable real-time strategy (RTS) engine with three game environments (Mini-RTS, Capture the Flag and Tower Defense). Mini-RTS, as a miniature version of StarCraft, captures key game dynamics and runs at 165K frame-per-second (FPS) on a laptop. When coupled with modern reinforcement learning methods, the system can train a full-game bot against built-in AIs end-to-end in one day with 6 CPUs and 1 GPU. In addition, our platform is flexible in terms of environment-agent communication topologies, choices of RL methods, changes in game parameters, and can host existing C/C++-based game environments like ALE. Using ELF, we thoroughly explore training parameters and show that a network with Leaky ReLU and Batch Normalization coupled with long-horizon training and progressive curriculum beats the rule-based built-in AI more than 70% of the time in the full game of Mini-RTS. Strong performance is also achieved on the other two games. In game replays, we show our agents learn interesting strategies. ELF, along with its RL platform, is open-sourced at https://github.com/facebookresearch/ELF.", "full_text": "ELF: An Extensive, Lightweight and Flexible\n\nResearch Platform for Real-time Strategy Games\n\nYuandong Tian1 Qucheng Gong1 Wenling Shang2 Yuxin Wu1 C. Lawrence Zitnick1\n1{yuandong, qucheng, yuxinwu, zitnick}@fb.com\n\n2wendy.shang@oculus.com\n\n1Facebook AI Research\n\n2Oculus\n\nAbstract\n\nIn this paper, we propose ELF, an Extensive, Lightweight and Flexible platform\nfor fundamental reinforcement learning research. Using ELF, we implement a\nhighly customizable real-time strategy (RTS) engine with three game environ-\nments (Mini-RTS, Capture the Flag and Tower Defense). Mini-RTS, as a minia-\nture version of StarCraft, captures key game dynamics and runs at 40K frame-\nper-second (FPS) per core on a laptop. When coupled with modern reinforcement\nlearning methods, the system can train a full-game bot against built-in AIs end-\nto-end in one day with 6 CPUs and 1 GPU. In addition, our platform is \ufb02exible in\nterms of environment-agent communication topologies, choices of RL methods,\nchanges in game parameters, and can host existing C/C++-based game environ-\nments like ALE [4]. Using ELF, we thoroughly explore training parameters and\nshow that a network with Leaky ReLU [17] and Batch Normalization [11] cou-\npled with long-horizon training and progressive curriculum beats the rule-based\nbuilt-in AI more than 70% of the time in the full game of Mini-RTS. Strong per-\nformance is also achieved on the other two games. In game replays, we show\nour agents learn interesting strategies. ELF, along with its RL platform, is open\nsourced at https://github.com/facebookresearch/ELF.\n\n1\n\nIntroduction\n\nGame environments are commonly used for research in Reinforcement Learning (RL), i.e. how to\ntrain intelligent agents to behave properly from sparse rewards [4, 6, 5, 14, 29]. Compared to the\nreal world, game environments offer an in\ufb01nite amount of highly controllable, fully reproducible,\nand automatically labeled data. Ideally, a game environment for fundamental RL research is:\n\n\u2022 Extensive: The environment should capture many diverse aspects of the real world, such\nas rich dynamics, partial information, delayed/long-term rewards, concurrent actions with\ndifferent granularity, etc. Having an extensive set of features and properties increases the\npotential for trained agents to generalize to diverse real-world scenarios.\n\n\u2022 Lightweight: A platform should be fast and capable of generating samples hundreds or\nthousands of times faster than real-time with minimal computational resources (e.g., a sin-\ngle machine). Lightweight and ef\ufb01cient platforms help accelerate academic research of RL\nalgorithms, particularly for methods which are heavily data-dependent.\n\n\u2022 Flexible: A platform that is easily customizable at different levels, including rich choices\nof environment content, easy manipulation of game parameters, accessibility of internal\nvariables, and \ufb02exibility of training architectures. All are important for fast exploration of\ndifferent algorithms. For example, changing environment parameters [35], as well as using\ninternal data [15, 19] have been shown to substantially accelerate training.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fTo our knowledge, no current game platforms satisfy all criteria. Modern commercial games (e.g.,\nStarCraft I/II, GTA V) are extremely realistic, but are not customizable and require signi\ufb01cant re-\nsources for complex visual effects and for computational costs related to platform-shifting (e.g., a\nvirtual machine to host Windows-only SC I on Linux). Old games and their wrappers [4, 6, 5, 14])\nare substantially faster, but are less realistic with limited customizability. On the other hand, games\ndesigned for research purpose (e.g., MazeBase [29], \u00b5RTS [23]) are ef\ufb01cient and highly customiz-\nable, but are not very extensive in their capabilities. Furthermore, none of the environments consider\nsimulation concurrency, and thus have limited \ufb02exibility when different training architectures are\napplied. For instance, the interplay between RL methods and environments during training is often\nlimited to providing simplistic interfaces (e.g., one interface for one game) in scripting languages\nlike Python.\nIn this paper, we propose ELF, a research-oriented platform that offers games with diverse prop-\nerties, ef\ufb01cient simulation, and highly customizable environment settings. The platform allows for\nboth game parameter changes and new game additions. The training of RL methods is deeply and\n\ufb02exibly integrated into the environment, with an emphasis on concurrent simulations. On ELF,\nwe build a real-time strategy (RTS) game engine that includes three initial environments including\nMini-RTS, Capture the Flag and Tower Defense. Mini-RTS is a miniature custom-made RTS game\nthat captures all the basic dynamics of StarCraft (fog-of-war, resource gathering, troop building,\ndefense/attack with troops, etc). Mini-RTS runs at 165K FPS on a 4 core laptop, which is faster than\nexisting environments by an order of magnitude. This enables us for the \ufb01rst time to train end-to-\nend a full-game bot against built-in AIs. Moreover, training is accomplished in only one day using\n6 CPUs and 1 GPU. The other two games can be trained with similar (or higher) ef\ufb01ciency.\nMany real-world scenarios and complex games (e.g. StarCraft) are hierarchical in nature. Our RTS\nengine has full access to the game data and has a built-in hierarchical command system, which\nallows training at any level of the command hierarchy. As we demonstrate, this allows us to train\na full-game bot that acts on the top-level strategy in the hierarchy while lower-level commands are\nhandled using build-in tactics. Previously, most research on RTS games focused only on lower-level\nscenarios such as tactical battles [34, 25]. The full access to the game data also allows for supervised\ntraining with small-scale internal data.\nELF is resilient to changes in the topology of the environment-actor communication used for train-\ning, thanks to its hybrid C++/Python framework. These include one-to-one, many-to-one and one-\nto-many mappings. In contrast, existing environments (e.g., OpenAI Gym [6] and Universe [33])\nwrap one game in one Python interface, which makes it cumbersome to change topologies. Paral-\nlelism is implemented in C++, which is essential for simulation acceleration. Finally, ELF is capable\nof hosting any existing game written in C/C++, including Atari games (e.g., ALE [4]), board games\n(e.g. Chess and Go [32]), physics engines (e.g., Bullet [10]), etc, by writing a simple adaptor.\nEquipped with a \ufb02exible RL backend powered by PyTorch, we experiment with numerous baselines,\nand highlight effective techniques used in training. We show the \ufb01rst demonstration of end-to-\nend trained AIs for real-time strategy games with partial information. We use the Asynchronous\nAdvantagous Actor-Critic (A3C) model [21] and explore extensive design choices including frame-\nskip, temporal horizon, network structure, curriculum training, etc. We show that a network with\nLeaky ReLU [17] and Batch Normalization [11] coupled with long-horizon training and progressive\ncurriculum beats the rule-based built-in AI more than 70% of the time in full-game Mini-RTS. We\nalso show stronger performance in others games. ELF and its RL platform, is open-sourced at\nhttps://github.com/facebookresearch/ELF.\n\n2 Architecture\n\nELF follows a canonical and simple producer-consumer paradigm (Fig. 1). The producer plays N\ngames, each in a single C++ thread. When a batch of M current game states are ready (M < N), the\ncorresponding games are blocked and the batch are sent to the Python side via the daemon. The con-\nsumers (e.g., actor, optimizer, etc) get batched experience with history information via a Python/C++\ninterface and send back the replies to the blocked batch of the games, which are waiting for the next\naction and/or values, so that they can proceed. For simplicity, the producer and consumers are in\nthe same process. However, they can also live in different processes, or even on different machines.\nBefore the training (or evaluation) starts, different consumers register themselves for batches with\n\n2\n\n\fFigure 1: Overview of ELF.\n\ndifferent history length. For example, an actor might need a batch with short history, while an op-\ntimizer (e.g., T -step actor-critic) needs a batch with longer history. During training, the consumers\nuse the batch in various ways. For example, the actor takes the batch and returns the probabilties\nof actions (and values), then the actions are sampled from the distribution and sent back. The batch\nreceived by the optimizer already contains the sampled actions from the previous steps, and can be\nused to drive reinforcement learning algorithms such as A3C. Here is a sample usage of ELF:\n\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n\n# We run 1024 games concurrently .\nnum games = 1024\n\n# Wait for a batch of 256 games.\nbatchsize = 256\n\n# The return states contain key \u2019s \u2019,\n# The reply contains key \u2019a\u2019 to be \ufb01lled from the Python side .\n# The de\ufb01nitions of the keys are in the wrapper of the game.\ninput spec = dict (s=\u2019\u2019 , r=\u2019\u2019 ,\nreply spec = dict (a=\u2019\u2019)\n\n\u2019r\u2019 and \u2019 terminal \u2019\n\nterminal =\u2019\u2019)\n\ncontext = Init (num games, batchsize ,\n\ninput spec ,\n\nreply spec )\n\nInitialization of ELF\n\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n\n# Start all game threads and enter main loop.\ncontext . Start ()\nwhile True:\n\n# Wait for a batch of game states to be ready\n# These games will be blocked, waiting for\nbatch = context .Wait()\n\nreplies .\n\n# Apply a model to the game state . The output has key \u2019pi\u2019\noutput = model(batch)\n\n# Sample from the output\nreply [\u2019a\u2019 ][:] = SampleFromDistribution(output )\n\nto get\n\nthe actions of this batch .\n\n# Resume games.\ncontext . Steps ()\n\n# Stop all game threads .\ncontext .Stop()\n\nMain loop of ELF\n\nParallelism using C++ threads. Modern reinforcement learning methods often require heavy par-\nallelism to obtain diverse experiences [21, 22]. Most existing RL environments (OpenAI Gym [6]\nand Universe [33], RLE [5], Atari [4], Doom [14]) provide Python interfaces which wrap only sin-\ngle game instances. As a result, parallelism needs to be built in Python when applying modern RL\nmethods. However, thread-level parallelism in Python can only poorly utilize multi-core processors,\ndue to the Global Interpreter Lock (GIL)1. Process-level parallelism will also introduce extra data\nexchange overhead between processes and increase complexity to framework design. In contrast,\nour parallelism is achieved with C++ threads for better scaling on multi-core CPUs.\nFlexible Environment-Model Con\ufb01gurations. In ELF, one or multiple consumers can be used.\nEach consumer knows the game environment identities of samples from received batches, and typi-\ncally contains one neural network model. The models of different consumers may or may not share\nparameters, might update the weights, might reside in different processes or even on different ma-\nchines. This architecture offers \ufb02exibility for switching topologies between game environments and\nmodels. We can assign one model to each game environment, or one-to-one (e.g, vanilla A3C [21]),\nin which each agent follows and updates its own copy of the model. Similarly, multiple environ-\nments can be assigned to a single model, or many-to-one (e.g., BatchA3C [35] or GA3C [1]), where\nthe model can perform batched forward prediction to better utilize GPUs. We have also incorporated\nforward-planning methods (e.g., Monte-Carlo Tree Search (MCTS) [7, 32, 27]) and Self-Play [27],\nin which a single environment might emit multiple states processed by multiple models, or one-to-\nmany. Using ELF, these training con\ufb01gurations can be tested with minimal changes.\nHighly customizable and uni\ufb01ed interface. Games implemented with our RTS engine can be\ntrained using raw pixel data or lower-dimensional internal game data. Using internal game data is\n\n1The GIL in Python forbids simultaneous interpretations of multiple statements even on multi-core CPUs.\n\n3\n\nActorGame\t1Game\tNDaemon(batchcollector)Producer\t(Games\tin\tC++)ModelBatchwithhistoryinfoGame\t2HistorybufferHistorybufferHistorybufferConsumers(Python)OptimizerReply\fFigure 2: Hierarchical layout of ELF. In the current repository (https://github.com/\nfacebookresearch/ELF, master branch), there are board games (e.g., Go [32]), Atari learn-\ning environment [4], and a customized RTS engine that contains three simple games.\n\nFigure 3: Overview of Real-time strategy engine. (a) Visualization of current game state. (b) The\nthree different game environments and their descriptions.\n\ntypically more convenient for research focusing on reasoning tasks rather than perceptual ones. Note\nthat web-based visual renderings is also supported (e.g., Fig. 3(a)) for case-by-case debugging.\nELF allows for a uni\ufb01ed interface capable of hosting any existing game written in C/C++, including\nAtari games (e.g., ALE [4]), board games (e.g. Go [32]), and a customized RTS engine, with a\nsimple adaptor (Fig. 2). This enables easy multi-threaded training and evaluation using existing RL\nmethods. Besides, we also provide three concrete simple games based on RTS engine (Sec. 3).\nReinforcement Learning backend. We propose a Python-based RL backend.\nIt has a \ufb02exible\ndesign that decouples RL methods from models. Multiple baseline methods (e.g., A3C [21], Policy\nGradient [30], Q-learning [20], Trust Region Policy Optimization [26], etc) are implemented, mostly\nwith very few lines of Python codes.\n\n3 Real-time strategy Games\n\nReal-time strategy (RTS) games are considered to be one of the next grand AI challenges after Chess\nand Go [27]. In RTS games, players commonly gather resources, build units (facilities, troops, etc),\nand explore the environment in the fog-of-war (i.e., regions outside the sight of units are invisible)\nto invade/defend the enemy, until one player wins. RTS games are known for their exponential and\nchanging action space (e.g., 510 possible actions for 10 units with 5 choices each, and units of each\nplayer can be built/destroyed when game advances), subtle game situations, incomplete information\ndue to limited sight and long-delayed rewards. Typically professional players take 200-300 actions\nper minute, and the game lasts for 20-30 minutes.\nVery few existing RTS engines can be used directly for research. Commercial RTS games (e.g.,\nStarCraft I/II) have sophisticated dynamics, interactions and graphics. The game play strategies\nhave been long proven to be complex. Moreover, they are close-source with unknown internal states,\nand cannot be easily utilized for research. Open-source RTS games like Spring [12], OpenRA [24]\nand Warzone 2100 [28] focus on complex graphics and effects, convenient user interface, stable\nnetwork play, \ufb02exible map editors and plug-and-play mods (i.e., game extensions). Most of them\nuse rule-based AIs, do not intend to run faster than real-time, and offer no straightforward interface\nwith modern machine learning architectures. ORTS [8], BattleCode [2] and RoboCup Simulation\nLeague [16] are designed for coding competitions and focused on rule-based AIs. Research-oriented\nplatforms (e.g., \u00b5RTS [23], MazeBase [29]) are fast and simple, often coming with various baselines,\n\n4\n\nELFRTSEngineALEBoardGamesMini-RTSCapturetheFlagTowerDefensePongBreakoutEnemybaseYourbaseYourbarracksWorkerEnemyunitSelectedunitResource(a)Game NameDescriptionsAvgGame LengthMini-RTSGather resourceandbuild troops to destroy opponent\u2019s base. 1000-6000 ticks Capture the FlagCapturethe flag and bring it to your own base1000-4000 ticksTower DefenseBuilds defensivetowers to block enemy invasion.1000-2000 ticks(b)\fRealistic Code Resource Rule AIs Data AIs RL backend\n\nStarCraft I/II\nTorchCraft\n\nORTS, BattleCode\n\u00b5RTS, MazeBase\n\nMini-RTS\n\nHigh\nHigh\nMid\nLow\nMid\n\nNo\nYes\nNo\nYes\nYes\nTable 1: Comparison between different RTS engines.\n\nHigh\nHigh\nLow\nLow\nLow\n\nYes\nYes\nYes\nYes\nYes\n\nNo\nYes\nYes\nYes\nYes\n\nNo\nNo\nNo\nNo\nYes\n\nPlatform\n\nFrame per second\n\nALE [4]\n\n6000\n\nRLE [5]\n\n530\n\nUniverse [33]\n\n60\n\nMalmo [13]\n\n120\n\nPlatform\n\nDeepMind Lab [3] VizDoom [14]\n\nTorchCraft [31]\n\nMini-RTS\n\n287(C)/866(G)\n\nFrame per second\nTable 2: Frame rate comparison. Note that Mini-RTS does not render frames, but save game infor-\nmation into a C structure which is used in Python without copying. For DeepMind Lab, FPS is 287\n(CPU) and 866 (GPU) on single 6CPU+1GPU machine. Other numbers are in 1CPU core.\n\n2,000 (frameskip=50)\n\n40,000\n\n\u223c 7,000\n\nbut often with much simpler dynamics than RTS games. Recently, TorchCraft [31] provides APIs for\nStarCraft I to access its internal game states. However, due to platform incompatibility, one docker\nis used to host one StarCraft engine, and is resource-consuming. Tbl. 1 summarizes the difference.\n\n3.1 Our approach\n\nMany popular RTS games and its variants (e.g., StarCraft, DoTA, Leagues of Legends, Tower De-\nfense) share the same structure: a few units are controlled by a player, to move, attack, gather or cast\nspecial spells, to in\ufb02uence their own or an enemy\u2019s army. With our command hierarchy, a new game\ncan be created by changing (1) available commands (2) available units, and (3) how each unit emits\ncommands triggered by certain scenarios. For this, we offer simple yet effective tools. Researchers\ncan change these variables either by adding commands in C++, or by writing game scripts (e.g.,\nLua). All derived games share the mechanism of hierarchical commands, replay, etc. Rule-based\nAIs can also be extended similarly. We provide the following three games: Mini-RTS, Capture the\nFlag and Tower Defense (Fig. 3(b)). These games share the following properties:\nGameplay. Units in each game move with real coordinates, have dimensions and collision checks,\nand perform durative actions. The RTS engine is tick-driven. At each tick, AIs make decisions\nby sending commands to units based on observed information. Then commands are executed, the\ngame\u2019s state changes, and the game continues. Despite a fair complicated game mechanism, Mini-\nRTS is able to run 40K frames-per-second per core on a laptop, an order of magnitude faster than\nmost existing environments. Therefore, bots can be trained in a day on a single machine.\nBuilt-in hierarchical command levels. An agent could issue strategic commands (e.g., more ag-\ngressive expansion), tactical commands (e.g., hit and run), or micro-command (e.g., move a partic-\nular unit backward to avoid damage). Ideally strong agents master all levels; in practice, they may\nfocus on a certain level of command hierarchy, and leave others to be covered by hard-coded rules.\nFor this, our RTS engine uses a hierarchical command system that offers different levels of controls\nover the game. A high-level command may affect all units, by issuing low-level commands. A\nlow-level, unit-speci\ufb01c durative command lasts a few ticks until completion during which per-tick\nimmediate commands are issued.\nBuilt-in rule-based AIs. We have designed rule-based AIs along with the environment. These AIs\nhave access to all the information of the map and follow \ufb01xed strategies (e.g., build 5 tanks and\nattack the opponent base). These AIs act by sending high-level commands which are then translated\nto low-level ones and then executed.\nWith ELF, for the \ufb01rst time, we are able to train full-game bots for real-time strategy games and\nachieve stronger performance than built-in rule-based AIs. In contrast, existing RTS AIs are either\nrule-based or focused on tactics (e.g., 5 units vs. 5 units). We run experiments on the three games to\njustify the usability of our platform.\n\n5\n\n\fFigure 4: Frame-per-second per CPU core (no hyper-threading) with respect to CPUs/threads. ELF\n(light-shaded) is 3x faster than OpenAI Gym [6] (dark-shaded) with 1024 threads. CPU involved in\ntesting: Intel E5-2680@2.50GHz.\n\n4 Experiments\n\n4.1 Benchmarking ELF\n\nWe run ELF on a single server with a different number of CPU cores to test the ef\ufb01ciency of paral-\nlelism. Fig. 4(a) shows the results when running Mini-RTS. We can see that ELF scales well with\nthe number of CPU cores used to run the environments. We also embed Atari emulator [4] into\nour platform and check the speed difference between a single-threaded ALE and paralleled ALE per\ncore (Fig. 4(b)). While a single-threaded engine gives around 5.8K FPS on Pong, our paralleled ALE\nruns comparable speed (5.1K FPS per core) with up to 16 cores, while OpenAI Gym (with Python\nthreads) runs 3x slower (1.7K FPS per core) with 16 cores 1024 threads, and degrades with more\ncores. Number of threads matters for training since they determine how diverse the experiences\ncould be, with the same number of CPUs. Apart from this, we observed that Python multiprocessing\nwith Gym is even slower, due to heavy communication of game frames among processes. Note that\nwe used no hyperthreading for all experiments.\n\n4.2 Baselines on Real-time Strategy Games\n\nWe focus on 1-vs-1 full games between trained AIs and built-in AIs. Built-in AIs have access to\nfull information (e.g., number of opponent\u2019s tanks), while trained AIs know partial information in\nthe fog of war, i.e., game environment within the sight of its own units. There are exceptions: in\nMini-RTS, the location of the opponent\u2019s base is known so that the trained AI can attack; in Capture\nthe Flag, the \ufb02ag location is known to all; Tower Defense is a game of complete information.\nDetails of Built-in AI. For Mini-RTS there are two rule-based AIs: SIMPLE gathers, builds \ufb01ve\ntanks and then attacks the opponent base. HIT N RUN often harasses, builds and attacks. For\nCapture the Flag, we have one built-in AI. For Tower Defense (TD), no AI is needed. We tested our\nbuilt-in AIs against a human player and \ufb01nd they are strong in combat but exploitable. For example,\nSIMPLE is vulnerable to hit-and-run style harass. As a result, a human player has a win rate of 90%\nand 50% against SIMPLE and HIT N RUN, respectively, in 20 games.\nAction Space. For simplicity, we use 9 strategic (and thus global) actions with hard-coded execution\ndetails. For example, AI may issue BUILD BARRACKS, which automatically picks a worker to\nbuild barracks at an empty location, if the player can afford. Although this setting is simple, detailed\ncommands (e.g., command per unit) can be easily set up, which bear more resemblance to StarCraft.\nSimilar setting applies to Capture the Flag and Tower Defense. Please check Appendix for detailed\ndescriptions.\nRewards. For Mini-RTS, the agent only receives a reward when the game ends (\u00b11 for win/loss).\nAn average game of Mini-RTS lasts for around 4000 ticks, which results in 80 decisions for a\nframe skip of 50, showing that the game is indeed delayed in reward. For Capturing the Flag, we\ngive intermediate rewards when the \ufb02ag moves towards player\u2019s own base (one score when the \ufb02ag\n\u201ctouches down\u201d). In Tower Defense, intermediate penalty is given if enemy units are leaked.\n\n4.2.1 A3C baseline\n\nNext, we describe our baselines and their variants. Note that while we refer to these as baseline, we\nare the \ufb01rst to demonstrate end-to-end trained AIs for real-time strategy (RTS) games with partial\ninformation. For all games, we randomize the initial game states for more diverse experience and\n\n6\n\nKFPS per CPU core for Mini-RTS7060504030201001 core2 cores4 cores8 cores16 cores64 threads 128 threads 256 threads 512 threads 1024 threads6543210KFPS per CPU core for Pong (Atari)64 threads 128 threads 256 threads 512 threads 1024 threads1 core2 cores4 cores8 cores16 coresOpenAI GymELF\fFrameskip\n\n50\n20\n10\n\nSIMPLE\n68.4(\u00b14.3)\n61.4(\u00b15.8)\n52.8(\u00b12.4)\n\nHIT N RUN\n63.6(\u00b17.9)\n55.4(\u00b14.7)\n51.1(\u00b15.0)\n\nRandom\nTrained AI\n\nCapture Flag Tower Defense\n36.3 (\u00b1 0.3)\n0.7 (\u00b1 0.9)\n59.9 (\u00b1 7.4)\n91.0 (\u00b1 7.6)\n\nTable 3: Win rate of A3C models competing with built-in AIs over 10k games. Left: Mini-RTS.\nFrame skip of the trained AI is 50. Right: For Capture the Flag, frame skip of trained AI is 10,\nwhile the opponent is 50. For Tower Defense the frame skip of trained AI is 50, no opponent AI.\n\nGame\n\nReLU\n\nLeaky ReLU\n\nBN\n\nLeaky ReLU + BN\n\nMini-RTS SIMPLE\nMini-RTS HIT N RUN\nMedian Mean (\u00b1 std) Median Mean (\u00b1 std)\n54.7 (\u00b1 4.2)\n57.0 (\u00b1 6.8)\n52.8\n61.0 (\u00b1 2.6)\n60.3 (\u00b1 3.3)\n59.8\n64.4 (\u00b1 7.4 )\n57.5 (\u00b1 6.8)\n61.0\n63.6 (\u00b1 7.9)\n68.4 (\u00b1 4.3)\n72.2\n\n60.4\n60.2\n55.6\n65.5\n\nTable 4: Win rate in % of A3C models using different network architectures. Frame skip of both\nsides are 50 ticks. The fact that the medians are better than the means shows that different instances\nof A3C could converge to very different solutions.\n\nuse A3C [21] to train AIs to play the full game. We run all experiments 5 times and report mean\nand standard deviation. We use simple convolutional networks with two heads, one for actions and\nthe other for values. The input features are composed of spatially structured (20-by-20) abstractions\nof the current game environment with multiple channels. At each (rounded) 2D location, the type\nand hit point of the unit at that location is quantized and written to their corresponding channels.\nFor Mini-RTS, we also add an additional constant channel \ufb01lled with current resource of the player.\nThe input feature only contains the units within the sight of one player, respecting the properties of\nfog-of-war. For Capture the Flag, immediate action is required at speci\ufb01c situations (e.g., when the\nopponent just gets the \ufb02ag) and A3C does not give good performance. Therefore we use frame skip\n10 for trained AI and 50 for the opponent to give trained AI a bit advantage. All models are trained\nfrom scratch with curriculum training (Sec. 4.2.2).\nNote that there are several factors affecting the AI performance.\nFrame-skip. A frame skip of 50 means that the AI acts every 50 ticks, etc. Against an opponent with\nlow frame skip (fast-acting), A3C\u2019s performance is generally lower (Fig. 3). When the opponent has\nhigh frame skip (e.g., 50 ticks), the trained agent is able to \ufb01nd a strategy that exploits the long-\ndelayed nature of the opponent. For example, in Mini-RTS it will send two tanks to the opponent\u2019s\nbase. When one tank is destroyed, the opponent does not attack the other tank until the next 50-\ndivisible tick comes. Interestingly, the trained model could be adaptive to different frame-rates and\nlearn to develop different strategies for faster acting opponents. For Capture the Flag, the trained bot\nlearns to win 60% over built-in AI, with an advantage in frame skip. For even frame skip, trained\nAI performance is low.\nNetwork Architectures. Since the input is sparse and heterogeneous, we experiment on CNN ar-\nchitectures with Batch Normalization [11] and Leaky ReLU [18]. BatchNorm stabilizes the gradient\n\ufb02ow by normalizing the outputs of each \ufb01lter. Leaky ReLU preserves the signal of negative linear\nresponses, which is important in scenarios when the input features are sparse. Tbl. 4 shows that\nthese two modi\ufb01cations both improve and stabilize the performance. Furthermore, they are compli-\nmentary to each other when combined.\nHistory length. History length T affects the convergence speed, as well as the \ufb01nal performance\nof A3C (Fig. 5). While Vanilla A3C [21] uses T = 5 for Atari games, the reward in Mini-RTS\nis more delayed (\u223c 80 actions before a reward).\nIn this case, the T -step estimation of reward\nt=1 \u03b3t\u22121rt + \u03b3T V (sT ) used in A3C does not yield a good estimation of the true reward if\n\nR1 =(cid:80)T\n\nV (sT ) is inaccurate, in particular for small T . For other experiments we use T = 6.\nInteresting behaviors The trained AI learns to act promptly and use sophisticated strategies (Fig.\n6). Multiple videos are available in https://github.com/facebookresearch/ELF.\n\n7\n\n\fFigure 5: Win rate in Mini-RTS with respect to the amount of experience at different steps T in\nA3C. Note that one sample (with history) in T = 2 is equivalent to two samples in T = 1. Longer\nT shows superior performance to small step counterparts, even if their samples are more expensive.\n\nFigure 6: Game screenshots between trained AI (blue) and built-in SIMPLE (red). Player colors are\nshown on the boundary of hit point gauges. (a) Trained AI rushes opponent using early advantage.\n(b) Trained AI attacks one opponent unit at a time.\n(c) Trained AI defends enemy invasion by\nblocking their ways. (d)-(e) Trained AI uses one long-range attacker (top) to distract enemy units\nand one melee attacker to attack enemy\u2019s base.\n\n4.2.2 Curriculum Training\n\nWe \ufb01nd that curriculum training plays an important role in training AIs. All AIs shown in Tbl. 3\nand Tbl. 4 are trained with curriculum training. For Mini-RTS, we let the built-in AI play the \ufb01rst\nk ticks, where k \u223c Uniform(0, 1000), then switch to the AI to be trained. This (1) reduces the\ndif\ufb01culty of the game initially and (2) gives diverse situations for training to avoid local minima.\nDuring training, the aid of the built-in AIs is gradually reduced until no aid is given. All reported\nwin rates are obtained by running the trained agents alone with greedy policy.\nWe list the comparison with and without curriculum training in Tbl. 6. It is clear that the performance\nimproves with curriculum training. Similarly, when \ufb01ne-tuning models pre-trained with one type\nof opponent towards a mixture of opponents (e.g., 50%SIMPLE + 50%HIT N RUN), curriculum\ntraining is critical for better performance (Tbl. 5). Tbl. 5 shows that AIs trained with one built-in AI\ncannot do very well against another built-in AI in the same game. This demonstrates that training\nwith diverse agents is important for training AIs with low-exploitability.\n\n4.2.3 Monte-Carlo Tree Search\n\nMonte-Carlo Tree Search (MCTS) can be used for planning when complete information about the\ngame is known. This includes the complete state s without fog-of-war, and the precise forward\nmodel s(cid:48) = s(cid:48)(s, a). Rooted at the current game state, MCTS builds a game tree that is biased\n\nSIMPLE\n68.4 (\u00b14.3)\n34.6(\u00b113.1)\n49.4(\u00b110.0)\n51.8(\u00b110.6)\n\nMini-RTS\nHIT N RUN\n26.6(\u00b17.6)\n63.6 (\u00b17.9)\n46.0(\u00b115.3)\n54.7(\u00b111.2)\n\nCombined\n47.5(\u00b15.1)\n49.1(\u00b110.5)\n47.7(\u00b111.0)\n53.2(\u00b18.5)\n\nTable 5: Training with a speci\ufb01c/combined AIs. Frame skip of both sides is 50. When against\ncombined AIs (50%SIMPLE + 50%HIT N RUN), curriculum training is particularly important.\n\nGame\n\nSIMPLE\n\nHIT N RUN\n\nCombined(No curriculum)\n\nCombined\n\n8\n\nSamples used (in thousands)AI_SIMPLET=4 T=8 T=12 T=16 T=20 0.750 200 400600 8000.550.350.15Best win rate in evaluationSamples used (in thousands)AI_HIT_AND_RUNT=4 T=8 T=12 T=16 T=20 0.750 200 400600 8000.550.350.15Best win rate in evaluation(a)(b)(c)(d)(e)Trained AI (Blue)AI_SIMPLE (Red)WorkerShort-range TankLong-range Tank\fGame\n\nMini-RTS SIMPLE Mini-RTS HIT N RUN\n\nno curriculum training\nwith curriculum training\n\n66.0(\u00b12.4)\n68.4 (\u00b14.3)\n\n54.4(\u00b115.9)\n63.6 (\u00b17.9)\n\nCapture the Flag\n54.2(\u00b120.0)\n59.9 (\u00b17.4)\n\nTable 6: Win rate of A3C models with and without curriculum training. Mini-RTS: Frame skip of\nboth sides are 50 ticks. Capture the Flag: Frame skip of trained AI is 10, while the opponent is\n50. The standard deviation of win rates are large due to instability of A3C training. For example in\nCapture the Flag, highest win rate reaches 70% while lowest win rate is only 27%.\n\nMini-RTS SIMPLE Mini-RTS HIT N RUN\n\nGame\nRandom\nMCTS\n\n24.2(\u00b13.9)\n73.2(\u00b10.6)\n\n25.9(\u00b10.6)\n62.7(\u00b12.0)\n\nTable 7: Win rate using MCTS over 1000 games. Both players use a frameskip of 50.\n\ntowards paths with high win rate. Leaves are expanded with all candidate moves and the win rate\nestimation is computed by random self-play until the game ends. We use 8 threads, each with 100\nrollouts. We use root parallelization [9] in which each thread independently expands a tree, and are\ncombined to get the most visited action. As shown in Tbl. 7, MCTS achieves a comparable win rate\nto models trained with RL. Note that the win rates of the two methods are not directly comparable,\nsince RL methods have no knowledge of game dynamics, and its state knowledge is reduced by\nthe limits introduced by the fog-of-war. Also, MCTS runs much slower (2-3sec per move) than the\ntrained RL AI (\u2264 1msec per move).\n\n5 Conclusion and Future Work\n\nIn this paper, we propose ELF, a research-oriented platform for concurrent game simulation which\noffers an extensive set of game play options, a lightweight game simulator, and a \ufb02exible envi-\nronment. Based on ELF, we build a RTS game engine and three initial environments (Mini-RTS,\nCapture the Flag and Tower Defense) that run 40KFPS per core on a laptop. As a result, a full-\ngame bot in these games can be trained end-to-end in one day using a single machine. In addition\nto the platform, we provide throughput benchmarks of ELF, and extensive baseline results using\nstate-of-the-art RL methods (e.g, A3C [21]) on Mini-RTS and show interesting learnt behaviors.\nELF opens up many possibilities for future research. With this lightweight and \ufb02exible platform, RL\nmethods on RTS games can be explored in an ef\ufb01cient way, including forward modeling, hierarchical\nRL, planning under uncertainty, RL with complicated action space, and so on. Furthermore, the\nexploration can be done with an affordable amount of resources. As future work, we will continue\nimproving the platform and build a library of maps and bots to compete with.\n\nReferences\n[1] Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Re-\nInternational\n\ninforcement learning through asynchronous advantage actor-critic on a gpu.\nConference on Learning Representations (ICLR), 2017.\n\n[2] BattleCode. Battlecode, mit\u2019s ai programming competition: https://www.battlecode.org/.\n\n2000. URL https://www.battlecode.org/.\n\n[3] Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich\nK\u00a8uttler, Andrew Lefrancq, Simon Green, V\u00b4\u0131ctor Vald\u00b4es, Amir Sadik, Julian Schrittwieser,\nKeith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen\nKing, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.03801,\n2016. URL http://arxiv.org/abs/1612.03801.\n\n[4] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning\nenvironment: An evaluation platform for general agents. CoRR, abs/1207.4708, 2012. URL\nhttp://arxiv.org/abs/1207.4708.\n\n9\n\n\f[5] Nadav Bhonker, Shai Rozenberg, and Itay Hubara. Playing SNES in the retro learning envi-\nronment. CoRR, abs/1611.02205, 2016. URL http://arxiv.org/abs/1611.02205.\n\n[6] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang,\nand Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016. URL http://arxiv.\norg/abs/1606.01540.\n\n[7] Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowl-\ning, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon\nColton. A survey of monte carlo tree search methods. IEEE Transactions on Computational\nIntelligence and AI in games, 4(1):1\u201343, 2012.\n\n[8] Michael Buro and Timothy Furtak. On the development of a free rts game engine. In Game-\n\nOnNA Conference, pages 23\u201327, 2005.\n\n[9] Guillaume MJ-B Chaslot, Mark HM Winands, and H Jaap van Den Herik. Parallel monte-carlo\ntree search. In International Conference on Computers and Games, pages 60\u201371. Springer,\n2008.\n\n[10] Erwin Coumans. Bullet physics engine. Open Source Software: http://bulletphysics.org, 2010.\n\n[11] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training\n\nby reducing internal covariate shift. ICML, 2015.\n\n[12] Stefan Johansson and Robin Westberg. Spring: https://springrts.com/. 2008. URL https:\n\n//springrts.com/.\n\n[13] Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for\nIn International joint conference on arti\ufb01cial intelli-\n\narti\ufb01cial intelligence experimentation.\ngence (IJCAI), page 4246, 2016.\n\n[14] Micha\u0142 Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Ja\u00b4skowski.\nVizdoom: A doom-based ai research platform for visual reinforcement learning. arXiv preprint\narXiv:1605.02097, 2016.\n\n[15] Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement\n\nlearning. arXiv preprint arXiv:1609.05521, 2016.\n\n[16] RoboCup\n\nSimulation\n\nLeague.\n\nhttps://en.wikipedia.org/wiki/robocup simulation league.\n//en.wikipedia.org/wiki/RoboCup_Simulation_League.\n\n1995.\n\nRobocup\n\nsimulation\n\nleague:\nURL https:\n\n[17] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Recti\ufb01er nonlinearities improve neural\n\nnetwork acoustic models. In Proc. ICML, volume 30, 2013.\n\n[18] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Recti\ufb01er nonlinearities improve neural\n\nnetwork acoustic models. 2013.\n\n[19] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Ban-\nino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and\nRaia Hadsell. Learning to navigate in complex environments. ICLR, 2017.\n\n[20] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G\nBellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.\nHuman-level control through deep reinforcement learning. Nature, 518(7540):529\u2013533, 2015.\n\n[21] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lill-\nicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep\nreinforcement learning. arXiv preprint arXiv:1602.01783, 2016.\n\n[22] Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De\nMaria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane\nLegg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Massively parallel methods\nfor deep reinforcement learning. CoRR, abs/1507.04296, 2015. URL http://arxiv.org/\nabs/1507.04296.\n\n10\n\n\f[23] Santiago Ontan\u00b4on. The combinatorial multi-armed bandit problem and its application to real-\ntime strategy games. In Proceedings of the Ninth AAAI Conference on Arti\ufb01cial Intelligence\nand Interactive Digital Entertainment, pages 58\u201364. AAAI Press, 2013.\n\n[24] OpenRA. Openra: http://www.openra.net/. 2007. URL http://www.openra.net/.\n\n[25] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang.\nMultiagent bidirectionally-coordinated nets for learning to play starcraft combat games. CoRR,\nabs/1703.10069, 2017. URL http://arxiv.org/abs/1703.10069.\n\n[26] John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust\n\nregion policy optimization. In ICML, pages 1889\u20131897, 2015.\n\n[27] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van\nDen Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanc-\ntot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529\n(7587):484\u2013489, 2016.\n\n[28] Pumpkin Studios. Warzone 2100: https://wz2100.net/. 1999. URL https://wz2100.\n\nnet/.\n\n[29] Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve, Soumith Chintala, and Rob Fergus.\nMazebase: A sandbox for learning from games. CoRR, abs/1511.07401, 2015. URL http:\n//arxiv.org/abs/1511.07401.\n\n[30] Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. Policy gra-\ndient methods for reinforcement learning with function approximation. In NIPS, volume 99,\npages 1057\u20131063, 1999.\n\n[31] Gabriel Synnaeve, Nantas Nardelli, Alex Auvolat, Soumith Chintala, Timoth\u00b4ee Lacroix, Zem-\ning Lin, Florian Richoux, and Nicolas Usunier. Torchcraft: a library for machine learn-\ning research on real-time strategy games. CoRR, abs/1611.00625, 2016. URL http:\n//arxiv.org/abs/1611.00625.\n\n[32] Yuandong Tian and Yan Zhu. Better computer go player with neural network and long-term\n\nprediction. arXiv preprint arXiv:1511.06410, 2015.\n\n[33] Universe. 2016. URL universe.openai.com.\n\n[34] Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic exploration\nICLR,\n\nfor deep deterministic policies: An application to starcraft micromanagement tasks.\n2017.\n\n[35] Yuxin Wu and Yuandong Tian. Training agent for \ufb01rst-person shooter game with actor-critic\n\ncurriculum learning. International Conference on Learning Representations (ICLR), 2017.\n\n11\n\n\f", "award": [], "sourceid": 1522, "authors": [{"given_name": "Yuandong", "family_name": "Tian", "institution": "Facebook AI Research"}, {"given_name": "Qucheng", "family_name": "Gong", "institution": "Facebook AI Research"}, {"given_name": "Wenling", "family_name": "Shang", "institution": "University of Amsterdam"}, {"given_name": "Yuxin", "family_name": "Wu", "institution": "Facebook AI Research"}, {"given_name": "C. Lawrence", "family_name": "Zitnick", "institution": "Facebook AI Research"}]}