{"title": "Speeding up the Parti-Game Algorithm", "book": "Advances in Neural Information Processing Systems", "page_first": 1595, "page_last": 1602, "abstract": null, "full_text": "Speeding up the Parti-Game Algorithm\n\nMaxim Likhachev\n\nSchool of Computer Science\nCarnegie Mellon University\n\nPittsburgh, PA 15213\n\nSven Koenig\n\nCollege of Computing\n\nGeorgia Institute of Technology\n\nAtlanta, GA 30312-0280\n\nmaxim+@cs.cmu.edu\n\nskoenig@cc.gatech.edu\n\nAbstract\n\nIn this paper, we introduce an ef\ufb01cient replanning algorithm for nonde-\nterministic domains, namely what we believe to be the \ufb01rst incremental\nheuristic minimax search algorithm. We apply it to the dynamic dis-\ncretization of continuous domains, resulting in an ef\ufb01cient implemen-\ntation of the parti-game reinforcement-learning algorithm for control in\nhigh-dimensional domains.\n\n1 Introduction\n\nWe recently developed Lifelong Planning A* (LPA*), a search algorithm for deterministic\ndomains that combines incremental and heuristic search to reduce its search time [1]. In-\ncremental search reuses information from previous searches to \ufb01nd solutions to series of\nsimilar search tasks faster than is possible by solving each search task from scratch [2],\nwhile heuristic search uses distance estimates to focus the search and solve search prob-\nlems faster than uninformed search. In this paper, we extend LPA* to nondeterministic\ndomains. We believe that the resulting search algorithm, called Minimax LPA*, is the \ufb01rst\nincremental heuristic minimax search algorithm. We apply it to the dynamic discretization\nof continuous domains, resulting in an ef\ufb01cient implementation of the popular parti-game\nalgorithm [3]. Our \ufb01rst experiments suggest that this implementation of the parti-game\nalgorithm can be an order of magnitude faster in two-dimensional domains than one with\nuninformed search from scratch and thus might allow the parti-game algorithm to scale up\nto larger domains. There also exist other ways of decreasing the amount of search per-\nformed by the parti-game algorithm. We demonstrate some advantages of Minimax LPA*\nover Prioritized Sweeping [4] in [5] but it is future work to compare it with the algorithms\ndeveloped in [6].\n\n2 Parti-Game Algorithm\n\nThe objective of the parti-game algorithm is to move an agent from given start coordinates\nto given goal coordinates in continuous and potentially high-dimensional domains with ob-\nstacles of arbitrary shapes. It is popular because it is simple, ef\ufb01cient, and applies to a broad\nrange of control problems. To solve these problems, one can \ufb01rst discretize the domains\nand then use conventional search algorithms to determine plans that move the agent to the\ngoal coordinates. However, uniform discretizations can prevent one from \ufb01nding a plan if\n\n\f(a)\n\n(d)\n\nS1\n\nS0\n\nS3\n\nA\n\nS2\n\nS5\n\nS4\n\nS7\n\nSgoal\n\nS9\n\nS8\n\nS11\n\nS10\n\n(b)\n\ngd=18\nh=0\ng=18\nrhs=18\ns3\n\n6\n\n6\n\n6\n\n6\n\n6\n\nh=6\ng=12\nrhs=12\ns2\n\ngd=12\n\n6\n\ngd=12\nh=6\ng=12\nrhs=12\ns5\n\n6\n\n6\n\n6\n\nh=6\ng=6\nrhs=6\ns4\ngd=6\n\n6\n\n6\n\n6\n\n6\n\n6\n\n(c)\n\n6\n\n6\n\ngd=30\nh=0\ng=30\nrhs=30\ns3\n\n6\n\n6\n\n6\n\n6\n\n6\n\nh=6\ng=12\nrhs=12\ns2\n\ngd=12\n\n6\n\ngd=12\nh=6\ng=12\nrhs=12\ns5\n\n6\n\n6\n\n6\n\nh=6\ng=6\nrhs=6\ns4\ngd=6\n\n6\n\n6\n\n6\n\ngd=24\nh=6\ng=\u00a5\nrhs=24\ns1\n\n6\n\n6\n\nh=6\ng=\u00a5\nrhs=18\ns0\n\ngd=18\n\n6\n\n6\n\ngd=24\nh=6\ng=24\nrhs=24\ns1\n\n6\n\n6\n\nh=6\ng=18\nrhs=18\ns0\n\ngd=18\n\n6\n\n6\n\ngd=6\nh=12\ng=6\nrhs=6\ns7\n\n6\n\n6\n\nh=12\ng=0\nrhs=0\nsgoal\ngd=0\n\ngd=6\nh=12\ng=6\nrhs=6\ns7\n\n6\n\n6\n\nh=12\ng=0\nrhs=0\nsgoal\ngd=0\n\n6\n\n6\n\n6\n\n6\n\ngd=12\nh=18\ng=\u00a5\nrhs=12\ns9\n\n6\n\n6\n\nh=18\ng=\u00a5\nrhs=6\ns8\ngd=6\n\ngd=12\nh=18\ng=12\nrhs=12\ns9\n\n6\n\n6\n\nh=18\ng=6\nrhs=6\ns8\ngd=6\n\n6\n\n6\n\n6\n\n6\n\n6\n\n6\n\n6\n\n6\n\ngd=18\nh=24\ng=\u00a5\nrhs=\u00a5\ns11\n\n6\n\n6\n\n6\n\n6\n\nh=24\ng=\u00a5\nrhs=\u00a5\ns10\ngd=12\n\ngd=18\nh=24\ng=\u00a5\nrhs=18\ns11\n\n6\n\n6\n\n6\n\n6\n\nh=24\ng=\u00a5\nrhs=12\ns10\ngd=12\n\ngd=24\nh=6\ng=24\nrhs=24\ns1\n\n6\n\n6\n\nh=6\ng=18\nrhs=18\ns0\n\ngd=18\n\n6\n\n6\n\ngd=\u00a5\nh=0\ng=\u00a5\nrhs=\u00a5\ns3\n\n6\n\n6\n\n6\n\n6\n\n6\n\nh=6\ng=12\nrhs=12\ns2\n\ngd=12\n\n6\n\ngd=12\nh=6\ng=12\nrhs=12\ns5\n\n6\n\n6\n\n6\n\nh=6\ng=6\nrhs=6\ns4\ngd=6\n\n6\n\n6\n\n6\n\ngd=6\nh=12\ng=6\nrhs=6\ns7\n\n6\n\n6\n\nh=12\ng=0\nrhs=0\nsgoal\ngd=0\n\n6\n\n6\n\n6\n\n6\n\ngd=12\nh=18\ng=12\nrhs=12\ns9\n\n6\n\n6\n\nh=18\ng=6\nrhs=6\ns8\ngd=6\n\n6\n\n6\n\n6\n\n6\n\ngd=18\n\nh=24\ng=18\nrhs=18\ns11\n\n6\n\n6\n\n6\n\n6\n\nh=24\ng=12\nrhs=12\ns10\ngd=12\n\n(e)\n\nS\u2019\u20191\n\nS\u20191\n\nS0\n\nS\u2019\u20193\n\nA\n\nS\u20193\n\nA\n\nS\u2019\u20192\n\nS\u20192\n\nS\u2019\u20195\n\nS\u20195\n\nS4\n\nS7\n\nSgoal\n\nS9\n\nS8\n\nS11\n\nS10\n\n(f)\n\n6\n\n6\n\n6\n\n6\n\n6\n\n6\n\nh=12\ng=\u00a5\nrhs=6\ns7\n\n6\n\n6\n\n6\n\nh=12\ng=0\nrhs=0\nsgoal\n\n6\n\nh=18\ng=\u00a5\nrhs=\u00a5\ns9\n\n6\n\n6\n\n6\n\nh=18\ng=\u00a5\nrhs=6\ns8\n\n6\n\n6\n\nh=24\ng=\u00a5\nrhs=\u00a5\ns11\n\n6\n\n6\n\nh=24\ng=\u00a5\nrhs=\u00a5\ns10\n\n6\n\n6\n\n6\n\n6\n\nh=6\ng=\u00a5\nrhs=\u00a5\ns\u2019\u20195\n\n3\n\n3\n\nh=6\ng=\u00a5\nrhs=11\ns\u20195\n\n5\n\n6\n\n6\n\n6\n\n6\n\n5\n\n6\n\n6\n\nh=6\ng=6\nrhs=6\ns4\n\n6\n\n6\n\n6\n\nh=6\ng=\u00a5\nrhs=\u00a5\ns\u2019\u20191\n\n3\n\n3\n\nh=6\ng=\u00a5\nrhs=24\ns\u20191\n\n5\n\n6\n\n6\n\n6\n\n5\n\nh=6\ng=\u00a5\nrhs=18\ns0\n\n6\n\n6\n\n6\n\nh=3\ng=\u00a5\nrhs=24\ns\u2019\u20193\n\n3\n\n3\n\nh=0\ng=15\nrhs=15\ns\u20193\n\n3\n\n3\n\nh=3\ng=12\nrhs=12\ns\u2019\u20192\n\n3\n\n3\n\nh=6\ng=\u00a5\nrhs=12\ns\u20192\n\nFigure 1: Example behavior of the parti-game algorithm\n\nthey are too coarse-grained (for example, because the resolution prevents one from notic-\ning small gaps between obstacles) and results in large state spaces that cannot be searched\nef\ufb01ciently if they are too \ufb01ne-grained. The parti-game algorithm solves this dilemma by\nstarting with a coarse discretization and re\ufb01nes it during execution only when and where it\nis needed (for example, around obstacles), resulting in a nonuniform discretization.\n\nWe use a simple two-dimensional robot navigation domain to illustrate the behavior of the\nparti-game algorithm. Figure 1(a) shows the initial discretization of our example domain\ninto 12 large cells together with the start coordinates of the agent (A) and the goal region\n(cell containing \u0002\u0001\u0004\u0003\u0006\u0005\b\u0007 ). Thus, it can always attempt to move towards the center of each\nadjacent cell (that is, cell that its current cell shares a border line with). The agent can\ninitially attempt to move towards the centers of either \n\t , \u0002\u000b , or \u0002\f , as shown in the \ufb01gure.\nFigure 1(b) shows the state space that corresponds to the discretized domain under this\nassumption. Each state corresponds to a cell and each action corresponds to a movement\noption. The parti-game algorithm initially ignores obstacles and makes the optimistic (and\nsometimes wrong) assumption that each action deterministically reaches the intended state,\nfor example, that the agent indeed reaches \nif it is somewhere in \u0002\r and moves towards\nthe center of \u0002\f . The costs of an action outcome approximates the Euclidean distance from\nthe center of the old cell of the agent to the center of its new cell. 1 (The cost of the action\n\n1We compute both the costs of action outcomes and the heuristics of states using an imaginary\nuniform grid, shown in gray in Figures 1(a) and (e), whose cell size corresponds to the resolution\nlimit of the parti-game algorithm. The cost of an action outcome is then computed as the maximum\nof the absolute values of the differences of the x and y coordinates between the imaginary grid cell\n\n\u00a5\n\u00a5\n\u00a5\n\u00a5\n\u00a5\n\f\n\ft1\n\nt0\n\nA\n\nO1\n\nO2\n\nA\n\nO3\n\nA\n\nt2\n\nFigure 2: Example of a nondeterministic action\n\nis\n\n\u000b or \n\n\u000b or \n\nif it is somewhere in \n\noutcome is in\ufb01nity if the old and new cells are identical since the action then cannot be\npart of a plan that minimizes the worst-case plan-execution cost from the current state of\nthe agent to \n\u0001\u0004\u0003\u0006\u0005\b\u0007 .) The parti-game algorithm then determines whether the minimax goal\ndistance of the current state \u0001\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\t\b\u0001\n of the agent is \ufb01nite. If so, the parti-game algorithm\nrepeatedly chooses the action that minimizes the worst-case plan-execution cost, until the\nagent reaches \u0002\u0001\u0004\u0003\u0006\u0005\n\u0007 or observes additional action outcomes. The minimax goal distance\n\u0002\r\u0014\u0013\u0016\u0015\u0018\u0017\u0014\u0019 and the agent minimizes the worst-case plan-execution cost by\nof \u0005\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u000e\u0010\u000f\u0012\u0011\nmoving from \u0002\r\n\f . Assume that it decides to move\ntowards the centers of either \ntowards the center of \n\u000b . The agent always continues to move until it either gets blocked by\nan obstacle or enters a new cell. It immediately gets blocked by the obstacle in \n\r . When\nthe agent observes additional action outcomes it adds them to the state space. Thus, it now\n\r and moves towards\nassumes that it can end up in either \n\u000b . The same scenario repeats when the agent \ufb01rst attempts to move towards\nthe center of \nthe center of \n\f and then attempts to move towards the center of \n\t but gets blocked twice\nby the obstacle in \u0002\r . Figure 1(c) shows the state space after the attempted moves towards\nthe centers of \n\u000b and \u0002\f , and Figure 1(d) shows the state space after the attempted move\ntowards the center of \n.\nis not guaranteed to reach\nWe say that \u0005\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\nIn this case, the parti-game algorithm re\ufb01nes the\n\u0001\u0004\u0003\u0006\u0005\ndiscretization by splitting all solvable cells that border unsolvable cells and all unsolvable\ncells that border solvable cells. Each cell is split into two cells perpendicular to its longest\naxis. (The axis of the split is chosen randomly for square cells.) Figure 1(e) shows the new\ndiscretization of the domain. The parti-game algorithm then removes those states (and their\nactions) from the state space that correspond to the old cells and adds states (and actions)\nfor the new cells, again making the optimistic assumption that each action for the new states\ndeterministically reaches the intended state. This ensures that the minimax goal distance\n\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\t\b\u0001\n becomes \ufb01nite. Figure 1(f) shows the resulting state space. The parti-game\nof \n\u0007 or the domain cannot\nalgorithm now repeats the process until either the agent reaches \nbe discretized any further because the resolution limit is reached.\n\n\t . The minimax goal distance of \nis unsolvable since an agent in \u0001\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\r\b\u001f\n\n\u0007 with \ufb01nite plan-execution cost.\n\n\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\t\b\u0001\n\nis now\n\n\u0013\u001c\u0015\u001e\u001d\n\n\u000e\u001a\u000f\u001b\u0011\n\n\u0001\b\u0003\n\nIf all actions either did indeed deterministically reach their intended states or did not change\nthe state of the agent at all (as in the example from Figure 1), then the parti-game algorithm\ncould determine the minimax goal distances of the states with a deterministic search al-\ngorithm after it has removed all actions that have an action outcome that leaves the state\nunchanged (since these actions cannot be part of a plan with minimal worst-case plan-\nexecution cost). However, actions can have additional outcomes, as Figure 2 illustrates.\nif it moves from some-\nFor example, an agent cannot only end up in\nwhere in\n. The parti-game algorithm therefore needs to determine\nthe minimax goal distances of the states with a minimax search algorithm. Furthermore,\nthe parti-game algorithm repeatedly determines plans that minimize the worst-case plan-\n\ntowards the center of\n\n\u000b but also in\n\nand\n\n \"!\n\nthat contains the center of the new and old state of the agent. Similarly, the heuristic of a state\nis computed as the maximum of the absolute differences of the x and y coordinates between the\nimaginary grid cell that contains the center of the state of the agent and the imaginary grid cell that\ncontains the center of the state in question. Note that the grid is imaginary and never needs to be\nconstructed. Furthermore, it is only used to compute the costs and heuristics and does not restrict\neither the placement of obstacles or the movement of the agent.\n\n\n\n\n\u0005\n \n \n\t\n \n\u000b\n \n!\n\fThe pseudocode uses the following functions to manage the priority queue\nU.TopKey\nsmallest priority in\ndoes nothing if the current priority of\n\nreturns the smallest priority of all states in\n\u0001\u0003\f\u000e\r\u0010\u000f\u0011\u0002\n\nand returns the state. U.Insert\n\n. (If\ninserts\n\n.) Finally, U.Remove\n\nalready equals\n\ninto\n\n\u0001\u0004\u0002\n\nreturns a state with the smallest priority of all states in\n\n.) U.Pop\n\nchanges the priority of\n\n\u0001\u0003\u0002\n\n.\ndeletes the state with the\n. (It\n\nin\n\nto\n\nreturns\n\u0001\u0004\u0002\n. U.Update\nfrom\n\n.\n\n\u0006\b\u0007\t\u0006\u000b\n\n\u0001\u0003\f\u000e\r\u0012\u000f\u0011\u0002\n\nis empty, then U.TopKey\n\n: U.Top\n\n\u0001\u0003\u0002\nwith priority\nremoves\n\u0001\u0013\f\u0014\u0002\n\n\u001a\u001b\u0001\u0013\u001c\u001d\u0001\u0003\f\u0014\u0002\u0010\r\u0010\u001e \u001f!\f\u000e\u0001\u0003\f\"\u0002#\u0002%$&\u001f'\u0001\u0003\f\u0014(\t)!*\"*\u0014+\u0010,%-.\r\u0010\f\u0014\u0002\u0010\u0007/\u0017\u0019\u0018\n\n\u001a\u001b\u0001\u0013\u001c\u001d\u0001\u0003\f\u0014\u0002\t\r\u0010\u001e\u000e\u001f!\f\u000e\u0001\u0013\f\u0014\u0002#\u00020\n\n;\n\n\u0001\u0013R \u0001\u0003C@\r#S'\r\t\f\n\n\u0002'$\u000b\u001c\u001d\u0001\u0003\f\n\n;\n\n\u0002T\u0002\n\n)\u001d(\u0010(\n\n)'Q\n\nF\u001bP\n;\n\n\u0001\u0003\f\u0014(\u0010)!*\"*\"+\u0012,\u001b-\"\u0002\n\n\u001e \u001f!\f\u000e\u0001\u0003\f\"(\u0010)!*\"*\"+\u0012,\u001b-\"\u00021D\n\n2\b\u001c\u001b\u0001\u0013\f\u0014(\u0010)!*B*\u0014+\u0012,\u001b-\"\u0002#\u0002\n\nis locally overconsistent */\n\n)\u001dJ\n\n>!F\u001dGIH\nCalculateKey\n\n\u0017\u0019K\"LNM\u0010O\n\u0001\u0003C%\u0002#\u0002\nOR\n\nUpdateState\n\n;\n\n\u0001\u0003\f\u0014\u0002\n\n;\n\n;\n\n\u0001\u0013\f\n\n;\n\n;\n\n\u0001\u0004\u0002\n\n/*\n\n\u0002#\u0002\n\n\u0001\u0013\f\u0014\u0002\n\nif\nif\nif\n\nreturn\n\nelse /*\n\n<\u0014=\"> ?\n\nCalculateKey\n\nwhile\n\u0016XCY2\nif\n\nU.TopKey\nU.Pop\n;\n\u0001\u0004\u0002\n\n2\b\f\n\u0001\u0013CED\n\u0001\u0013CU6\u000bI\u0002\n\u001c\u001b\u0001\u0003C%\u00021D\n\n\u0001\u0013\u001c\u001d\u0001\u0003C%\u0002IZ3\u001e \u001f!\f\u000e\u0001\u0003C%\u0002#\u0002\n;\n\u001c\u001d\u0001\u0003C%\u0002;2\b\u001e \u001f!\f\u000e\u0001\u0003C%\u0002\nUpdateState\nfor all\n\f968[1\u001e.\\B]\u001d\u0001\u0003C'\u0002\nis locally underconsistent */\n\n\u0017\u0019\u0018\n;\n\f\u00196879\u001e \u001f\u001d\f \u0001\u0003\f\u0014\u0002:23\u001c\u001b\u0001\u0013\f\u0014\u0002;2\b\u0006\n<\u0014=B>\u000e?\n\u0001\u0003\f\n\nprocedure CalculateKey\n\u0015 01\nprocedure Initialize\n\u0015 02\n\u001613254\n\u0015 03\nfor all\n\u0015 04\n\u0016;\u001e \u001f!\f\u000e\u0001\u0003\f\u0014<\u0014=\"> ?/\u0002@25A\n\u0015 05\nU.Insert\nprocedure UpdateState\n\u0001\u0003C%\u0002\n\u0015 06\n<\u0014=\"> ?\n\u0002\u001d\u001e \u001f!\f\u000e\u0001\u0003C%\u0002@2\b\u0017\u0019\u0018\n\u0015 07\nU.Remove\n\u0001\u0003C%\u0002\n\u0015 08\nU.Insert\n2\b\u001e \u001f!\f\u000e\u0001\u0003C%\u0002#\u0002\n\u0001\u0013C@\r\nprocedure ComputePlan\n\u0001\u0004\u0002\n\u0015 09\nW CalculateKey\n\u0001\u0004\u0002:V\n\u0015 10\n\u0015 11\n\u0015 12\n\u0015 13\n\u0015 14\n\u0015 15\n\u0015 16\nprocedure Main()\n\u0015 17\n(\t)!*\"*\u0014+\u0012,\u001b-\n\u0016;\f\n2\b\f\nInitialize\n\u0015 18\n\u0001\u0004\u0002\n\u0015 19\nComputePlan\n;\n\u0001\u0004\u0002\n\u0015 20\nwhile\n(\u0010)!*\"*\"+\u0012,\u001b-\n\u0001\u0013\f\n\u0015 21\n/* if\n\u0001\u0003\u001e \u001f!\f\u000e\u0001\u0003\f\u0014(\t)!*\"*\u0014+\u0012,\u001b-\"\u0002:25\u0006&\u0002\n\u0015 22\nExecute\nK\u0014_T`a\u0017\u0019\u0018\n\u0015 23\nSet\n\f\u0014(\t)!*\"*\u0014+\u0010,%-\nScan for changed action costs;\n\u0015 24\n\u0015 25\nif any action costs have changed\nfor all actions with changed action costs\n\u0015 26\n\u0015 27\n\u0015 28\n\u0015 29\n\u0015 30\n\u0015 31\n\n\u0001\u0003C%\u0002\nCalculateKey\n;\n\nUpdate the action cost\nUpdateState\n\nR\u000e\u0001\u0013C@\r/S%\r\tcN\u0002\n;\n\n\f968[1\u001e.\\B]\u001d\u0001\u0003C'\u0002%^\n\n\f96\u000b\nU.Update\n\u0001\u0013\f\u000e\r\nComputePlan\n\u0001\u0004\u0002\n\n\u001c\u001d\u0001\u0003C%\u0002;25\u0006\nfor all\n\n(\u0010)!*\"*\"+\u0012,\u001b-\n\n\u0001\u0013\f\u0014\u0002#\u0002\n\nfor all\n\n\u0001\u0003\f\u0014\u0002\n\nC'\u0016\n\n*\"-\n\n;\n\n;\n\n;\n\n;\n\n;\n\n;\n\n<\u0014=\"> ?\n\n23\f\nthen the agent is not guaranteed to reach\n\u0001\u0013R \u0001\u0003\f\n>NF\u001bGaH\nto the current state of the agent after the action execution;\n\n\u0017\u0019K\"L\n\nF\u001bP\n\n)\u001d(\u0010(\n\n(\t)!*\"*\u0014+\u0010,%-\n\nwith \ufb01nite plan-execution cost */\n\n\f.<\u0014=\"> ?\n(\t)!*\"*\u0014+\u0010,%-\n\n;\n\n\tS'\r\t\f\u0014\u0002%$E\u001c\u001d\u0001\u0003\f\u0014\u0002T\u0002\n\nR\u000e\u0001\u0003Cb\r\tS'\r\tc\u0011\u0002\n\nFigure 3: Minimax LPA*\n\n\u0001\b\u0003\n\nto \n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\nexecution cost from \n\u0007 . It is therefore important to make the searches fast.\nIn the next sections, we describe Minimax LPA* and how to implement the parti-game al-\ngorithm with it. Figures 1(b), (c), (d) and (f) show the state spaces for our example directly\nafter the parti-game algorithm has used Minimax LPA* to determine the minimax goal\n\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\r\b\u001f\n . All expanded states (that is, all states whose minimax goal distances\ndistance of \nhave been computed) are shown in gray. Minimax LPA* speeds up the searches by reusing\ninformation from previous searches, which is the reason why it expands only three states in\nFigure 1(d). Minimax LPA* also speeds up the searches by using heuristics to focus them,\nwhich is the reason why it expands only four states in Figure 1(f).\n\n3 Minimax LPA*\n\n\u0001\u0004\u0003\n\nto \n\n\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\r\b\u001f\n\nMinimax LPA* repeatedly determines plans that minimize the worst-case plan-execution\nin nondeterministic domains\ncost from \nwhere the costs of actions increase or decrease over time. It generalizes two incremental\nsearch algorithms, namely our LPA* [1] and DynamicSWSF-FP [7]. Figure 3 shows the\nalgorithm, that we describe in the following. Numbers in curly braces refer to the line\nnumbers in the \ufb01gure.\n\n\u0007 as the agent moves towards \n\n\u0001\b\u0003\n\n\n\n\n\n\u0005\n\n\f\n\n\u000f\n\f\n\n\u000f\n\f\n\u000f\n\f\n\n\u0016\n\u0005\n\u0016\n\u0016\n\n\u0016\n\u001a\nH\n>\nJ\nO\nO\n\u0016\n\u0016\n\u0001\n\u0016\n\u0001\n\u0016\nC\n\u0016\n\u0016\n\u0016\nC\n\u0016\n\u0016\n\u0015\nM\n-\n>\n\u0016\n\u0016\n\u0016\nD\n\u0002\n\u0016\n\u0016\n\u001a\nM\nJ\nM\nH\nM\nQ\n>\nJ\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\u0005\n\u0005\n\u0005\n\u0007\n\f\u0001\u0004\u0003\u0006\u0005\n\nis the start state, and \n\n3.1 Notation\n denotes the \ufb01nite set of states.\n\u0004\u000b\n\u0004\u0003\n\u0002\u0001\u0003\n\nis the set of actions that can be executed in \u0006\u0003\nstate. \u0005\n\u0005\u0013\nof successor states that can result from the execution of \u000e\u0012\u0003\n\u0014\u0007\n\t\u001c\t\n\u001b\u001a\n\u0018\u0017\u0019\u0003\n\u0018\u0017\u0019\u0003\n\r\f\u000f\u000e\u0010\u0013 for some \u000e\u001d\u0003\n\u001b\u001a\n!\u0011\"\u0002#\n\u0013'\u001e\n\u0018\u0017\n$\u0003\n\u0018\u0017\b\u0003\n\u000f\u0012\u0011\nof (\u0003\n. The agent incurs cost )+*\nin 0\u0003\nresults in \u0018\u00171\u0003\n\u000e\u0010\u000f\u001b\u0011\n\u0013;D\n\u0018\u0017\n3?>\u0002@\n3\u0019465\n\u0001CB\n7\rA\n\u0005\u0018798;:\n\u0002\u0005\r\n\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\t\b\u0001\n\n\u000e\u0010\u000f\u0012\u0011\n\n\u0014\u0007\u0015\t\u000b\t\nis the set of successor states of  \u0003\n\u0005\u0013\u001f\u001e\n\u0018\u0017%\f\u000f\u000e\u0010\u0013 for some \u000e&\u0003\n\r\f\u000f\u000e\n\f\n,\u0017\n\u0013.-\nif the execution of \u000e/\u0003\nis the minimax goal distance of 0\u0003\n\u0007 , and\n\u0005\u0013\n\u0005\u0013\n\u000e\u001a\u000f\u001b\u0011\n\u000e\u001a\u000f\u001b\u0011\n with &F\n\u0018\u0017\n\u0013\u000b\u0013 for all E\u0003\n\u0007 . \u0005\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\nis\n\u0001\u0004\u0003\nthe current state of the agent, and the minimal worst-case plan-execution cost from \n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\nto \n\n.\nis the set of predecessor states\n\u0005\u0013\n\nde\ufb01ned as the solution of the system of equations:\n\n\r\f\u000f\u000e\u0010\u0013\u0011\u0010\nin \u0013\u0003\n\nis the goal\nis the set\n\n. \b\u0007\n\t\u000b\t\n\u0005\u0013\n\n\u0005\u00132-\n\u000e\u0010\u000f\u0012\u0011\n\n\u0014\u0007\u0015\t\u000b\t\n\n\r\f\u000f\u000e\n\f\n\n\u0007 is\n\n\u0013 .\n\nif \n\n\u0005\u0013\n\n\u0001=<\n\n\u0001\u0004\u0003\u0006\u0005\n\n\u0001\u0004\u0003\u0006\u0005\n\n.\n\n.\n\n,\n\n3.2 Heuristics and Variables\n\n\u0001\u0004\u0003\u0006\u0005\b\u0007\n\n\u0018\u0017\n\n\u0018\u0017\n\n\u0018\u0017\n\n,\u0017\n\n\u0017;\u0003\n\n\u0013;D\n\n\u0018\u0017K\f\n\n\u0014\u0007\u0015\t\u000b\t\n\nto \u0005\u0003\u0002\n\nand \u000e2\u0003\n\n\u0013 for all \r\f\n\r\f\n\n\r\f\n\u0005\u0013\n\u0013 with \u0018\u0017;\u0003\n\nMinimax LPA* searches backward from \n\nsearch. The heuristics need to be non-negative and satisfy G\n\r\f\n\u0018\u0017\n\u0017J\f\u000f\u000e\n\f\nwords, the heuristics G\n\n) and G\nL\u0017\n\u0013 approximate the best-case plan-execution cost from \n\n\u0004\u0006\u0004\f\u0007\r\b\u0001\n and uses heuristics to focus its\n\u0013I-\n9\f\nH\u0017\n\u0017K\f\u000f\u000e\u0010\u0013 . In other\n\u0018\u0017\nto \u0002\u0017 .\n\nMinimax LPA* maintains two variables for each state that it encounters during the search.\nThe g-value of a state estimates its minimax goal distance.\nIt is carried forward from\none search to the next one and can be used after each search to determine a plan that\n\u0007 . The rhs-value of\nminimizes the worst-case plan-execution cost from \na state also estimates its minimax goal distance. It is a one-step lookahead value based\non the g-values of its successors and thus potentially better informed than its g-value. It\n\u0007 , and\nalways satis\ufb01es the following relationship (Invariant 1):\n\u0007 . A\n\u0002\u0001\r\nstate is called locally consistent iff its g-value is equal to its rhs-value. Minimax LPA* also\nthat always contains exactly the locally inconsistent states\n(Invariant 2). Their priorities are always identical to their current keys (Invariant 3), where\n\n\u0005\u0013\u001c\u0015\n\u0013\u000b\u0013 for all \u0019\u0003\n\n with MF\n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n9\fC\u000e\n\f\n\n\u0001\u0004\u0003\u0006\u0005\n\u0002\u0001\b\u0003\n\n\u0013;D\n\nto \n\nif \n\n\u0018\u0017\n\n\u0018\u0017\n\n\u0001CB\n\n\u0001\u0004\u0003\n\n\u0001=<\n\n7\rA\n\n\u0005,798;:\n\n\u0015\u001d3\u0019465\n\n3?>\u0002@\nmaintains a priority queue N\nis the pair P\nthe key O\n\n\u0005\u0013 of \n\ncalculated by CalculateKey(). The keys are compared according to a lexicographic order-\ning.\n\n3\u00194Q5\n\n\u0013\u000b\f\n\n\u0005\u0013\"\u0013RD\n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u0013\u000bST3\u00194Q5\n\n\u0005\u0013\u001f\f\n\n\u0005\u0013\u000b\u0013VU , as\n\n3.3 Algorithm\n\n\u0001\u0004\u0003\n\nis the only locally inconsistent state\n\nMinimax LPA* operates as follows. The main function Main() \ufb01rst calls Initialize() \u0016 18\u001e\n\u0016 03\u001e . The only exception is the rhs-\nto set the g-values and rhs-values of all states to \u001d\n\u0007 , that is set to zero \u0016 04\u001e . Thus, \u0002\u0001\u0004\u0003\nvalue of \nand is inserted into the otherwise empty priority queue \u0016 02, 05\u001e . (Note that, in an actual\n\n\u0007 yet \u0016 20\u001e , it executes the \ufb01rst action of the plan \u0016 22\u001e and updates \u001f\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\r\b\u001f\n\nimplementation, Minimax LPA* needs to initialize a state only once it encounters it during\nthe search and thus does not need to initialize all states up front. This is important because\nthe number of states can be large and only a few of them might be reached during the\nsearch.) Then, Minimax LPA* calls ComputePlan() to compute a plan that minimizes the\n\u0016 19\u001e . If the agent has not reached\nworst-case plan-execution cost from \u001f\u0003\u0002\n\u0016 23\u001e .\nIt then scans for changed action costs \u0016 24\u001e . To maintain Invariants 1, 2, and 3, it calls\nUpdateState() if some action costs have changed \u0016 28\u001e\nto update the rhs-values and keys of\nthe states potentially affected by the changed action costs as well as their membership in\nthe priority queue if they become locally consistent or inconsistent. It then recalculates the\npriorities of all states in the priority queue \u0016 29-30\u001e . This is necessary because the heuristics\n\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\t\b\u0001\n . This only\n\nchange when the agent moves, since they are computed with respect to \n\n\u0004\u0006\u0004\f\u0007\r\b\u0001\n\nto \n\n\u0001\u0004\u0003\u0006\u0005\b\u0007\n\n\u0001\u0004\u0003\u0006\u0005\n\n\u0005\n\n\u0007\n\u0003\n\n\u0011\n\n\u0011\n\n\u0005\n\u0011\n\n\u0011\n\u0015\n\u0016\n\u0011\n\u0005\n\u0011\n\n\n\u0013\n\u0015\n\u0016\n\u0011\n\u0005\n\u0011\n\n\t\n\u0011\n\u001d\n\u0005\n\u0011\n\n\n)\n-\n\u001d\n\n\u0015\n)\n\u0015\n\n\u0015\n\u0001\nO\n:\n\u0005\n<\n\u0011\n\t\n\u0011\n\u0015\n\n\u0005\n\n\u0011\n\u0015\n\u0011\nG\n\u0011\n\u0017\n\t\n\u0011\n\n\u0005\n\u0011\n\u0017\n\u0011\n\u0011\n\u0005\n\"\nG\n\n\u0011\n)\n\u0015\n\n\"\nG\n\n\u0011\n\n\u0013\n\u0001\nO\n:\n\u0005\n<\n\u0011\n\t\n\u0011\n\u000e\n\u0011\n\u0015\n\u0005\n\u0011\n\u0011\n\u000e\n\u0011\n\n\"\nG\n\n\u0011\nG\n\u0011\n\n\f\n\n\u0011\n\u000e\n\u0011\n\"\nG\n\n\u0011\n\u0005\n\u0005\n\u0007\n\n\f\u0016:\f\u0014(\u0010)!*\"*\"+\u0012,\u001b-12\b\f\nwhile (\n\n\f\u0014(\u0010)!*\"*\"+\u0012,\u001b-9D\n\n;\n\n*B-\n2\b\f\u0014<\u0014=\"> ?\n\n)\n\nAND\n;\n\n(\t)!*\"*\u0014+\u0010,%-\n\nRe\ufb01ne the discretization, if possible (initially: construct the \ufb01rst discretization);\nConstruct the state space that corresponds to the current discretization;\nInitialize();\nComputePlan();\nif (\n\u001e \u001f!\f\u000e\u0001\u0003\f\u0014(\t)!*\"*\u0014+\u0010,%-\u0012\u0002;2\b\u0006\nwhile (\n\f\u0001\n*\u0014+\u0001\u0002\u0004\u0003\nExecute\nSet\n(\u0010)!*\"*\"+\u0012,\u001b-\nif\n\u0001\u0003\f\u0014(\t)!*\"*\u0014+\u0010,%-\n*\u0014+\u0001\u0002\u0004\u0003\n7bC\u001dRBR \u0001\u0003\f\n*\u0014+\u0001\u0002\u0004\u0003\n7bC\u001dRBR \u0001\u0003\f\n[1\u001e.\\B]\u001d\u0001\u0003\f\u0014(\u0010)!*B*\u0014+\u0012,\u001b-\"\u0002:25[1\u001e \\\u0012]\u001b\u0001\u0003\f\"(\u0010)!*\"*\"+\u0012,\u001b-\"\u0002'^\nUpdateState(\nfor all\nU.Update(\n\n<\u0014=\">\u000e?\n2\b\f\u0014(\t)!*\"*\u0014+\u0012,\u001b-\nto the new state of the agent after the action execution;\n687bC\u001dRBR \u0001\u0003\f\n\n) stop with no solution;\n(\t)!*\"*\u0014+\u0012,\u001b-\n\u0017\u0019K\"L\n\n(\t)!*\"*\u0014+\u0010,%-\n\r/S\u001d\u0002#\u0002\n*\u0014+\u0005\u0002\u0006\u0003\n*\u0014+\u0001\u0002\u0004\u0003\n\n*\u0014+\u0001\u0002\u0004\u0003\n\r\tS!\u0002:2\b7@C\u001dRBR \u0001\u0003\f\n\u0002:257bC\u001dRBR\u000e\u0001\u0013\f\n\n2\b\f\n25K\u0014_T`;\u0017\u0019\u0018\n\n*\u0014+\u0005\u0002\u0006\u0003\n, CalculateKey(\n\n(\t)!*\"*\u0014+\u0010,%-\n*\u0014+\u0001\u0002\u0004\u0003\n\n25\u0006\nF%P\n\n(\t)!*\"*\u0014+\u0010,%-\n\n(\t)!*\"*\u0014+\u0010,%-\n\n\u001e \u001f!\f\u000e\u0001\u0003\f\n\n\tS\u001d\u0002\u001b^\n\nF\u001bGIH\n\n)\u001d(\u0010(\n\n\u0002%^\n\n));\n\n;\n\n;\n\n);\n\n)\n\n;\n\nprocedure Main()\n\u0015 17\u2019\n\u0015 18\u2019\n\u0015 19\u2019\n\u0015 20\u2019\n\u0015 21\u2019\n\u0015 22\u2019\n\u0015 23\u2019\n\u0015 24\u2019\n\u0015 25\u2019\n\u0015 26\u2019\n\u0015 27\u2019\n\u0015 28\u2019\n\u0015 29\u2019\n\u0015 30\u2019\n\u0015 31\u2019\n\u0015 32\u2019\n\u0015 33\u2019\n\u0015 34\u2019\n\u0015 35\u2019\n\n\f96\n\nComputePlan();\n\n\u0001\u0013R\u000e\u0001\u0013\f\n\n(\u0010)!*B*\u0014+\u0012,\u001b-\n\n\tS\n\n\t\f\n\n\u0002'$\u000b\u001c\u001d\u0001\u0003\f\n\n\u0002#\u0002\n\n;\n\nFigure 4: Parti-game algorithm using Minimax LPA*\n\nchanges the priorities of the states in the priority queue but not which states are locally\n\nconsistent and thus in the priority queue. Finally, it recalculates a plan \u0016 31\u001e and repeats the\n\nprocess.\n\nComputePlan() operates as follows. It repeatedly removes the locally inconsistent state\n\nwith the smallest key from the priority queue \u0016 10\u001e and expands it \u0016 11-16\u001e . It distinguishes\n\ntwo cases. A state is called locally overconsistent iff its g-value is larger than it rhs-value.\nWe can prove that the rhs-value of a locally overconsistent state that is about to be expanded\nis equal to its minimax goal distance. ComputePlan() therefore sets the g-value of the state\n\nto its rhs-value \u0016 12\u001e . A state is called locally underconsistent iff its g-value is smaller than\nit rhs-value. In this case, ComputePlan() sets the g-value of the state to in\ufb01nity \u0016 15\u001e . In\neither case, ComputePlan() ensures that Invariants 1, 2 and 3 continue to hold \u0016 13, 16\u001e . It\n\nis locally consistent and its key is less than or equal to the\n\nterminates as soon as \nkeys of all locally inconsistent states.\n\n\u0004\u0006\u0004\f\u0007\r\b\u0001\n\n\u0003\u0002\n\nTheorem 1 ComputePlan\u0007\u0005\b of Minimax LPA* expands each state at most twice and thus terminates.\nand always executes an action\nAssume that, after ComputePlan() terminates, one starts in\nR\"C\u001b\u001e.\u001e.\\\u000b\n\r\f\n\u0007\u0019\u0018\r\u0007\nis\n\u000e\u0010\u000f\u0012\u0011\n\t\r\u001a\nS\u001d\u0002\nreached (ties can be broken arbitrarily). Then, the plan-execution cost is no larger than the minimax\ngoal distance of\n\n\b in the current state\n\nthat minimizes\n\n\b\"\b until\n\n7bC\u001bR\u0012R\u000e\u0001\u0003\f\u000e\n\n\b\u0017\u001d\u001f\u001e \u0007\n\n\u0014\u0010\u0015\u0017\u0016\n\n\u001c\u0006#.S!$\n\n\u000f\u0012\u0013\n\n\u001a\u0006\t\u001c\u001b\n\n\t!\u001b\n\n.\n\nR\"C\u001b\u001e \u001e.\\\u000b\n%\f\n\nWe can also prove several additional theorems about the ef\ufb01ciency of Minimax LPA*,\nincluding the fact that it only expands those states whose g-values are not already correct\n[5]. To reduce its search time, we optimize Minimax LPA* in several ways, for example, to\navoid unnecessary re-computations of the rhs-values [5]. We use these optimizations in the\nexperiments. A more detailed description, the intuition behind Minimax LPA*, examples\nof its operation, and additional theorems and their proofs can be found in [5].\n\n4 Using Minimax LPA* to Implement the Parti-Game Algorithm\n\nFigure 4 shows how Minimax LPA* can be used to implement the parti-game algorithm in a\nmore ef\ufb01cient way than with uninformed search from scratch, using some of the functions\nfrom Figure 3.\nInitially, the parti-game algorithm constructs a \ufb01rst (coarse) discretiza-\n\ntion of the terrain \u0016 19\u2019\u001e , constructs the corresponding state space (which includes setting\nto the state that includes the goal coordinates, and\n\u0005\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u0014\u0007\u0015\t\u000b\t\n\u0005\u0013 according to the optimistic assumption that each action\ndeterministically reaches the intended state) \u0016 20\u2019\u001e , and uses ComputePlan() to \ufb01nd a \ufb01rst\nplan from scratch \u0016 21\u2019-22\u2019\u001e . If the minimax goal distance of \nis in\ufb01nity, then it\n\nto the state of the agent, \n\u000f\u001b\u0011\n\n\u0005\u0013 , and !\u0011\"\u0002#\n\n9\fC\u000e\u0010\u0013 , \u0014\u0007\u0015\t\u000b\t\n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u0001\u0004\u0003\u0006\u0005\b\u0007\n\nM\n-\n>\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\u0016\n\f\nD\n\u0002\nD\n\u0016\n=\n)\nM\n\u0016\nS\n\u001a\n>\nO\nM\nJ\nM\nO\nH\nM\nQ\n>\nO\nJ\nO\nO\nO\n\u0016\n\f\n\u0016\nD\n\n=\n)\nM\n\u0016\n\n=\n)\nM\n\n=\n)\nM\n\u0015\n\f\n\u0016\n\u0016\n\n=\n)\nM\n\n=\n)\nM\n\u0015\n\f\n\u0016\n\u0016\n\u0015\n\f\n\n=\n)\nM\n\u0016\n\u0016\n\f\n\n=\n)\nM\n\u0016\n\n\u0016\n\f\n\f\n\u0016\n\t\n\u0007\n\t\n\t\n\f\nO\n6\n\u000e\n\t\n\t\n\u0011\n\u0011\n\f\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\nactions until either \n\nis unsolvable or the agent reaches \n\nstops unsuccessfully \u0016 23\u2019\u001e . Otherwise, it repeatedly executes the action that minimizes the\nworst-case plan-execution cost \u0016 26\u2019-27\u2019\u001e . If it observes an unknown action outcome \u0016 28\u2019\u001e ,\nthen it records it \u0016 29\u2019-31\u2019\u001e , ensures that Invariants 1, 2 and 3 continue to hold \u0016 32\u2019-34\u2019\u001e ,\nuses ComputePlan() to \ufb01nd a new plan incrementally \u0016 35\u2019\u001e , and then continues to execute\n\u0016 24\u2019\u001e . In the former\ncase, it re\ufb01nes the discretization \u0016 19\u2019\u001e , uses ComputePlan() to \ufb01nd a new plan from scratch\n\u0016 20\u2019-22\u2019\u001e , and then repeats the process.\n\nrather than incrementally (because the discretization changes the state space substantially)\n\nThe heuristic of a state in our version of the parti-game algorithm approximates the Eu-\nclidean distance from the center of the current cell of the agent to the center of the cell\nthat corresponds to the state in question. The resulting heuristics have the property that we\ndescribed in Section 3.2. Figures 1(b), (c), (d) and (f) show the heuristics, g-values and\nrhs-values of all states directly after the call to ComputePlan(). All expanded states are\nshown in gray, and all locally inconsistent states (that is, states in the priority queue) are\nshown in bold.\n\n\u0001\u0004\u0003\u0006\u0005\n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\nIt happens quite frequently that \nis unsolvable and the parti-game algorithm thus\nhas to re\ufb01ne the discretization. If \nis unsolvable, Minimax LPA* expands a large\nnumber of states because it has to disprove the existence of a plan rather than \ufb01nd one. We\nis unsolvable but every other\nspeed up Minimax LPA* for the special case where \u001f\u0003\u0002\n\u0004\u0006\u0004\f\u0007\r\b\u0001\n\nstate is solvable since it occurs about half of the time when \nis unsolvable. If states\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u0004\u0006\u0004\f\u0007\r\b\u0001\n .\nother than \nis unsolvable but every other state is solvable, Minimax LPA* can\nTo prove that \ntherefore show that all predecessors of \u0001\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\t\b\u0001\n are solvable but \u0005\u0003\u0002\u0005\u0004\u0006\u0004\u0006\u0007\t\b\u0001\n\nitself is not. To\n\u0004\u0006\u0004\f\u0007\r\b\u0001\n are solvable, Minimax LPA* checks that they are\nshow that all predecessors of \nlocally consistent, their keys are no larger than U.TopKey(), and their rhs-values are \ufb01nite.\nTo show that \u0001\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\nis in\ufb01nite. We use this optimization in the experiments.\n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n become unsolvable, some of them need to be predecessors of \n\nis unsolvable, Minimax LPA* checks that the rhs-value of \n\n\u0003\u0002\u0005\u0004\u000b\u0004\f\u0007\r\b\u0001\n\n\u0003\u0002\n\n\u0003\u0002\n\n5 Experimental Results\n\nAn implementation of the parti-game algorithm can use search from scratch or incremental\nsearch. It can also use uninformed search (using the zero heuristic) and informed search\n(using the heuristic that we used in the context of the example from Figure 1). We compare\nthe four resulting combinations. All of them use binary heaps to implement the priority\nqueue and the same optimizations but the implementations with search from scratch do not\ncontain any code needed only for incremental search. Since all implementations move the\nagent in the same way, we compare their number of state expansions, their total run times,\nand their total search times (that is, the part of the run times spent in the search routines),\naveraged over 25 two-dimensional terrains of size \u0017\ndensity, where the resolution limit is one cell. In each case, the goal coordinates are in the\ncenter of the terrain, and the start coordinates are in the vertical center and ten percent to\nthe right of the left edge. We also report the average of the ratios of the three measures for\neach of the four implementations and the one with incremental heuristic search (which is\ndifferent from the ratio of the averages), together with their 95-percent con\ufb01dence intervals.\n\n)9)\r) with 30 percent obstacle\n\n)9)\r)\u0001\n\nImplementation of Parti-\nGame Algorithm with . . . Expansions\nUninformed from Scratch 69,527,969 20.55\n8.06\nInformed from Scratch\n1.23\nUninformed Incremental\nInformed Incremental\n1.00\n\n31,303,253\n2,628,879\n2,172,430\n\nRatio\n\nRatio\n\nRun Time (Search Time)\n3.61)\n2.70)\n0.05)\n0.00)\n\n3.52 (15.29\n2.50 ( 7.20\n0.02 ( 1.19\n0.00 ( 1.00\n\nExpansions\n\nRun Time\n\n(Search Time)\n\n4.12 39 min 51 sec (37 min 43 sec) 11.83\n6.08\n2.59 22 min 58 sec (20 min 49 sec)\n1.04\n1 min 54 sec ( 1 min 41 sec)\n0.03\n0.00\n1 min 45 sec ( 1 min 28 sec)\n1.00\n\nThe average number of searches, measured by calls to ComputePlan(), is 29,885 until the\n\u0007 . The table shows that the search times of the parti-game algorithm\nagent reaches \u0002\u0001\b\u0003\n\n\u0007\n\u0017\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0002\n\u0005\n\fare substantial due to the large number of searches performed (even though each search is\nfast), and that the searches take up most of its run time. Thus, speeding up the searches is\nimportant. The table also shows that incremental and heuristic search individually speed\nup the parti-game algorithm and together speed it up even more.\n\nThe implementations of the parti-game algorithm in [3] and [6] make slightly different as-\nsumptions from ours, for example, minimize state transitions rather than cost. Al-Ansari\nreports that the original implementation of the parti-game algorithm with value iteration\nperforms about 80 percent and that his implementation with a simple uninformed incre-\nmental search method performs about 15 percent of the state expansions of the implemen-\ntation with uninformed search from scratch [6]. Our results show that our implementation\nwith Minimax LPA* performs about 5 percent of the state expansions of the implementa-\ntion with uninformed search from scratch. While these results are not directly comparable,\nwe have also \ufb01rst results where we ran the original implementation with value iteration\nand our implementation with Minimax LPA* on a very similar environment and the orig-\ninal implementation expanded one to two orders of magnitude more states than ours even\nthough its number of searches and its \ufb01nal number of states was smaller. However, these\nresults are very preliminary since the time per state expansion is different for the differ-\nent implementations and it is future work to compare the various implementations of the\nparti-game algorithm in a common testbed.\n\nReferences\n\n[1] S. Koenig and M. Likhachev. Incremental A*. In T. Dietterich, S. Becker, and Z. Ghahramani,\neditors, Advances in Neural Information Processing Systems 14, Cambridge, MA, 2002. MIT\nPress.\n\n[2] D. Frigioni, A. Marchetti-Spaccamela, and U. Nanni. Fully dynamic algorithms for maintaining\n\nshortest paths trees. Journal of Algorithms, 34(2):251\u2013281, 2000.\n\n[3] A. Moore and C. Atkeson. The parti-game algorithm for variable resolution reinforcement learn-\n\ning in multidimensional state-spaces. Machine Learning, 21(3):199\u2013233, 1995.\n\n[4] A. Moore and C. Atkeson. Prioritized sweeping: Reinforcement learning with less data and less\n\ntime. Machine Learning, 13(1):103\u2013130, 1993.\n\n[5] M. Likhachev and S. Koenig. Speeding up reinforcement learning with incremental heuristic\nminimax search. Technical Report GIT-COGSCI-2002/5, College of Computing, Georgia Insti-\ntute of Technology, Atlanta (Georgia), 2002.\n\n[6] M. Al-Ansari. Ef\ufb01cient Reinforcement Learning in Continuous Environments. PhD thesis, Col-\n\nlege of Computer Science, Northeastern University, Boston (Massachusetts), 2001.\n\n[7] G. Ramalingam and T. Reps. An incremental algorithm for a generalization of the shortest-path\n\nproblem. Journal of Algorithms, 21:267\u2013305, 1996.\n\n\f", "award": [], "sourceid": 2303, "authors": [{"given_name": "Maxim", "family_name": "Likhachev", "institution": null}, {"given_name": "Sven", "family_name": "Koenig", "institution": null}]}