{"title": "Exploration via Hindsight Goal Generation", "book": "Advances in Neural Information Processing Systems", "page_first": 13485, "page_last": 13496, "abstract": "Goal-oriented reinforcement learning has recently been a practical framework for robotic manipulation tasks, in which an agent is required to reach a certain goal defined by a function on the state space. However, the sparsity of such reward definition makes traditional reinforcement learning algorithms very inefficient. Hindsight Experience Replay (HER), a recent advance, has greatly improved sample efficiency and practical applicability for such problems. It exploits previous replays by constructing imaginary goals in a simple heuristic way, acting like an implicit curriculum to alleviate the challenge of sparse reward signal. In this paper, we introduce Hindsight Goal Generation (HGG), a novel algorithmic framework that generates valuable hindsight goals which are easy for an agent to achieve in the short term and are also potential for guiding the agent to reach the actual goal in the long term. We have extensively evaluated our goal generation algorithm on a number of robotic manipulation tasks and demonstrated substantially improvement over the original HER in terms of sample efficiency.", "full_text": "Exploration via Hindsight Goal Generation\n\nZhizhou Ren\u2020, Kefan Dong\u2020\n\nInstitute for Interdisciplinary Information Sciences, Tsinghua University\n\nDepartment of Computer Science, University of Illinois at Urbana-Champaign\n\n{rzz16, dkf16}@mails.tsinghua.edu.cn\n\nYuan Zhou\n\nDepartment of Industrial and Enterprise Systems Engineering\n\nUniversity of Illinois at Urbana-Champaign\n\nyuanz@illinois.edu\n\nQiang Liu\n\nDepartment of Computer Science\n\nUniversity of Texas at Austin\n\nlqiang@cs.utexas.edu\n\nJian Peng\n\nDepartment of Computer Science\n\nUniversity of Illinois at Urbana-Champaign\n\njianpeng@illinois.edu\n\nAbstract\n\nGoal-oriented reinforcement learning has recently been a practical framework for\nrobotic manipulation tasks, in which an agent is required to reach a certain goal\nde\ufb01ned by a function on the state space. However, the sparsity of such reward\nde\ufb01nition makes traditional reinforcement learning algorithms very inef\ufb01cient.\nHindsight Experience Replay (HER), a recent advance, has greatly improved\nsample ef\ufb01ciency and practical applicability for such problems. It exploits previous\nreplays by constructing imaginary goals in a simple heuristic way, acting like an\nimplicit curriculum to alleviate the challenge of sparse reward signal. In this paper,\nwe introduce Hindsight Goal Generation (HGG), a novel algorithmic framework\nthat generates valuable hindsight goals which are easy for an agent to achieve in\nthe short term and are also potential for guiding the agent to reach the actual goal\nin the long term. We have extensively evaluated our goal generation algorithm on a\nnumber of robotic manipulation tasks and demonstrated substantially improvement\nover the original HER in terms of sample ef\ufb01ciency.\n\n1\n\nIntroduction\n\nRecent advances in deep reinforcement learning (RL), including policy gradient methods (Schulman\net al., 2015, 2017) and Q-learning (Mnih et al., 2015), have demonstrated a large number of successful\napplications in solving hard sequential decision problems, including robotics (Levine et al., 2016),\ngames (Silver et al., 2016; Mnih et al., 2015), and recommendation systems (Karatzoglou et al., 2013),\namong others. To train a well-behaved policy, deep reinforcement learning algorithms use neural\nnetworks as functional approximators to learn a state-action value function or a policy distribution\nto optimize a long-term expected return. The convergence of the training process, particularly in\nQ-learning, is heavily dependent on the temporal pattern of the reward function (Szepesv\u00e1ri, 1998).\nFor example, if only a non-zero reward/return is provided at the end of an rollout of a trajectory\nwith length L, while no rewards are observed before the L-th time step, the Bellman updates of the\nQ-function would become very inef\ufb01cient, requiring at least L steps to propagate the \ufb01nal return to the\n\n\u2020Work done while Zhizhou and Kefan were visiting students at UIUC.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fQ-function of all earlier state-action pairs. Such sparse or episodic reward signals are ubiquitous in\nmany real-world problems, including complex games and robotic manipulation tasks (Andrychowicz\net al., 2017). Therefore, despite its notable success, the application of RL is still quite limited to\nreal-world problems, where the reward functions can be sparse and very hard to engineer (Ng et al.,\n1999). In practice, human experts need to design reward functions which would re\ufb02ect the task\nneeded to be solved and also be carefully shaped in a dense way for the optimization in RL algorithms\nto ensure good performance. However, the design of such dense reward functions is non-trivial in\nmost real-world problems with sparse rewards. For example, in goal-oriented robotics tasks, an\nagent is required to reach some state satisfying prede\ufb01ned conditions or within a state set of interest.\nMany previous efforts have shown that the sparse indicator rewards, instead of the engineered dense\nrewards, often provide better practical performance when trained with deep Q-learning and policy\noptimization algorithms (Andrychowicz et al., 2017). In this paper, we will focus on improving\ntraining and exploration for goal-oriented RL problems.\nA notable advance is called Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), which\ngreatly improves the practical success of off-policy deep Q-learning for goal-oriented RL problems,\nincluding several dif\ufb01cult robotic manipulation tasks. The key idea of HER is to revisit previous\nstates in the experience replay and construct a number of achieved hindsight goals based on these\nvisited intermediate states. Then the hindsight goals and the related trajectories are used to train\nan universal value function parameterized by a goal input by algorithms such as deep deterministic\npolicy gradient (DDPG, Lillicrap et al. (2016)). A good way to think of the success of HER is to\nview HER as an implicit curriculum which \ufb01rst learns with the intermediate goals that are easy to\nachieve using current value function and then later with the more dif\ufb01cult goals that are closer to the\n\ufb01nal goal. A notable difference between HER and curriculum learning is that HER does not require\nan explicit distribution of the initial environment states, which appears to be more applicable to many\nreal problems.\nIn this paper, we study the problem of automatically generating valuable hindsight goals which are\nmore effective for exploration. Different from the random curriculum heuristics used in the original\nHER, where a goal is drawn as an achieved state in a random trajectory, we propose a new approach\nthat \ufb01nds intermediate goals that are easy to achieve in the short term and also would likely lead\nto reach the \ufb01nal goal in the long term. To do so, we \ufb01rst approximate the value function of the\nactual goal distribution by a lower bound that decomposes into two terms, a value function based on\na hindsight goal distribution and the Wasserstein distance between the two distributions. Then, we\nintroduce an ef\ufb01cient discrete Wasserstein Barycenter solver to generate a set of hindsight goals that\noptimizes the lower bound. Finally, such goals are used for exploration.\nIn the experiments, we evaluate our Hindsight Goal Generation approach on a broad set of robotic\nmanipulation tasks. By incorporating the hindsight goals, a signi\ufb01cant improvement on sample\nef\ufb01ciency is demonstrated over DDPG+HER. Ablation studies show that our exploration strategy is\nrobust across a wide set of hyper-parameters.\n\n2 Background\n\nReinforcement Learning The goal of reinforcement learning agent is to interact with a given\nenvironment and maximize its expected cumulative reward. The environment is usually modeled by a\nMarkov Decision Process (MDP), given by tuples (cid:104)S,A, P, R, \u03b3(cid:105) , where S,A represent the set of\nstates and actions respectively. P : S \u00d7 A \u2192 S is the transition function and R : S \u00d7 A \u2192 [0, 1]\nis the reward function. \u03b3 is the discount factor. The agent trys to \ufb01nd a policy \u03c0 : S \u2192 A that\nmaximizes its expected curriculum reward V \u03c0(s0), where s0 = s is usually given or drawn from a\ndistribution \u00b50 of initial state. The value function V \u03c0(s) is de\ufb01ned as\n\nV \u03c0(s) = Es0=s,at\u223c\u03c0(\u00b7|st),st+1\u223cP (\u00b7|st,at)\n\n\u03b3tR(st, at)\n\n.\n\n(cid:35)\n\n(cid:34) \u221e(cid:88)\n\nt=0\n\nGoal-oriented MDP In this paper, we consider a speci\ufb01c class of MDP called goal-oriented MDP.\nWe use G to denote the set of goals. Different from traditional MDP, the reward function R is a\ngoal-conditioned sparse and binary signal indicating whether the goal is achieved:\n\nRg(st, at, st+1) :=\n\n(cid:107)\u03c6(st+1) \u2212 g(cid:107)2 \u2264 \u03b4g\n\notherwise.\n\n(1)\n\n(cid:26) 0,\n\n\u22121,\n\n2\n\n\f\u03c6 : S \u2192 G is a known and tractable mapping that de\ufb01nes goal representation. \u03b4g is a given threshold\nindicating whether the goal is considered to be reached (see Plappert et al. (2018)).\nUniversal value function The idea of universal value function is to use a single functional approxi-\nmator, such as neural networks, to represent a large number of value functions. For the goal-oriented\nMDPs, the goal-based value function of a policy \u03c0 for any given goal g is de\ufb01ned as V \u03c0(s, g), for all\nstate s \u2208 S. That is\n\n(cid:34) \u221e(cid:88)\n\n(cid:35)\n\nV \u03c0(s, g) := Es0=s,at\u223c\u03c0(\u00b7|st,g),st+1\u223cP (\u00b7|st,at)\n\n\u03b3tRg(st, at, st+1)\n\n.\n\n(2)\n\nt=0\n\nLet T \u2217 : S \u00d7 G \u2192 [0, 1] be the joint distribution over starting state s0 \u2208 S and goal g \u2208 G.. That is,\nat the start of every episode, a state-goal pair (s0, g) will be drawn from the task distribution T \u2217. The\nagent tries to \ufb01nd a policy \u03c0 : S \u00d7 G \u2192 A that maximizes the expectation of discounted cumulative\nreward\n\nV \u03c0(T \u2217) :=\n\nE\n\n(s0,g)\u223cT \u2217\n\n[V \u03c0(s0, g)]\n\n(3)\n\nGoal-oriented MDP characterizes several reinforcement benchmark tasks, such as the robotics tasks\nin the OpenAI gym environment (Plappert et al., 2018). For example, in the FetchPush (see Figure 1)\ntask, the agent needs to learn pushing a box to a designated point. In this task, the state of the system\ns contains the status for both the robot and the box. The goal g, on the other hand, only indicates the\ndesignated position of the box. Thus, the mapping \u03c6 is de\ufb01ned as a mapping from a system state s to\nthe position of the box in s.\nAccess to Simulator One of the common assumption made by previous work is an universal simulator\nthat allows the environment to be reset to any given state (Florensa et al., 2017; Ecoffet et al., 2019).\nThis kind of simulator is excessively powerful, and hard to build when acting in the real world. On\nthe contrary, our method does not require an universal simulator, and thus is more realizable.\n\n3 Related Work\n\nMulti-Goal RL The role of goal-conditioned policy has been investigated widely in deep reinforce-\nment learning scenarios (Pong et al., 2019). A few examples include grasping skills in imitation\nlearning (Pathak et al., 2018; Srinivas et al., 2018), disentangling task knowledge from environment\n(Mao et al., 2018a; Ghosh et al., 2019), and constituting lower-level controller in hierarchical RL\n(Oh et al., 2017; Nachum et al., 2018; Huang et al., 2019; Eysenbach et al., 2019). By learning a\nuniversal value function which parameterizes the goal using a function approximator (Schaul et al.,\n2015), an agent is able to learn multiple tasks simultaneously (Kaelbling, 1993; Veeriah et al., 2018)\nand identify important decision states (Goyal et al., 2019b). It is shown that multi-task learning with\ngoal-conditioned policy improves the generalizability to unseen goals (e.g., Schaul et al. (2015)).\nHindsight Experience Replay Hindsight Experience Replay (Andrychowicz et al., 2017) is an\neffective experience replay strategy which generates reward signal from failure trajectories. The\nidea of hindsight experience replay can be extended to various goal-conditioned problems, such\nas hierarchical RL (Levy et al., 2019), dynamic goal pursuit (Fang et al., 2019a), goal-conditioned\nimitation (Ding et al., 2019; Sun et al., 2019) and visual robotics applications (Nair et al., 2018;\nSahni et al., 2019). It is also shown that hindsight experience replay can be combined with on-policy\nreinforcement learning algorithms by importance sampling (Rauber et al., 2019).\nCurriculum Learning in RL Curriculum learning in RL usually suggests using a sequence of\nauxiliary tasks to guide policy optimization, which is also related to multi-task learning, lifelong\nlearning, and transfer learning. The research interest in automatic curriculum design has seen rapid\ngrowth recently, where approaches have been proposed to schedule a given set of auxiliary tasks\n(Riedmiller et al., 2018; Colas et al., 2019), and to provide intrinsic motivation (Forestier et al.,\n2017; P\u00e9r\u00e9 et al., 2018; Sukhbaatar et al., 2018; Colas et al., 2018). Generating goals which leads to\nhigh-value states could substantially improve the sample ef\ufb01ciency of RL agent (Goyal et al., 2019a).\nGuided exploration through curriculum generation is also an active research topic, where either\nthe initial state (Florensa et al., 2017) or the goal position (Baranes and Oudeyer, 2013; Florensa\net al., 2018) is considered as a manipulable factor to generate the intermediate tasks. However, most\n\n3\n\n\fcurriculum learning methods are domain-speci\ufb01c, and it is still open to build a generalized framework\nfor curriculum learning.\n\n4 Automatic Hindsight Goal Generation\n\nAs discussed in the previous section, HER provides an effective\nsolution to resolve the sparse reward challenge in object manipu-\nlation tasks, in which achieved state in some past trajectories will\nbe replayed as imaginary goals. In the other words, HER modi\ufb01es\nthe task distribution in replay buffer to generate a set of auxiliary\nnearby goals which can used for further exploration and improve\nthe performance of an off-policy RL agent which is expected to\nreach a very distant goal. However, the distribution of hindsight\ngoals where the policy is trained on might differ signi\ufb01cantly from\nthe original task or goal distribution. Take Figure 1 as an example,\nthe desired goal distribution is lying on the red segment, which is\nfar away from the initial position. In this situation, those hindsight\ngoals may not be effective enough to promote policy optimization in original task. The goal of\nour work is to develop a new approach to generate valuable hindsight goals that will improve the\nperformance on the original task.\nIn the rest of this section, we will present a new algorithmic framework as well as our implementation\nfor automatic hindsight goal generation for better exploration.\n\nFigure 1: Visualization of hind-\nsight goals (pink particles).\n\n4.1 Algorithmic Framework\n\nFollowing Florensa et al. (2018), our approach relies on the following generalizability assumption.\nAssumption 1. A value function of a policy \u03c0 for a speci\ufb01c goal g has some generalizability to\nanother goal g(cid:48) close to g.\nOne possible mathematical characterization for Assumption 1 is via the Lipschitz continuity. Similar\nassumptions have been widely applied in many scenarios (Asadi et al., 2018; Luo et al., 2019):\n\n|V \u03c0(s, g) \u2212 V \u03c0(s(cid:48), g(cid:48))| \u2264 L \u00b7 d((s, g), (s(cid:48), g(cid:48))),\n\n(4)\n\nwhere d((s, g), (s(cid:48), g(cid:48))) is a metric de\ufb01ned by\n\nd((s, g), (s(cid:48), g(cid:48))) = c(cid:107)\u03c6(s) \u2212 \u03c6(s(cid:48))(cid:107)2 + (cid:107)g \u2212 g(cid:48)(cid:107)2.\n\n(5)\nfor some hyperparameter c > 0 that provides a trade-off between the distances between initial states\nand the distance between \ufb01nal goals. \u03c6(\u00b7) is a state abstraction to map from the state space to the goal\nspace. When experimenting with the tasks in the OpenAI Gym environment (Plappert et al., 2018),\nwe simply adopt the state-goal mappings as de\ufb01ned in (1). Although the Lipschitz continuity may\nnot hold for every s, s(cid:48) \u2208 S, g, g(cid:48) \u2208 G, we only require continuity over some speci\ufb01c region. It is\nreasonable to claim that bound Eq. (4) holds for most of the (s, g), (s(cid:48), g(cid:48)) when d((s, g), (s(cid:48), g(cid:48))) is\nnot too large.\nPartly due to the reward sparsity of the distant goals, optimizing the expected cumulative reward\n(see Eq. (3)) from scratch is very dif\ufb01cult. Instead, we propose to optimize a relaxed lower bound\nwhich introduces intermediate goals that may be easier to optimize. Here we provide Theorem 1 that\nestablishes the such a lower bound.\nTheorem 1. Assuming that the generalizability condition (Eq. (4)) holds for two distributions\n(s, g) \u223c T and (s(cid:48), g(cid:48)) \u223c T (cid:48), we have\n\nV \u03c0(T (cid:48)) \u2265 V \u03c0(T ) \u2212 L \u00b7 D(T ,T (cid:48)).\n\n(6)\n\nwhere D(\u00b7,\u00b7) is the Wasserstein distance based on d(\u00b7,\u00b7)\n\nD(T (1),T (2)) =\n\ninf\n\n(1), g(1)), (s0\nwhere \u0393(T (1),T (2)) denotes the collection of all joint distribution \u00b5(s0\nmarginal probabilities are T (1),T (2), respectively.\n\n\u00b5\u2208\u0393(T (1),T (2))\n\n(cid:16)E\u00b5[d((s0\n\n(cid:17)\n\n(2), g(2)))]\n\n(1), g(1), s0\n\n(2), g(2)) whose\n\n4\n\n\fThe proof of Theorem 1 is deferred to Appendix A.\nIt follows from Theorem 1 that optimizing cumulative rewards Eq. (3) can be relaxed into the\nfollowing surrogate problem\n\nV \u03c0(T ) \u2212 L \u00b7 D(T ,T \u2217).\n\nmaxT ,\u03c0\n\n(7)\n\nNote that this new objective function is very intuitive. Instead of optimizing with the dif\ufb01cult goal/task\ndistribution T \u2217, we hope to \ufb01nd a collection of surrogate goals T , which are both easy to optimize\nand are also close or converging towards T \u2217. However the joint optimization of \u03c0 and T is non-trivial.\nThis is because a) T is a high-dimensional distribution over tasks, b) policy \u03c0 is optimized with\nrespect to a shifting task distribution T , c) the estimation of value function V \u03c0(T ) may not be quite\naccurate during training.\nInspired by Andrychowicz et al. (2017), we adopt the idea of using hindsight goals here. We \ufb01rst\nenforce T to be a \ufb01nite set of K particles which can only be from those already achieved states/goals\nfrom the replay buffer B. In another word, the support of the set T should lie inside B. In the\nmeanwhile, we notice that a direct implementation of problem Eq. (7) may lead to degeneration\nof hindsight goal selection of the training process, i.e., the goals may be all drawn from a single\ntrajectory, thus not being able to provide suf\ufb01cient exploration. Therefore, we introduce an extra\ndiversity constraint, i.e, for every trajectory \u03c4 \u2208 B, at most \u00b5 states can be selected in T . In practice,\nwe \ufb01nd that simply setting it to 1 would result in reasonable performance. It is shown in Section 5.3\nthat this diversity constraint indeed improves the robustness of our algorithm.\nFinally, the optimization problem we aim to solve is,\n\nmax\n\n\u03c0,T :|T |=K\n\nV \u03c0(T ) \u2212 L \u00b7 D(T ,T \u2217)\n\ns.t. (cid:88)\n(cid:88)\n\ns0,st\u2208\u03c4\n\n(cid:88)\n\n\u03c4\u2208B\n\ns0,st\u2208\u03c4\n\n1[(s0, \u03c6(st)) \u2208 T ] \u2264 1,\n\n\u2200\u03c4 \u2208 B\n\n1[(s0, \u03c6(st)) \u2208 T ] = K.\n\nTo solve the above optimization, we adapt a two-stage iterative algorithm. First, we apply a policy\noptimization algorithm, for example DDPG, to maximize the value function conditioned on the\ntask set T . Then we \ufb01x \u03c0 and optimize the the hindsight set T subject to the diversity constraint,\nwhich is a variant of the well-known Wasserstein Barycenter problem with a bias term (the value\nfunction) for each particle. Then we iterate the above process until the policy achieves a desirable\nperformance or we reach a computation budget. It is not hard to see that the \ufb01rst optimization of\nvalue function is straightforward. In our work, we simply use the DDPG+HER framework for it.\nThe second optimization of hindsight goals is non-trivial. In the following, we describe an ef\ufb01cient\napproximation algorithm for it.\n\n4.2 Solving Wasserstein Barycenter Problem via Bipartite Matching\nSince we assume that T is hindsight and with K particles, we can approximately solve the above\nWasserstein Barycenter problem in the combinatorial setting as a bipartite matching problem. Instead\nof dealing with T \u2217, we draw K samples from T \u2217 to empirically approximate it by a set of K particles\n\n(cid:98)T \u2217. In this way, the hindsight task set T can be solved in the following way. For every task instance\n0, \u02c6gi) \u2208 (cid:98)T \u2217, we \ufb01nd a state trajectory \u03c4 i = {si\n(cid:88)\n0,\u02c6gi)\u2208(cid:98)T \u2217\n\nt} \u2208 B that together minimizes the sum\nw((\u02c6si\n\n0, \u02c6gi), \u03c4 i)\n\n(8)\n\n(\u02c6si\n\n(\u02c6si\n\nwhere we de\ufb01ne\n\n0, \u02c6gi), \u03c4 i) := c(cid:107)\u03c6(\u02c6si\n\n0) \u2212 \u03c6(si\n\n0)(cid:107)2 + min\n\nw((\u02c6si\n\n(9)\nFinally we select each corresponding achieved state st \u2208 \u03c4 to construct hindsight goal (\u02c6s0, \u03c6(st)) \u2208\nT . It is not hard to see that the above combinatorial optimization exactly identi\ufb01es optimal solution\n\n0, \u03c6(si\n\nV \u03c0(si\n\nt))\n\n.\n\nt)(cid:107)2 \u2212 1\nL\n\n(cid:107)\u02c6gi \u2212 \u03c6(si\n\nt\n\n(cid:19)\n\n(cid:18)\n\n5\n\n\fT in the above-mentioned Wasserstein Barycenter problem. In practice, the Lipschitz constant L is\nunknown and therefore treated as a hyper-parameter.\nThe optimal solution of the combinatorial problem in Eq. (8) can be solved ef\ufb01ciently by the well-\nknown maximum weight bipartite matching (Munkres, 1957; Duan and Su, 2012). The bipartite\ngraph G({Vx, Vy}, E) is constructed as follows. Vertices are split into two partitions Vx, Vy. Every\nvertex in Vx represents a task instance (\u02c6s0, \u02c6g) \u2208 \u02c6T \u2217, and vertex in Vy represents a trajectory \u03c4 \u2208 B.\nThe weight of edge connecting (\u02c6s0, \u02c6g) and \u03c4 is \u2212w((\u02c6s0, \u02c6g), \u03c4 ) as de\ufb01ned in Eq. (9). In this paper, we\napply the Minimum Cost Maximum Flow algorithm to solve this bipartite matching problem (for\nexample, see Ahuja et al. (1993)).\n\nSample {(\u02c6si\nFind K distinct trajectories {\u03c4 i}K\n\nAlgorithm 1 Exploration via Hindsight Goal Generation (HGG)\n1: Initialize \u03c0\n2: B \u2190 \u2205\n3: for iteration = 1, 2, . . . , N do\ni=1 \u223c T \u2217\n0, \u02c6gi)}K\n4:\n(cid:18)\nK(cid:88)\n5:\n(cid:32)\n\ni=1 that minimize\n0) \u2212 \u03c6(si\n0)(cid:107)2 + min\n(cid:18)\n\nConstruct intermediate task distribution {(\u02c6si\n\n0, \u02c6gi), \u03c4 i) =\n\nK(cid:88)\n\nc(cid:107)\u03c6(\u02c6si\n\n(cid:18)\n\ni=1\n\nw((\u02c6si\n\ni=1\n\ni=1 where\n\n0, gi)}M\nt)(cid:107)2 \u2212 1\nL\n\n(cid:107)\u02c6gi \u2212 \u03c6(si\n\nV \u03c0(si\n\n0, \u03c6(si\n\nt))\n\n(cid:19)(cid:33)\n\n(cid:46) initialize neural networks\n\n(cid:46) sample from target distribution\n(cid:46) weighted bipartite matching\n\n(cid:19)(cid:19)\n\n(cid:107)\u02c6gi \u2212 \u03c6(si\n\nt)(cid:107)2 \u2212 1\nL\n\nt\n\nV \u03c0(si\n\n0, \u03c6(si\n\nt))\n\n6:\n\n7:\n8:\n9:\n10:\n11:\n12:\n13:\n14:\n15:\n16:\n17:\n\ngi = \u03c6\n\narg min\n\nt\u2208\u03c4i\nsi\n\nfor i = 1, 2, . . . , K do\n0, gi)\n\n(s0, g) \u2190 (\u02c6si\nfor t = 0, 1, . . . , H \u2212 1 do\nat \u2190 \u03c0(\u00b7|st, g) + noise\nst+1 \u223c P (\u00b7|st, at)\nrt \u2190 Rg(st, at, st+1)\n\u03c4 \u2190 {s0, a0, r0, s1, . . .}\nB \u2190 B \u222a {\u03c4}\nfor i = 1 . . . M do\n\n(cid:46) critical step: hindsight goal-oriented exploration\n\n(cid:46) together with \u0001-greedy or Gaussian exploration\n\nSample a minibatch b from replay buffer using HER\nPerform one step on value and policy update on minibatch b using DDPG\n\nOverall Algorithm The overall description of our algorithm is shown in Algorithm 1. Note that our\nexploration strategy the only modi\ufb01cation is in Step 8, in which we generate hindsight goals to guide\nthe agent to collect more valuable trajectories. So it is complementary to other improvements in\nDDPG/HER around Step 16, such as the prioritized experience replay strategy (Schaul et al., 2016;\nZhao and Tresp, 2018; Zhao et al., 2019) and other variants of hindsight experience replay (Fang\net al., 2019b; Bai et al., 2019).\n\n5 Experiments\n\nOur experiment environments are based on the standard robotic manipulation environments in the\nOpenAI Gym (Brockman et al., 2016)3. In addition to the standard settings, to better visualize the\nimprovement of the sample ef\ufb01ciency, we vary the target task distributions in the following ways:\n\u2022 Fetch environments: Initial object position and goal are generated uniformly at random from\n\u2022 Hand-manipulation environments : These tasks require the agent to rotate the object into a\ngiven pose, and only the rotations around z-axis are considered here. We restrict the initial\n\ntwo distant segments.\n\n3Our code is available at https://github.com/Stilwell-Git/Hindsight-Goal-Generation.\n\n6\n\n\faxis-angle in a small interval, and the target pose will be generated in its symmetry. That is,\nthe object needs to be rotated in about \u03c0 degree.\n\u2022 Reach environment: FetchReach and HandReach do not support randomization of the initial\n\nstate, so we restrict their target distribution to be a subset of the original goal space.\n\nRegarding baseline comparison, we consider the original DDPG+HER algorithm. We also investigate\nthe integration of the experience replay prioritization strategies, such as the Energy-Based Prioritiza-\ntion (EBP) proposed by Zhao and Tresp (2018), which draws the prior knowledge of physics system\nto exploit valuable trajectories. More details of experiment settings are included in the Appendix B.\n\n5.1 HGG Generates Better Hindsight Goals for Exploration\n\nHGG\n\nHER\n\n(a) Episode 500\n\n(b) Episode 1000\n\n(c) Episode 2000\n\n(d) Episode 3000\n\nFigure 2: Visualization of goal distribution generated by HGG and HER on FetchPush. The initial\nobject position is shown as a black box. The blue segment indicates target goal distribution. The\nabove row presents the distribution of the hindsight goals generated by our HGG method, where\nbright green particles is a batch of recently generated goals, and dark green particles present the\ngoals generated in the previous iterations. The bottom row presents the distribution of replay goals\ngenerated by HER.\n\nWe \ufb01rst check whether HGG is able to generate meaningful hindsight goals for exploration. We\ncompare HGG and HER in the FetchPush environment. It is shown in Figure 2 that HGG algorithm\ngenerates goals that gradually move towards the target region. Since those goals are hindsight, they\nare considered to be achieved during training. In comparison, the replay distribution of a DDPG+HER\nagent has been stuck around the initial position for many iterations, indicating that those goals may\nnot be able to ef\ufb01ciently guide exploration.\nPerformance on benchmark robotics tasks\n\nFigure 3: Learning curves for variant a number of goal-oriented robotic manipulation tasks. All\ncurves presented in this \ufb01gure are trained with default hyper-parameters included in Appendix C.1.\nNote that since FetchReach and HandReach do not contain object instances for EBP, so we do not\ninclude the +EBP versions for them.\n\n7\n\n\fThen we check whether the exploration provided by the goals generated by HGG can result in better\npolicy training performance. As shown in Figure 3, we compare the vanilla HER, HER with Energy-\nBased Prioritization (HER+EBP), HGG, HGG+EBP. It is worth noting that since EBP is designed\nfor the Bellman equation updates, it is complementary to our HGG-based exploration approach.\nAmong the eight environments, HGG substantially outperforms HER on four and has comparable\nperformance on the other four, which are either too simple or too dif\ufb01cult. When combined with EBP,\nHGG+EBP achieves the best performance on six environments that are eligible.\nPerformance on tasks with obstacle In a more\ndif\ufb01cult task, crafted metric may be more suit-\nable than (cid:96)2-distance used in Eq. (5). As shown\nin Figure 4, we created an environment based on\nFetchPush with a rigid obstacle. The object and\nthe goal are uniformly generated in the green\nand the red segments respectively. The brown\nblock is a static wall which cannot be moved.\nIn addition to (cid:96)2, we also construct a distance\nmetric based on the graph distance of a mesh\ngrid on the plane, the blue line is a successful trajectory in such hand-craft distance measure. A more\ndetailed description is deferred to Appendix B.3. Intuitively speaking, this crafted distance should be\nbetter than (cid:96)2 due to the existence of the obstacle. Experimental results suggest that such a crafted\ndistance metric provides better guidance for goal generation and training, and signi\ufb01cantly improves\nsample ef\ufb01ciency over (cid:96)2 distance. It would be a future direction to investigate ways to obtain or\nlearn a good metric.\n\nFigure 4: Visualization of FetchPush with obstacle.\n\n5.2 Comparison with Explicit Curriculum Learning\n\nSince our method can be seen as an explicit curriculum learn-\ning for exploration, where we generate hindsight goals as\nintermediate task distribution, we also compare our method\nwith another recently proposed curriculum learning method for\nRL. Florensa et al. (2018) leverages Least-Squares GAN (Mao\net al., 2018b) to mimic the set called Goals of Intermediate\nDif\ufb01cult as exploration goal generator.\nSpeci\ufb01cally,\nin our task settings, we de\ufb01ne a goal set\nGOID(\u03c0) = {g : \u03b1 \u2264 f (\u03c0, g) \u2264 1 \u2212 \u03b1}, where f (\u03c0, g)\nrepresents the average success rate in a small region closed by\ngoal g. To sample from GOID, we implement an oracle goal\ngenerator based on rejection sampling, which could uniformly\nsample goals from GOID(\u03c0). Result in Figure 5 indicates\nthat our Hindsight Goal Generation substantially outperforms HER even with GOID from the oracle\ngenerator. Note that this experiment is run on a environment with \ufb01xed initial state due to the\nlimitation of Florensa et al. (2018). The choice of \u03b1 is also suggested by Florensa et al. (2018).\n\nFigure 5: Comparison with curricu-\nlum learning. We compare HGG\nwith the original HER, HER+GOID\nwith two threshold values.\n\n5.3 Ablation Studies on Hyperparameter Selection\n\nIn this section, we set up a set of ablation tests on several hyper-parameters used in the Hindsight\nGoal Generation algorithm.\nLipschitz L: The selection of Lipschitz constant is task dependent, since it iss related with scale of\nvalue function and goal distance. For the robotics tasks tested in this paper, we \ufb01nd that it is easier\nto set L by \ufb01rst divided it with the upper bound of the distance between any two \ufb01nal goals in a\nenvironment. We test a few choices of L on several environments and \ufb01nd that it is very easy to \ufb01nd\na range of L that works well and shows robustness for all the environments tested in this section. We\nshow the learning curves on FetchPush with different L. It appears that the performance of HGG is\nreasonable as long as L is not too small. For all tasks we tested in the comparisons, we set L = 5.0.\nDistance weight c: Parameter c de\ufb01nes the trade-off between the initial state similarity and the goal\nsimilarity. Larger c encourages our algorithm to choose hindsight goals that has closer initial state.\n\n8\n\n\fResults in Figure 6 indicates that the choice of c is indeed robust. For all tasks we tested in the\ncomparisons, we set c = 3.0.\nNumber of hindsight goals K: We \ufb01nd that for the simple tasks, the choice of K is not critical.\nEven a greedy approach (corresponds to K = 1) can achieved competitive performance, e.g. on\nFetchPush in the third panel of Figure 6. For more dif\ufb01cult environment, such as FetchPickAndPlace,\nlarger batch size can signi\ufb01cantly reduce the variance of training results. For all tasks tested in the\ncomparisons, we ploted the best results given by K \u2208 {50, 100}.\n\nFigure 6: Ablation study of hyper-parameter selection. Several curves are omitted in the forth panel\nto provide a clear view of variance comparison. A full version is deferred to Appendix D.4.\n\n6 Conclusion\n\nWe present a novel automatic hindsight goal generation algorithm, by which valuable hindsight\nimaginary tasks are generated to enable ef\ufb01cient exploration for goal-oriented off-policy reinforcement\nlearning. We formulate this idea as a surrogate optimization to identify hindsight goals that are easy\nto achieve and also likely to lead to the actual goal. We introduce a combinatorial solver to generate\nsuch intermediate tasks. Extensive experiments demonstrated better goal-oriented exploration of\nour method over original HER and curriculum learning on a collection of robotic learning tasks. A\nfuture direction is to incorporate the controllable representation learning (Thomas et al., 2017) to\nprovide task-speci\ufb01c distance metric (Ghosh et al., 2019; Srinivas et al., 2018), which may generalize\nour method to more complicated cases where the standard Wasserstein distance cannot be applied\ndirectly.\n\nReferences\nRavindra K. Ahuja, Thomas L. Magnanti, and James B. Orlin. Network Flows: Theory, Algorithms,\nand Applications. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1993. ISBN 0-13-617549-X.\nMarcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob\nMcGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay.\nIn Advances in Neural Information Processing Systems, pages 5048\u20135058, 2017.\n\nKavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model-based reinforce-\n\nment learning. In International Conference on Machine Learning, pages 264\u2013273, 2018.\n\nChenjia Bai, Peng Liu, Wei Zhao, and Xianglong Tang. Guided goal generation for hindsight\n\nmulti-goal reinforcement learning. Neurocomputing, 2019.\n\nAdrien Baranes and Pierre-Yves Oudeyer. Active learning of inverse models with intrinsically\n\nmotivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49\u201373, 2013.\n\nGreg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and\n\nWojciech Zaremba. Openai gym, 2016.\n\nC\u00e9dric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer. Gep-pg: Decoupling exploration and\nexploitation in deep reinforcement learning algorithms. In International Conference on Machine\nLearning, pages 1038\u20131047, 2018.\n\nC\u00e9dric Colas, Pierre-Yves Oudeyer, Olivier Sigaud, Pierre Fournier, and Mohamed Chetouani.\nCurious: Intrinsically motivated modular multi-goal reinforcement learning. In International\nConference on Machine Learning, pages 1331\u20131340, 2019.\n\n9\n\n\fYiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation\n\nlearning. In Advances in Neural Information Processing Systems, 2019.\n\nRan Duan and Hsin-Hao Su. A scaling algorithm for maximum weight matching in bipartite graphs.\nIn Proceedings of the twenty-third annual ACM-SIAM symposium on Discrete Algorithms, pages\n1413\u20131424. Society for Industrial and Applied Mathematics, 2012.\n\nAdrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new\n\napproach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019.\n\nBenjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer:\nBridging planning and reinforcement learning. In Advances in Neural Information Processing\nSystems, 2019.\n\nMeng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hindsight experience\n\nreplay for dynamic goals. In International Conference on Learning Representations, 2019a.\n\nMeng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight\n\nexperience replay. In Advances in Neural Information Processing Systems, 2019b.\n\nCarlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse\nIn Conference on Robot Learning, pages\n\ncurriculum generation for reinforcement learning.\n482\u2013495, 2017.\n\nCarlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for\nreinforcement learning agents. In International Conference on Machine Learning, pages 1514\u2013\n1523, 2018.\n\nS\u00e9bastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration\n\nprocesses with automatic curriculum learning. arXiv preprint arXiv:1708.02190, 2017.\n\nDibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal-\n\nconditioned policies. In International Conference on Learning Representations, 2019.\n\nAnirudh Goyal, Philemon Brakel, William Fedus, Soumye Singhal, Timothy Lillicrap, Sergey\nLevine, Hugo Larochelle, and Yoshua Bengio. Recall traces: Backtracking models for ef\ufb01cient\nreinforcement learning. In International Conference on Learning Representations, 2019a.\n\nAnirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle,\nInfobot: Transfer and exploration via the information\n\nSergey Levine, and Yoshua Bengio.\nbottleneck. In International Conference on Learning Representations, 2019b.\n\nZhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal\n\nreaching. In Advances in Neural Information Processing Systems, 2019.\n\nLeslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pages 1094\u20131099. Citeseer, 1993.\n\nAlexandros Karatzoglou, Linas Baltrunas, and Yue Shi. Learning to rank for recommender systems.\nIn Proceedings of the 7th ACM conference on Recommender systems, pages 493\u2013494. ACM, 2013.\n\nSergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep\n\nvisuomotor policies. The Journal of Machine Learning Research, 17(1):1334\u20131373, 2016.\n\nAndrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies\n\nwith hindsight. In International Conference on Learning Representations, 2019.\n\nTimothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,\nIn\n\nDavid Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.\nInternational Conference on Learning Representations, 2016.\n\nYuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorith-\nmic framework for model-based deep reinforcement learning with theoretical guarantees. In\nInternational Conference on Learning Representations, 2019.\n\n10\n\n\fJiayuan Mao, Honghua Dong, and Joseph J Lim. Universal agent for disentangling environments and\n\ntasks. In International Conference on Learning Representations, 2018a.\n\nXudong Mao, Qing Li, Haoran Xie, Raymond Yiu Keung Lau, Zhen Wang, and Stephen Paul Smolley.\nOn the effectiveness of least squares generative adversarial networks. IEEE transactions on pattern\nanalysis and machine intelligence, 2018b.\n\nVolodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare,\nAlex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control\nthrough deep reinforcement learning. Nature, 518(7540):529, 2015.\n\nJames Munkres. Algorithms for the assignment and transportation problems. Journal of the society\n\nfor industrial and applied mathematics, 5(1):32\u201338, 1957.\n\nO\ufb01r Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-ef\ufb01cient hierarchical\nreinforcement learning. In Advances in Neural Information Processing Systems, pages 3303\u20133313,\n2018.\n\nAshvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual\nIn Advances in Neural Information Processing\n\nreinforcement learning with imagined goals.\nSystems, pages 9191\u20139200, 2018.\n\nAndrew Y Ng, Daishi Harada, and Stuart J Russell. Policy invariance under reward transformations:\nTheory and application to reward shaping. In Proceedings of the Sixteenth International Conference\non Machine Learning, pages 278\u2013287. Morgan Kaufmann Publishers Inc., 1999.\n\nJunhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with\nmulti-task deep reinforcement learning. In International Conference on Machine Learning, pages\n2661\u20132670, 2017.\n\nDeepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan\nShelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In\nInternational Conference on Learning Representations, 2018.\n\nAlexandre P\u00e9r\u00e9, S\u00e9bastien Forestier, Olivier Sigaud, and Pierre-Yves Oudeyer. Unsupervised learning\nof goal spaces for intrinsically motivated goal exploration. In International Conference on Learning\nRepresentations, 2018.\n\nMatthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell,\nJonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learn-\ning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464,\n2018.\n\nVitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-\ufb01t:\n\nState-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.03698, 2019.\n\nPaulo Rauber, Avinash Ummadisingu, Filipe Mutz, and J\u00fcrgen Schmidhuber. Hindsight policy\n\ngradients. In International Conference on Learning Representations, 2019.\n\nMartin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele,\nVlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse\nreward tasks from scratch. In International Conference on Machine Learning, pages 4341\u20134350,\n2018.\n\nHimanshu Sahni, Toby Buckley, Pieter Abbeel, and Ilya Kuzovkin. Addressing sample complexity\nin visual tasks using her and hallucinatory gans. In Advances in Neural Information Processing\nSystems, 2019.\n\nTom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators.\n\nIn International conference on machine learning, pages 1312\u20131320, 2015.\n\nTom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In\n\nInternational Conference on Learning Representations, 2016.\n\n11\n\n\fJohn Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region\npolicy optimization. In International Conference on Machine Learning, pages 1889\u20131897, 2015.\n\nJohn Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy\n\noptimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\n\nDavid Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche,\nJulian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering\nthe game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.\n\nAravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan-\nning networks: Learning generalizable representations for visuomotor control. In International\nConference on Machine Learning, pages 4739\u20134748, 2018.\n\nSainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus.\nIntrinsic motivation and automatic curricula via asymmetric self-play. In International Conference\non Learning Representations, 2018.\n\nHao Sun, Zhizhong Li, Xiaotong Liu, Dahua Lin, and Bolei Zhou. Policy continuation with hindsight\n\ninverse dynamics. In Advances in Neural Information Processing Systems, 2019.\n\nCsaba Szepesv\u00e1ri. The asymptotic convergence-rate of q-learning. In Advances in Neural Information\n\nProcessing Systems, pages 1064\u20131070, 1998.\n\nValentin Thomas, Jules Pondard, Emmanuel Bengio, Marc Sarfati, Philippe Beaudoin, Marie-Jean\nMeurs, Joelle Pineau, Doina Precup, and Yoshua Bengio. Independently controllable features.\narXiv preprint arXiv:1708.01289, 2017.\n\nVivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning. arXiv preprint\n\narXiv:1806.09605, 2018.\n\nRui Zhao and Volker Tresp. Energy-based hindsight experience prioritization. In Conference on\n\nRobot Learning, pages 113\u2013122, 2018.\n\nRui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularized multi-goal reinforcement\n\nlearning. In International Conference on Machine Learning, pages 7553\u20137562, 2019.\n\n12\n\n\f", "award": [], "sourceid": 7458, "authors": [{"given_name": "Zhizhou", "family_name": "Ren", "institution": "Tsinghua University"}, {"given_name": "Kefan", "family_name": "Dong", "institution": "Tsinghua University"}, {"given_name": "Yuan", "family_name": "Zhou", "institution": "UIUC"}, {"given_name": "Qiang", "family_name": "Liu", "institution": "UT Austin"}, {"given_name": "Jian", "family_name": "Peng", "institution": "University of Illinois at Urbana-Champaign"}]}