{"title": "Reinforcement Learning for Mixed Open-loop and Closed-loop Control", "book": "Advances in Neural Information Processing Systems", "page_first": 1026, "page_last": 1032, "abstract": null, "full_text": "Reinforcement  Learning for  Mixed \nOpen-loop and  Closed-loop  Control \n\nEric  A.  Hansen,  Andrew  G.  Barto, and  Shlorno  Zilberstein \n\nDepartment  of Computer  Science \n\nUniversity  of Massachusetts \n\nAmherst,  MA  01003 \n\n{hansen.barto.shlomo }<Dcs.umass .edu \n\nAbstract \n\nClosed-loop  control  relies  on  sensory  feedback  that  is  usually  as(cid:173)\nsumed  to  be  free .  But  if sensing  incurs  a  cost,  it  may  be  cost(cid:173)\neffective  to  take  sequences  of actions  in  open-loop  mode.  We  de(cid:173)\nscribe  a  reinforcement  learning  algorithm  that  learns  to  combine \nopen-loop  and  closed-loop  control  when  sensing  incurs  a  cost.  Al(cid:173)\nthough  we  assume reliable  sensors,  use  of open-loop control means \nthat  actions  must  sometimes  be  taken  when  the  current  state  of \nthe  controlled  system  is  uncertain.  This  is  a  special  case  of the \nhidden-state  problem  in  reinforcement  learning,  and  to  cope,  our \nalgorithm relies  on short-term memory.  The main result  of the pa(cid:173)\nper is a rule that significantly limits exploration of possible  memory \nstates  by  pruning  memory states  for  which  the  estimated  value  of \ninformation is  greater than its cost.  We  prove that  this  rule  allows \nconvergence  to an optimal  policy. \n\n1 \n\nIntroduction \n\nReinforcement  learning  (RL)  is  widely-used  for  learning  closed-loop  control  poli(cid:173)\ncies.  Closed-loop  control  works  well  if the  sensory  feedback  on  which  it  relies  is \naccurate,  fast,  and  inexpensive.  But  this  is  not  always  the  case.  In  this  paper,  we \naddress  problems  in  which  sensing  incurs  a  cost,  either  a  direct  cost  for  obtaining \nand  processing  sensory  data or an  indirect  opportunity  cost  for  dedicating  limited \nsensors to one control task rather than another.  If the cost  for sensing is  significant, \nexclusive  reliance  on  closed-loop  control  may make it impossible  to optimize  a  per(cid:173)\nformance  measure  such  as  cumulative  discounted  reward.  For  such  problems,  we \ndescribe an  RL algorithm that learns to combine open-loop and closed-loop control. \nBy learning  to take open-loop sequences  of actions between sensing,  it can optimize \na  tradeoff'between  the cost  and  value  of sensing. \n\n\fReinforcement Learning for Mixed Open-loop and Closed-loop Control \n\n1027 \n\nThe  problem  we  address  is  a  special  case  of the problem  of hidden  state  or partial \nobservability  in  RL  (e.g.,  Whitehead  &.  Lin,  1995;  McCallum,  1995).  Although  we \nassume sensing  provides perfect information (a significant limiting assumption),  use \nof open-loop control means that actions must sometimes  be taken when  the current \nstate  of  the  controlled  system  is  uncertain.  Previous  work  on  RL  for  partially \nobservable  environments has focused  on coping  with sensors  that provide imperfect \nor incomplete  information,  in  contrast  to deciding  whether  or  when  to  sense.  Tan \n(1991)  addressed  the problem  of sensing  costs by showing  how  to use  RL to learn  a \ncost-effective  sensing  procedure  for  state identification,  but his  work  addressed  the \nquestion of which sensors  to use,  not when to sense, and so still assumed closed-loop \ncontrol. \n\nIn  this  paper, we  formalize  the problem of mixed  open-loop and closed-loop control \nas a  Markov decision  process  and  use  RL in  the form  of Q-Iearning  to learn  an op(cid:173)\ntimal,  state-dependent  sensing interval.  Because  there  is  a  combinatorial explosion \nof open-loop  action  sequences,  we  introduce  a  simple  rule  for  pruning  this  large \nsearch  space.  Our  most  significant  result  is  a  proof that  Q-Iearning  converges  to \nan  optimal  policy  even  when  a  fraction  of the  space  of possible  open-loop  action \nsequences  is  explored. \n\n2  Q-learning  with  sensing  costs \n\nQ-Iearning  (Watkins,  1989)  is  a  well-studied  RL  algorithm  for  learning  to  control \na  discrete-time,  finite  state  and  action  Markov  decision  process  (MDP).  At  each \ntime  step,  a  controller  observes  the current  state  x,  takes  an action  a,  and receives \nan  immediate  reward  r  with  expected  value  r(x, a).  With  probability  p(x, a, y) \nthe  process  makes  a  transition  to  state  y,  which  becomes  the  current  state  on  the \nnext  time step.  A  controller  using  Q-Iearning  learns  a  state-action  value  function, \nQ(x, a),  that  estimates  the  expected  total  discounted  reward  for  taking  action  a \nin  state  x  and  performing  optimally  thereafter.  Each  time  step,  Q is  updated  for \nstate-action  pair  (x, a)  after  receiving  reward  r  and observing  resulting  state  y,  as \nfollows: \n\nQ(x, a)  ~ Q(x, a)  + Q:  [r + I'V(y)  - Q(x, a)]  , \n\nwhere  Q:  E  (0,1]  is  a  learning  rate  parameter,  I' E  [0,1)  is  a  discount  factor,  and \nV(y)  =  maXb Q(y, b).  Watkins  and  Dayan  (1992)  prove  that  Q converges  to  an \noptimal state-action  value  function  Q  (and  V converges  to an optimal  state  value \nfunction  V)  with  probability  one  if all  actions  continue  to  be  tried  from  all  states, \nthe  state-action  value  function  is  represented  by  a  lookup-table,  and  the  learning \nrate is  decreased  in  an appropriate manner. \n\nIf there is  a cost for  sensing,  acting optimally may require a mixed  strategy of open(cid:173)\nloop  and  closed-loop  control  that  allows  a  controller  to  take  open-loop  sequences \nof actions  between  sensing.  This  possibility  can  be  modeled  by an  MDP  with  two \nkinds of actions:  control actions that have an effect  on the current  state but do  not \nprovide information, and a  sensing  action that reveals  the current state but  has no \nother  effect.  We  let  0  (for  observation)  denote  the  sensing  action  and  assume  it \nprovides perfect  information about the underlying state.  Separating control actions \nand the sensing action gives  an agent control over when to receive  sensory feedback, \nand  hence,  control over sensing  costs. \n\nWhen one control action follows  another without an intervening sensing action,  the \nsecond  control  action  is  taken  without  knowing  the  underlying  state.  We  model \nthis  by  including  \"memory  states\"  in  the  state  set  of  the  MDP.  Each  memory \nstate  represents  memory of the  last  observed  state and  the  open-loop  sequence  of \ncontrol actions taken since;  because we assume sensing provides perfect information, \n\n\f1028 \n\nE.  A. Hansen, A. G.  Barto and S.  Zilberstein \n\nXcl Z xaa \n\nxah \n\n~X'~'I \n\nxh  ~ \nxhb \n\nFigure  1:  A  tree  of memory states  rooted  at observed  state  x.  The set  of control \nactions  is  {a, b}  and the length bound  is  2. \n\nremembering  this  much  history  provides  a  sufficient  statistic  for  action  selection \n(Monahan,  1982).  Possible  memory  states  can  be  represented  using  a  tree  like \nthe  one  shown  in  Figure  1,  where  the  root  represents  the  last  observed  state  and \nthe  other  nodes  represent  memory  states,  one  for  each  possible  open-loop  action \nsequence.  For  example,  let  xa denote  the  memory  state  that  results  from  taking \ncontrol action  a  in  state  x.  Similarly,  let  xab denote  the memory state  that results \nfrom  taking  control action  b in  memory state  xa.  Note that a  control action  causes \na  deterministic  transition  to  a  subsequent  memory  state,  while  a  sensing  action \ncauses  a  stochastic  transition  to an  observed  state  - the  root  of some  tree.  There \nis  a  tree  like  the one in  figure  1 for  each  observable  state. \n\nThis problem is  a special case of a  partially observable  MDP and can be formalized \nin  an analogous  way  (Monahan,  1982).  Given  a  state-transition  and reward  model \nfor  a  core  MDP  with  a  state  set  that  consists  only  of the  underlying  states  of a \nsystem (which for this problem we also call observable states), we can define a state(cid:173)\ntransition  and reward  model  for  an  MDP  that includes  memory  states  in  its  state \nset.  As  a  convenient  notation, let  p(x, al .. a\", y)  denote  the  probability  that taking \nan  open-loop  action  sequence  a} .. a\"  from  state  x  results  in  state  y,  where  both  x \nand  y  are  states  of the  underlying  system.  These  probabilities  can  be  computed \nrecursively  from  the  single-step  state-transition  probabilities  of the  core  MDP  as \nfollows: \n\nState-transition  probabilities  for  the  sensing  action  can then  be defined  as \n\nz \n\np(xal .. a\" , 0, y)  = p(x, al .. a\", y), \n\nand a  reward  function  for  the generalized  MDP  can  be similarly  defined  as \n\nr(xal .. a\"_l, a,,)  =  LP(x, al .. a\"_l, y)r(y, a,,). \n\ny \n\nwhere  the  cost  of sensing  in state  x  of the core  MDP is  r(x,o). \n\nIf we  assume a  bound  on  the number  of control  actions  that can  be taken  between \nsensing  actions  (i.e .\u2022  a  bound  on  the  depth  of each  tree)  and  also  assume  a  finite \nIt \nnumber  of underlying  states,  the  number  of possible  memory  states  is  finite. \nfollows  that the  MDP  we  have  constructed  is  a  well-defined  finite  state and  action \nMDP,  and  all  of the  theory developed  for  Q-Iearning  continues  to apply,  including \nits  convergence  proof.  (This  is  not  true  of partially  observable  MDPs in  general.) \nTherefore,  Q-Iearning can in  principle find  an optimal policy for interleaving control \nactions  and sensing,  assuming sensing  provides  perfect  information. \n\n3  Limiting  Exploration \n\nA  problem  with  including  memory  states  in  the  state  set  of an  MDP  is  that  it \nincreases  the  size  of the  state  set  exponentially.  The  combinatorial  explosion  of \n\n\fReinforcement Learningfor Mixed Open-loop and Closed-loop Control \n\n1029 \n\nstate-action values  to be learned raises  doubt about the computational feasibility  of \nthis generalization of RL. We present a solution in the form of a rule for pruning each \ntree  of memory states,  thereby limiting  the  number of memory states that must be \nexplored.  We prove that even if some memory states are never explored,  Q-Iearning \nconverges  to an optimal state-action value function.  Because  the state-action value \nfunction  is  left  undefined  for  unexplored  memory  states,  we  must  carefully  define \nwhat  we  mean  by an  optimal state-action  value  function. \n\nDefinition:  A  state-action  value  function  is  optimal if it is  sufficient for  generat(cid:173)\ning  optimal behavior  and  the  values  of the  state-action pairs  visited  when  behaving \noptimally are  optimal. \n\nA  state-action  value function  that is  undefined  for  some  states  is  optimal,  by  this \ndefinition,  if a  controller  that  follows  it  behaves  identically  to  a  controller  with  a \ncomplete, optimal state-action value function.  This is possible if the states for which \nthe state-action value function  is  undefined  are not encountered  when an agent acts \noptimally.  Barto,  Bradtke,  and  Singh  (1995)  invoke  a  similar  idea  for  a  different \nclass  of problems. \n\nLet  g(xal .. ak)  denote  the  expected  reward  for  taking  actions  al .. ak  in  open-loop \nmode  after  observing  state  x: \n\ng( xa1 .. ak)  = r(x, ad + L \ni=l \n\nk-l \n\n-yi r(xa l .. ai, ai+d\u00b7 \n\nLet  h(xa1 .. ak)  denote  the  discounted  expected  value  of perfect  information  after \nreaching memory state xa1 .. ak, which is  equal to the discounted Q-value for sensing \nin  memory state  xal .. ak  minus  the cost  for  sensing in  this  state: \n\nh(xa1 .. ak)  =  -y\"  LP(xal .. ak,o,y)V(y) =  -yk(Q(xa1 .. ak,o) - r(xal .. ak,o)). \n\ny \n\nBoth  g  and  h  are  easily  learned  during  Q-Iearning,  and  we  refer  to  the  learned \nestimates  as 9 and h.  These  are  used  in  the  pruning rule,  as  follows: \n\nPruning rule:  If g( xal .. ak) + h{ xa1' .ak)  ~ V (x),  then memory states that descend \nfrom  xal\u00b7 .ak  do  not need  to  be  explored.  A  controller should immediately execute  a \nsensing  action  when  it reaches  one  of these  memory states. \n\nThe  intuition  behind  the  pruning  rule  is  that a  branch  of a  tree  of memory states \ncan  be  pruned  after  reaching  a  memory  state  for  which  the  value  of information \nis  greater  than  or equal  to its  cost.  Because  pruning  is  based  on  estimated  values, \nmemory states that are pruned at one point during learning may later be explored as \nlearned estimates change.  The net effect  of pruning, however, is  to focus exploration \non a  subset  of memory states, and as Q-Iearning  converges,  the subset of unpruned \nmemory states  becomes  stable.  The following  theorem  is  proved  in  an appendix. \n\nTheorem:  Q-learning  converges  to  an  optimal  state-action  value  function  with \nprobability  one  if,  in  addition  to  the  conditions  for  convergence  given  by  Watkins \nand  Dayan  (1992),  exploration is  limited by  the  pruning rule. \n\nThis  result  is  closely  related  to  a  similar  result  for  solving  this  class  of problems \nusing  dynamic  programming  (Hansen,  1997),  where  it  is  shown  that  pruning  can \nassure  convergence  to  an  optimal  policy  even  if no  bound  is  placed  on  the  length \nof open-loop  action  sequences  - under  the  assumption  that  it  is  optimal  to  sense \nat  finite  intervals.  This  additional  result  can  be  extended  to  Q-Iearning  as  well, \nalthough  we  do  not  present  the extension  in  this  paper.  An artificial  length  bound \ncan  be set as low  or high as desired  to ensure  a  finite  set  of memory  states. \n\n\f1030 \n\nE. A.  Hansen, A. G.  Barto and S.  Zilberstein \n\n~ r -\nItli ll  I \n~ 3 \n'4 '5 \n6  7  X \n\nl)  10  II  I~ \n13  14  1510 \n17  IX  19  20 \n\n(in.1i  Sill\" \n\n7  WN:-iNO \n\nI~  NWNO \n\nI  WO \n\n:'>10 \n\n.1  \"\\10 \n\n~  :>;:>;0 \n\nX  WWNNNO \n\nI ~  WWO \n\nC)  NWO \n\nI~  wwwo \n\nIII  WNwn \n\n17  NNNO \n\nII  WW() \n\n11\\  WNNNO \n\n~  NNWO \n\n12  WWWO \n\n1<)  WWNO \n\n6  NNNO \n\nL'  NNO \n\n20  WWWNO \n\n( h) \n\nFigure  2:  (a)  Grid  world  with  numbered  states  (b)  Optimal  policy \n\nWe  use  the  notation  9  and  h  in  our  statement  of the  pruning  rule  to  emphasize \nits  relationship  to  pruning  in  heuristic  search.  If we  regard  the  root  of a  tree  of \nmemory  states  as  the  start  state  and  the  memory  state  that  corresponds  to  the \nbest  open-loop  action  sequence  as  the  goal  state,  then  9  can  be  regarded  as  the \ncost-to-arrive function  and the value of perfect  information h can be regarded as an \nupper  bound  on  the cost-to-go function. \n\n4  Example \n\nWe describe  a  simple  example  to illustrate  the extent of pruning  possible  using  this \nrule.  Imagine  that  a  \"robot\"  must  find  its  way  to  a  goal  location  in  the  upper \nleft-hand  corner  of the  grid  shown  in  Figure  2a.  Each  cell  of the  grid  corresponds \nto  a  state,  with  the  states  numbered  for  convenient  reference.  The  robot  has  five \ncontrol  actions;  it  can  move  north,  east,  south,  or  west,  one  cell  at  a  time,  or  it \ncan  stop.  The  problem  ends  when  the  robot  stops.  If it  stops  in  the  goal  state it \nreceives  a  reward  of 100,  otherwise  it receives  no  reward.  The robot  must  execute \na  sequence  of actions  to reach  the goal state,  but its  move actions are  stochastic.  If \nthe  robot  attempts  to  move  in  a  particular  direction,  it  succeeds  with  probability \no.s.  With  probability  0.05  it  moves  in  a  direction  90  degrees  off to  one  side  of its \nintended direction,  with probability 0.05 it moves in a direction  90 degrees off to the \nother side,  and with probability 0.1 it does not move at all.  If the robot's movement \nwould  take it  outside  the  grid,  it  remains  in  the  same cell.  Because  its  progress  is \nuncertain,  the robot must interleave  sensing and control actions to  keep  track of its \nlocation.  The reward for  sensing is  - 1  (i.e., a  cost  of 1)  and for  each move action it \nis  -4.  To optimize  expected  total reward,  the robot  must  find  its  way  to  the  goal \nwhile  minimizing  the combined  cost  of moving and  sensing. \n\nFigure 2b shows the optimal open-loop sequence of actions for each observable state. \nIf the  bound  on  the  length  of an  open-loop sequence  of control  actions  is  five,  the \nnumber  of possible  memory states  for  this  problem  is  over  64,000,  a  number  that \ngrows  explosively  as  the  length  bound  is  increased  (to  over  16  million  when  the \nbound is  nine) .  Using  the pruning rule,  Q-Iearning  must explore just less  than  1000 \nmemory states (and no deeper than nine levels in any tree) to converge to an optimal \npolicy,  even  when  there  is  no  bound on  the interval  between  sensing  actions. \n\n5  Conclusion \n\nWe  have  described  an  extension  of  Q-Iearning  for  MDPs  with  sensing  costs  and \na  rule  for  limiting  exploration  that  makes  it  possible  for  Q-Iearning  to  converge \nto  an  optimal  policy  despite  exploring  a  fraction  of possible  memory  states.  As \nalready pointed out, the problem we  have formalized  is  a  partially observable  MDP, \n\n\fReinforcement Learningfor Mixed Open-loop and Closed-loop Control \n\n1031 \n\nalthough  one  that  is  restricted  by  the  assumption  that  sensing  provides  perfect \ninformation .  An  interesting  direction  in  which  to  pursue  this  work  would  be  to \nexplore its relationship  to work on  RL for  partially observable  MDPs,  which  has so \nfar  focused  on  the  problem  of sensor  uncertainty  and  hidden  state.  Because  some \nof this  work also  makes  use of tree representations  of the state space and of learned \nstate-action  values  (e.g.,  McCallum,  1995),  it  may  be  that  a  similar  pruning  rule \ncan  constrain  exploration for  such  problems. \n\nAcknowledgement s \n\nSupport for  this work  was  provided in  part by  the National Science  Foundation un(cid:173)\nder  grants ECS-9214866 and IRI-9409827 and in  part by  Rome  Laboratory, USAF, \nunder  grant F30602-95-1-0012. \n\nReferences \n\nBarto,  A.G.;  Bradtke,  S.J .;  &.  Singh,  S.P.  (1995)  Learning  to  act  using  real-time \ndynamic  programming.  Artificial Intelligence  72(1/2}:81-138. \n\nHansen,  E.A.  (1997)  Markov decision  processes  with  observation  costs.  University \nof Massachusetts  at Amherst,  Computer Science  Technical  Report  97-01. \n\nMcCallum,  R.A.  (1995)  Instance-based  utile distinctions  for  reinforcement  learning \nwith  hidden  state.  In  Proc.  12th Int.  Machine  Learning  Conf.  Morgan  Kaufmann. \n\nMonahan,  G .E.  (1982)  A survey  of partially  observable  Markov  decision  processes: \nTheory,  models, and algorithms.  Management  Science  28:1-16. \n\nTan, M.  (1991)  Cost-sensitive  reinforcement  learning for  adaptive classification  and \ncontrol.  In Proc.  9th Nat.  Conf.  on Artificial  Intelligence.  AAAI Press/MIT Press. \n\nWatkins,  C.J.C.H.  (1989)  Learning from  delayed rewards.  Ph.D. Thesis,  University \nof Cambridge,  England. \nWatkins,  C.J.C.H. &.  Dayan, P.  (1992) Technical note:  Q-Iearning.  Machine  Learn(cid:173)\ning 8(3/4}:279-292. \nWhitehad,  S.D. &.  Lin,  L.-J.(1995}  Reinforcement  learning  of non-Markov decision \nprocesses.  Artificial Intelligence 73 :271-306. \n\nAppendix \n\nProof of theorem:  Consider  an  MDP  with  a  state  set  that  consists  only  of the \nmemory  states  that  are  not  pruned.  We  call  it  a  \"pruned  MDP\"  to  distinguish \nit  from  the  original  MDP  for  which  the  state  set  consists  of all  possible  memory \nstates.  Because  the pruned  MDP is  a  finite  state and action  MDP,  Q-Iearning with \npruning converges with probability one.  What we  must show is  that the state-action \nvalues  to  which  it  converges  include  every  state-action  pair  visited  by  an  optimal \ncontroller  for  the  original  MDP,  and  that  for  each  of these  state-action  pairs  the \nlearned state-action  value  is  equal to the optimal state-action  value for  the original \nMDP. \nLet Q and if denote the values that are learned by Q-Iearning when its exploration is \nlimited by the pruning rule,  and let  Q and V  denote value functions that are optimal \nwhen the state set ofthe MDP includes all possible memory states.  Because an MDP \nhas  an  optimal  stationary  policy  and  each  control  action  causes  a  deterministic \ntransition  to  a  subsequent  memory  state,  there  is  an  optimal  path  through  each \ntree of memory states.  The learned  value of the root state of each  tree is  optimal if \nand  only if the learned  value  of each  memory state along this  path is  also  optimal. \n\n\f1032 \n\nE.  A. Hansen, A.  G.  Barto and S.  Zilberstein \n\nTherefore  to  show  that  Q-Iearning  with  pruning  converges  to  an  optimal  state(cid:173)\naction  value  function,  it is  sufficient  to show that V = V  for  every  observable  state \nx.  Our  proof is  by  induction  on  the  number  of control  actions  that  can  be  taken \nbetween  one  sensing  action  and  the  next .  We  use  the  fact  that  if Q-Iearning  has \nconverged,  then  g(xal .. ai)  =  g(xal .. ai)  and  h(xat .. ai)  =  Eyp(x,al .. ai'Y)V(y)  for \nevery  memory state  xat .. ai . \nFirst  note  that  if g(xat) + 1'r(xal' o)  + h(xat}  >  V(x),  that  is,  if  V  for  some \nobservable  state  x  can  be  improved  by  exploring  a  path of a  single  control  action \nfollowed  by sensing,  then it is  contradictory to suppose Q-Iearning with pruning has \nconverged  because  single-depth  memory  states  in  a  tree  are  never  pruned.  Now, \nmake the inductive  hypothesis that Q-Iearning with  pruning has not converged if V \ncan be improved for some observable state by exploring  a  path of less  than k  control \nactions before sensing.  We show that it has not converged if V can be  improved for \nsome observable  state  by exploring  a  path of k control  actions  before sensing. \nSuppose V for  some  observable  state  x  can  be  improved  by  exploring  a  path that \nconsists  of taking the sequence  of control actions  at .. aA:  before  sensing,  that is, \n\ng(xat .. aA:)  + l'  r(xat .. aA:, o) + h(xat .. aA:)  >  V(x), \n\nA \n\nA \n\nA: \n\nSince only pruning can prevent improvement in this case, let  xat .. a,  be the memory \nstate at which  application of the pruning rule  prevents  xal .. aA:  from  being explored. \nBecause  the tree  has  been  pruned at this  node,  V(x)  2::  g(xat .. ai) + h(xat .. ai),  and \nso \n\ng(xat .. aA:)  + l'  r(xat .. aA:, o) + h(xat .. aA:)  > g(xai .. ai) + h(xat .. ai). \n\nA: \n\nA \n\nA \n\nWe  can  expand this  inequality  as follows: \ng(xat .. a,;) + 1\"  L p(x, al .. ai, y)  [g(ya,+1 .. aA:)  + 1'A:-i r(yai+t .. aA:, 0) + h(yai+t .. aA:)] \n\ny \n\n> g(xat .. a,;)  + h(:z:at .. ai)' \n\nSimplification  and expansion  of h  yields \nL p(x, at .. ai, y)  [g(yai+t .. aA:)  + 1'A:-i r(yai+t .. aA:, 0) + 1'A:-i L p(y, ai+1 .. aA:,  Z)V(Z)] \n\nyES \n\nz \n\n> L p(x, al .. ai, y)V(y). \n\ny \n\nTherefore,  there is  some observable  state,  y,  such  that \n\nz \n\nBecause  the  value  of observable  state  y  can  be  improved  by  taking  less  than  k \ncontrol  actions  before  sensing,  by  the  inductive  hypothesis  Q-Iearning  has  not  yet \nconverged.  0 \n\nThe  proof provides  insight  into  how  pruning  works.  If a  state-action  pair  along \nsome  optimal path is  temporarily  pruned,  it  must  be  possible  to improve  the  value \nof some  observable  state  by  exploring  a  shorter  path  of memory  states  that  has \nnot  been  pruned.  The  resulting  improvement  of the  value  function  changes  the \nthreshold  for  pruning  and  the  state-action  pair  that  was  formerly  pruned  may  no \nlonger  be  so,  making further  improvement  of the learned  value  function  possible. \n\n\f", "award": [], "sourceid": 1278, "authors": [{"given_name": "Eric", "family_name": "Hansen", "institution": null}, {"given_name": "Andrew", "family_name": "Barto", "institution": null}, {"given_name": "Shlomo", "family_name": "Zilberstein", "institution": null}]}