{"title": "Why did TD-Gammon Work?", "book": "Advances in Neural Information Processing Systems", "page_first": 10, "page_last": 16, "abstract": null, "full_text": "Why did TD-Gammon Work? \n\nJordan B. Pollack & Alan D. Blair \n\nComputer Science Department \n\nBrandeis University \nWaltham, MA 02254 \n\n{pollack,blair} @cs.brandeis.edu \n\nAbstract \n\nAlthough TD-Gammon is one of the major successes in machine learn(cid:173)\ning,  it has  not led to  similar impressive breakthroughs  in  temporal dif(cid:173)\nference  learning  for  other applications  or even  other games.  We  were \nable  to  replicate  some  of the  success  of TD-Gammon,  developing  a \ncompetitive evaluation function on a 4000 parameter feed-forward neu(cid:173)\nral network, without using back-propagation, reinforcement or temporal \ndifference learning methods. Instead we apply simple hill-climbing in  a \nrelative fitness  environment.  These results and further  analysis  suggest \nthat the surprising success of Tesauro's program had more to do with the \nco-evolutionary  structure of the  learning task and  the  dynamics  of the \nbackgammon game itself. \n\n1  INTRODUCTION \nIt took great chutzpah  for Gerald Tesauro  to  start  wasting  computer cycles on  temporal \ndifference learning in the game of Backgammon (Tesauro,  1992). Letting a machine learn(cid:173)\ning program play itself in the hopes of becoming an expert, indeed! After all, the dream of \ncomputers mastering a domain by self-play or \"introspection\" had been  around since the \nearly days  of AI,  forming  part  of Samuel's  checker player  (Samuel,  1959)  and  used  in \nDonald Michie's MENACE tic-tac-toe learner (Michie,  1961).  However such self-condi(cid:173)\ntioning  systems,  with  weak or non-existent  internal  representations,  had  generally  been \nfraught with problems of scale and abandoned by the field  of AI. Moreover, self-playing \nlearners  usually develop  eccentric  and  brittle  strategies  which allow  them to draw  each \nother, yet play poorly against humans and other programs. \nYet Tesauro's 1992 result showed that this self-play approach could be powerful, and after \nsome  refinement  and  millions  of iterations  of self-play,  his  TD-Gammon  program  has \nbecome one of the  best backgammon  players  in  the  world  (Tesauro,  1995).  His  derived \nweights are viewed by his corporation as  significant enough intellectual property to keep \nas a trade secret, except to leverage sales of their minority operating system (International \nBusiness  Machines,  1995). Others  have  replicated  this  TD  result  both  for  research  pur(cid:173)\nposes (Boyan,  1992) and commercial purposes. \n\n\fWhy did TD-Gammon Work? \n\n11 \n\nWith respect to the goal of a self-organizing learning machine which starts from a minimal \nspecification and rises  to great sophistication, TD-Gammon stands alone.  How is  its  suc(cid:173)\ncess to be understood, explained, and replicated in other domains? Is TD-Gammon unbri(cid:173)\ndled good news about the reinforcement learning method? \nOur hypothesis is that the success ofTD-gammon is not due to the back-propagation, rein(cid:173)\nforcement, or temporal-difference technologies, but to an inherent bias from the dynamics \nof the game of backgammon, and the co-evolutionary setup of the training,  by which the \ntask dynamically  changes  as  the learning progresses.  We  test  this  hypothesis  by  using  a \nmuch simpler co-evolutionary learning method for backgammon - namely hill-climbing. \n2  SETUP \nWe  use a standard feedforward  neural network with two layers and the  sigmoid function, \nset up in the same fashion as Tesauro with 4 units to represent the number of each player's \npieces on each of the 24 points, plus 2 units each to indicate how many are on the bar and \noff the board. In addition, we added one more unit which reports whether or not the game \nhas reached the endgame or \"race\" situation, making a total of 197 input units. These are \nfully connected to 20 hidden units, which are then connected to one output unit that judges \nthe position. Including bias on the  hidden  units,  this makes a total  of 3980 weights.  The \ngame is  played  by  generating  all  legal  moves,  converting  them  into  the  proper network \ninput, and picking the position judged as  best by the network. We started with all  weights \nset to zero. \nOur initial algorithm was hillclimbing: \n1.  add gaussian noise to the weights \n2.  play the network against the mutant for a number of games \n3.  if the mutant wins more than half the games, select it for the next generation. \nThe noise was set so each step would  have a 0.05 RMS distance  (which is the euclidean \ndistance divided by  J3980). \nSurprisingly, this  worked reasonably well!  The networks so evolved improved rapidly  at \nfirst, but then sank into mediocrity. The problem we perceived is that comparing two close \nbackgammon players is  like tossing a biased coin repeatedly:  it may take dozens or even \nhundreds  of games  to  find  out for  sure  which  of them  is  better.  Replacing  a  well-tested \nchampion is dangerous without enough information to prove the challenger is really a bet(cid:173)\nter player and not just a lucky  novice. Rather than burden the system with so  much com(cid:173)\nputation, we instead introduced the following modifications to the algorithm to avoid this \n\"Buster Douglas Effect\": \nFirstly, the games are played in pairs, with the order of play reversed and the same random \nseed  used  to generate the  dice rolls for both games. This washes out some of the unfair(cid:173)\nness due to the dice rolls when the two networks are very close - in particular, if they were \nidentical, the result would always  be one win  each. Secondly,  when  the challenger wins \nthe contest,  rather than just replacing  the champion  by  the  challenger,  we  instead  make \nonly a small adjustment in that direction: \n\nchampion := 0.95*champion + 0.05*challenger \n\nThis  idea,  similar  to  the  \"inertia\"  term  in  back-propagation,  was  introduced  on  the \nassumption that small changes in weights would lead to small changes in decision-making \nby  the  evaluation function.  So,  by  preserving  most of the  current champion's decisions, \nwe  would  be less  likely  to have  a catastrophic replacement of the  champion by  a lucky \nnovice challenger. \nIn  the  initial  stages  of evolution,  two pairs of parallel games  were played  and  the chal(cid:173)\nlenger was required to win 3 out of 4 of these games. \n\n\f12 \n\n1.  B.  Pollack and A.  D.  Blair \n\n~ o  50 \n~ \n\n25 \n\nGeneration (x 103) \n\n5 \n\n10 \n\n15 \n\n20 \n\n25 \n\n30 \n\n35 \n\nFigure 1. Percentage of losses of our first 35,000 generation players \nagainst PUBEVAL. Each match consisted of 200 games. \n\nFigure 1 shows the first 35,000 players rated against PUBEVAL, a  strong public-domain \nplayer trained by Tesauro using human expert preferences. There are three things to note: \n(1) the percentage of losses against PUBEVAL falls  from  100% to about 67% by  20,000 \ngenerations, (2) the frequency of successful challengers increases over time as  the player \nimproves, and (3) there are epochs (e.g. starting at 20,000) where the performance against \nPUBEVAL begins to falter. The first fact shows that our simple self-playing hill-climber is \ncapable of learning.  The  second  fact  is  quite counter-intuitive - we  expected that as  the \nplayer improved, it would be harder to challenge it!  This is true with respect to a uniform \nsampling of the 4000 dimensional weight space, but not true for a sampling in the  neigh(cid:173)\nborhood of a given player: once the player is in a good part of weight space, small changes \nin weights can lead to mostly similar strategies, ones which make mostly the same moves \nin  the  same situations.  However,  because of the  few  games  we  were using  to determine \nrelative fitness,  this increased frequency of change allows the system to drift,  which may \naccount for the subsequent degrading of performance. \nTo counteract the drift, we decided to change the rules of engagement as the evolution pro(cid:173)\nceeds according to the following \"annealing schedule\": after 10,000 generations, the num(cid:173)\nber of games that the challenger is required to win  was increased from  3 out of 4 to 5 out \nof 6;  after 70,000 generations, it was further increased to 7 out of 8.  The numbers  10,000 \nand 70,000 were chosen on an  ad  hoc  basis  from  observing the frequency  of successful \nchallenges. \nAfter 100,000 games, we have developed a surprisingly strong player, capable of winning \n40% of the games against PUBEV AL.  The networks were sampled every  100 generations \nin order to test their performance. Networks at generation 1,000,  10,000 and  100,000 were \nextracted and used as benchmarks. Figure 2 shows the percentage of losses of the sampled \nplayers  against the three  benchmark  networks.  Note that the three curves  cross the  50% \nline at  1,  10, and 100, respectively and show a general improvement over time. \nThe end-game of backgammon, called the \"bear-off,\" can be used as another yardstick of \nthe  progress  of learning.  The  bear-off  occurs  when  all  of a  player's  pieces  are  in  the \nplayer's home, or first 6  points, and then  the dice rolls can be used to remove pieces. We \nset up a racing board with two pieces on each player's 1 through 7 point and one piece on \nthe 8 point,  and played  a player against itself 200 games,  averaging the  number of rolls. \nWe found a monotonic improvement, from 22 to less then  19 rolls, over the  lOOk genera(cid:173)\ntions. PUBEV AL scored 16.6 on this task. \n\n\fWhy did TD-Gammon Work? \n\n13 \n\nl00.-------r-----~~----~------~------~ \n\n20 \n\n40 \n\n60 \nGeneration (x 103) \n\n80 \n\n100 \n\nFigure 2. Percentage of losses against benchmark networks at \ngeneration  1,000 [lower),  10,000 [middle) and  100,000 [upper). \n\n3  DISCUSSION \n3.1  Machine Learning and Evolution \nWe  believe that our evidence of success in learning backgammon using simple hillclimb(cid:173)\ning indicates that the reinforcement and temporal difference methodology used by Tesauro \nin TO-gammon was non-essential for its success. Rather, the success came from the setup \nof co-evolutionary  self-play  biased  by  the  dynamics of backgammon.  Our result is  thus \nsimilar to the bias found by Mitchell, Crutchfield & Graber in Packard's evolution of cel(cid:173)\nlular automata to the \"edge of chaos\"(Packard,  1988, Mitchell et aI.,  1993). \nTO-Gammon is  a major milestone for a kind of evolutionary machine learning in  which \nthe initial specification of model is far simpler than expected because the learning environ(cid:173)\nment is specified implicitly, and emerges as a result of the co-evolution between a learning \nsystem and  its  training  environment:  The learner is  embedded in  an  environment which \nresponds to its own improvements in a never-ending spiral. While this effect has been seen \nin population models, it is completely unexpected for a \"1 + 1\" hillclimbing evolution. \nCo-evolution was explored by Hillis (Hillis, 1992) on the sorting problem, by Angeline & \nPollack (Angeline  and  Pollack, 1994) on genetically programmed  tic-tac-toe players,  on \npredator/prey games, e.g.  (Cliff and Miller,  1995,  Reynolds,  1994), and  by  Juille &  Pol(cid:173)\nlack on the intertwined spirals problem (Juille and Pollack,  1995). Rosin & Belew applied \ncompetitive  fitness  to  several  games  (Rosin  and  Belew,  1995).  However,  besides \nTesauro's TO-Gammon, which has not to date been viewed as an instance of co-evolution(cid:173)\nary learning, Sims' artificial robot game (Sims,  1994) is the only other domain as complex \nas Backgammon to have had substantial success. \n\n3.2  Leamability and Unleamability \nLearnability can be formally defined as a time constraint over a search space. How hard is \nit to randomly pick 4000 floating-point weights to make a good backgammon evaluator? It \nis  simply  impossible. How  hard  is  it to  find  weights better than the current set? Initially, \nwhen all weights are random, it is quite easy. As the playing improves, we would expect it \nto get harder and harder, perhaps similar to the probability of a tornado constructing a 747 \nout of a junkyard. However, if we  search in the neighborhood of the current weights,  we \nwill find  many  players  which  make mostly the  same moves  but which can capitalize on \neach other's slightly different choices and exposed weaknesses in a tournament. \n\n\f14 \n\nJ  B.  Pollack and A.  D. Blair \n\nAlthough the setting of parameters in our initial runs  involved some guesswork, now that \nwe  have a large  set of \"players\" to  examine,  we  can try to  understand  the phenomenon. \nTaking the  1000th,  1O,000th, and loo,OOOth champions from our run, we sampled random \nplayers  in  their neighborhoods  at different  RMS  distances  to find  out how  likely  is  it  to \nfind  a  winning challenger.  We  took  1000 random neighbors at each of 11  different RMS \ndistances,  and played  them  8 games  against the  corresponding champion.  Figure 3 plots \n\nl00r---------------~----------------_, \n\n~  75 f u  50~~ . . .  =_:_::~--__ \n\"' :,:,::. :~ ..... \n\n.... \n\n.. ..........  10k \n\n\"\"\"' ,:,:,:::::~,::,::, \",. \n\nRMS distance from champion \n\nO~--------------~----------------~ \no \n0.1 \n\n0.05 \n\nFigure 3. Distance versus probability of random challenger winning \nagainst champions at generation  1,000, 10,000 and  100,000. \n\nthe  average  number of games won  against the three champions in the range of neighbor(cid:173)\nhoods.  This graph demonstrates that as  the  players  improve over time,  the probability of \nfinding good challengers in their neighborhood increases. This accounts for why  the fre(cid:173)\nquency of successful challenges  goes  up.  Each  successive challenger is  only required  to \ntake the small step of changing a few moves of the champion in order to beat it. Therefore, \nunder  co-evolution  what  was  apparently  unlearnable  becomes  learnable  as  we  convert \nfrom a single question to a continuous stream of questions, each one dependent on the pre(cid:173)\nvious answer. \n\n3.3  Avoiding Mediocre Stable States \nIn general, the problem with learning through self-play is that the player could keep play(cid:173)\ning  the  same  kinds  of games  over and  over,  only  exploring  some  narrow  region  of the \nstrategy space, missing out on critical areas of the game where it could then be vulnerable \nto other programs or human experts. Such a learning system might declare success when \nin reality it has simply converged to a \"mediocre stable state\" of continual draws or a long \nterm cooperation which merely mimics competition. Such a state can arise in human edu(cid:173)\ncation systems, where the student gets all the answers right and rewards the teacher with \npositive feedback for not asking harder questions. \nThe problem is particularly prevalent in self-play for deterministic games such as chess or \ntic-tac-toe.  We  have  worked  on  using  a  population  to  get  around  it  (Angeline  and \nPollack,  1994).  Schraudolph  et al.,  1994  added  non-determinism  to  the  game  of Go  by \nchoosing moves according to the Boltzmann distribution of statistical mechanics. Others, \nsuch  as  Fogel,  1993,  expanded exploration  by  forcing  initial  moves.  Epstein,  1994,  has \nstudied a mix of training using self-play, random testing, and playing against an expert \nin order to better understand this phenomenon. \nWe  are  not  suggesting that  1 + 1 hillclimbing  is  an  advanced machine learning  technique \nwhich others should bring to many tasks.  Without internal cognition about an  opponent's \nbehavior, co-evolution usually requires  a population.  Therefore, there must be something \nabout the dynamics of backgammon itself which is  helpful  because it permitted both TD \n\n\fWhy did TD-Gammon Work? \n\n15 \n\nlearning and hill-climbing to  succeed where they would clearly fail on other tasks and in \nother games of this scale. If we can understand why the backgammon domain led to  suc(cid:173)\ncessful acquisition of expert strategies from random initial conditions, we might be able to \nre-cast other domains in its image. \nTesauro,  1992 pointed out some of the features of Backgammon that make it suitable for \napproaches  involving  self-play  and  random  initial  conditions.  Unlike  chess,  a  draw  is \nimpossible and  a game played by an untrained network making random moves  will even(cid:173)\ntually terminate  (though it may  take much longer than  a game between competent play(cid:173)\ners). Moreover the randomness of the dice rolls leads self-play into a much larger part of \nthe search space than it would be likely to explore in a deterministic game. \nWe  believe it is  not  simply the dice rolls  which overcome the problems  of self-learning. \nOthers  have tried  to  add randomness to deterministic games  and  have  not generally  met \nwith success.  There is  something critical about the dynamics of backgammon which sets \nits  apart  from  other games  with  random  elements  like  Monopoly.  Namely,  that the  out(cid:173)\ncome of the game continues to be uncertain until all contact is  broken and one side has a \nclear advantage. What many observers find exciting about backgammon, and what helps a \nnovice sometimes overcome an expert, is the number of situations where one dice roll, or \nan improbable sequence, can dramatically reverse which player is expected to win. \nA learning system can be viewed as  a meta-game between teacher and student, which are \nidentical  in  a  self-play situation.  The teacher's  goal  is  to  expose the student's  mistakes, \nwhile the student's goal is to placate the teacher and avoid such exposure. A mediocre sta(cid:173)\nble  state for a  self-learning system can  be seen as  an  equilibrium situation  in  this  meta(cid:173)\ngame. A player which learns to repeatedly draw itself will have found  a meta-game equi(cid:173)\nlibrium and stop learning. If draws are not allowed, it may still be possible for a self-play(cid:173)\ning  learner  to  collude  with  itself - to  simulate  competition  while  actually  cooperating \n(Angeline,  1994).  For  example,  if slightly  suboptimal  moves  would  allow  a  player  to \n\"throw\"  a game,  a player under self-play  could  find  a meta-game  equilibrium  by  alter(cid:173)\nnately throwing games to itself! Our hypothesis is that the dynamics of backgammon dis(cid:173)\ncussed  above  actively  prevent  this  sort of collusion  from  forming  in  the  meta-game  of \nself-learning. \n4  CONCLUSIONS \nTesauro's  1992 result beat Sun's Gammontool and achieved parity against his own Neuro(cid:173)\ngammon  1.0,  trained  on  expert knowledge.  Neither of these  is  available.  Following  the \n1992 paper on TO-learning, he incorporated a number of hand-crafted expert-knowledge \nfeatures,  eventually  producing  a  network  which  achieved  world  master  level  play \n(Tesauro, 1995). These features included concepts like existence of a prime, probability of \nblots  being  hit,  and  probability  of escape  from  behind  the  opponent's  barrier.  Our best \nplayers  win  about  45%  against  PUBEVAL  which  was  trained  using  \"comparison \ntraining\"(Tesauro,  1989).  Therefore  we  believe  our  players  achieve  approximately  the \nsame power as Tesauro's  1992 results, without any advanced learning algorithms.  We  do \nnot  claim  that our 100,000 generation player is  as  good  as  TO-Gammon,  ready  to chal(cid:173)\nlenge the best humans, just that it is surprisingly good considering its humble origins from \nhill-climbing with a relative fitness  measure. Tuning our parameters or adding more input \nfeatures  would make more powerful players, but that is not the point of this study. \nTO-Gammon remains  a tremendous  success  in Machine Learning,  but the  causes  for  its \nsuccess have not been well understood. Replicating some ofTO-Gammon's success under \na much simpler learning paradigm, we find that the primary cause for success must be the \ndynamics  of backgammon combined  with  the  power of co-evolutionary  learning.  If we \ncan isolate the features of the backgammon domain which enable evolutionary learning to \nwork so well, it may lead to a better understanding of the conditions necessary, in general, \nfor complex self-organization. \n\n\f16 \n\n1.  B.  Pollack and A. D.  Blair \n\nAcknowledgments \nThis  work  is  supported  by  ONR  grant  NOOOI4-96-1-0418  and  a  Krasnow  Foundation \nPostdoctoral  fellowship.  Thanks  to  Gerry  Tesauro  for  providing  PUBEVAL  and  subse(cid:173)\nquent means to calibrate it, Jack Laurence and Pablo Funes for development of the WWW \nfront  end  to  our  evolved  player.  Interested  players  can  challenge  our  evolved  network \nusing a web browser through our home page at:  http://www.demo.cs.brandeis.edu \n\nReferences \nAngeline,  P.  J.  (1994).  An alternate interpretation of the  iterated prisoner's dilemma and \nthe evolution of non-mutual cooperation. In Brooks, R. and Maes, P.,  editors, Proceedings \n4th Artificial Life Conference, pages 353-358. MIT Press. \nAngeline,  P.  J.  and  Pollack,  J. B.  (1994).  Competitive  environments  evolve  better  solu(cid:173)\ntions  for  complex  tasks.  In  Forrest,  S.,  editor,  Genetic  Algorithms:  Proceedings  of the \nFifth Inter national Conference. \nBoyan, J. A.  (1992).  Modular neural networks for learning context-dependent game strat(cid:173)\negies.  Master's  thesis,  Computer Speech  and  Language  Processing,  Cambridge Univer(cid:173)\nsity. \nCliff,  D.  and  Miller,  G.  (1995).  Tracking  the  red  queen:  Measurements  of  adaptive \nprogress in co-evolutionary simulations. In Third European Conference on Artificial Life, \npages 200-218. \nHillis, D.  (1992). Co-evolving parasites improves simulated evolution as  an optimization \nprocedure.  In  C. Langton,  C. Taylor,  J.  E  and  Rasmussen,  S.,  editors,  Artificial Life  II. \nAddison-Wesley, Reading, MA. \nInternational Business Machines (Sept.  12,  1995). IBM's family funpak for osl2 warp hits \nretail shelves. \nJuille, H.  and Pollack, J. (1995). Massively parallel genetic programming. In  Angeline, P. \nand Kinnear, K., editors, Advances in Genetic Programming II.  MIT Press, Cambridge. \nMichie, D. (1961). Trial and error. In Science Survey,  part 2, pages  129-145. Penguin. \nMitchell,  M.,  Hraber,  P.  T.,  and  Crutchfield,  J. P.  (1993). Revisiting  the  edge  of chaos: \nEvolving cellular automata to perform computations. Complex Systems, 7. \nPackard,  N.  (1988).  Adaptation  towards  the  edge of chaos.  In  Kelso,  J.  A. S.,  Mandell, \nA. J.,  and  Shlesinger,  M. E, editors,  Dynamic patterns  in  complex systems,  pages  293-\n301. World Scientific. \nReynolds, C.  (1994). Competition,  coevolution,  and the game of tag.  In  Proceedings 4th \nArtificial Life Conference. MIT Press. \nRosin,  C.  D.  and  Belew,  R.  K.  (1995).  Methods  for  competitive  co-evolution:  finding \nopponents  worth  beating.  In  Proceedings of the  6th  international conference on  Genetic \nAlgorithms, pages 373-380. Morgan Kaufman. \nSamuel, A. L. (1959). some studies of machine learning using the game of checkers. IBM \n]oural of Research and Development. \nSims, K.  (1994). Evolving 3d morphology and behavior by competition. In Brooks, R.  and \nMaes, P.,  editors, Proceedings 4th Artificial Life Conference. MIT Press. \nTesauro, G.  (1989).  Connectionist learning of expert preferences by comparison training. \nIn Touretzky,  D., editor, Advances in Neural Information  Processing Systems, volume 1, \npages 99-106, Denver 1988. Morgan Kaufmann, San Mateo. \nTesauro, G.  (1992).  Practical  issues  in temporal  difference  learning. Machine  Learning, \n8:257-277. \nTesauro, G.  (1995). Temporal difference learning and td-gammon.  Communications of the \nACM, 38(3):58-68. \n\n\f", "award": [], "sourceid": 1292, "authors": [{"given_name": "Jordan", "family_name": "Pollack", "institution": null}, {"given_name": "Alan", "family_name": "Blair", "institution": null}]}