{"title": "Different Cortico-Basal Ganglia Loops Specialize in Reward Prediction at Different Time Scales", "book": "Advances in Neural Information Processing Systems", "page_first": 701, "page_last": 708, "abstract": "", "full_text": "Different Cortico-Basal Ganglia Loops \nSpecialize in Reward Prediction on    \n\nDifferent Time Scales  \n\n \n\n \n\nSaori Tanaka    \nNara Institute of Science and Technology \n\n                               Kenji Doya \n\nATR Computational Neuroscience Laboratories \n\nCREST, Japan Science and Technology Corporation \n\nxsaori@atr.co.jp                      doya@atr.co.jp \n\nKyoto, Japan \n\nGo Okada           Kazutaka Ueda               Yasumasa Okamoto      Shigeto Yamawaki \n\nHiroshima University School of Medicine \n\nCREST, Japan Science and Technology Corporation \n\nHiroshima, Japan \n\nAbstract \n\nTo understand the brain mechanisms involved in reward prediction \non different time scales, we developed a Markov decision task that \nrequires prediction of both immediate and future rewards, and ana-\nlyzed subjects\u2019 brain activities using functional MRI. We estimated \nthe time course of reward prediction and reward prediction error on \ndifferent time scales from subjects' performance data, and used them \nas  the  explanatory  variables  for  SPM  analysis.  We  found  topog-\nraphic  maps  of  different  time  scales  in  medial  frontal  cortex  and \nstriatum.  The  result  suggests  that  different  cortico-basal  ganglia \nloops are specialized for reward prediction on different time scales. \n\n1  Introduction \nIn our daily life, we make decisions based on the prediction of rewards on different \ntime scales; immediate and long-term effects of an action are often in conflict, and \nbiased evaluation of immediate or future outcome can lead to pathetic behaviors.  \nLesions in the central serotonergic system result in impulsive behaviors in humans [1], \nand animals [2, 3], which  can be attributed to deficits in reward prediction on a long \ntime scale. Damages  in  the ventral part  of  medial  frontal  cortex  (MFC)  also cause \ndeficits in decision-making that requires assessment of future outcomes [4-6]. \nA possible mechanism underlying these observations is that different brain areas are \nspecialized  for  reward  prediction  on  different  time  scales,  and  that  the  ascending \n\n\f \n\nserotonergic system activates those specialized for predictions in longer time scales \n[7].  \nThe  theoretical  framework  of  temporal  difference  (TD)  learning  [8]  successfully \nexplains reward-predictive activities of the midbrain dopaminergic system as well as \nthose  of  the  cortex  and  the  striatum  [9-13].  In  TD  learning  theory,  the  predicted \namount of future reward starting from a state s(t) is formulated as the \u201cvalue function\u201d \nV(t) = E[r(t + 1) + \u03b3 r(t + 2) + \u03b3 2r(t + 3) + \u2026]                    (1) \n\nand learning is based on the TD error \n\n\u03b4(t) = r(t) + \u03b3 V(t) \u2013 V(t - 1).                                     (2) \nThe \u2018discount factor\u2019 \u03b3  controls the time scale of prediction; while only the immediate \nreward r(t + 1) is considered with \u03b3 = 0, rewards in the longer future are taken into \naccount with \u03b3  closer to 1. \nIn order to test the above hypothesis [7], we developed a reinforcement learning task \nwhich requires a large value of discount factor for successful performance, and ana-\nlyzed subjects\u2019 brain activities using functional MRI. In addition to conventional \nblock-design analysis, a novel model-based regression analysis revealed topographic \nrepresentation of prediction time scale with in the cortico-basal ganglia loops. \n\n2  Methods \n\n2.1  Markov Decision Task \n\nIn the Markov decision task (Fig. 1), markers on the corners of a square present four \nstates, and the subject selects one of two actions by pressing a button (a1 = left button, \na2 = right button) (Fig. 1A). The action determines both the amount of reward and the \nmovement  of  the  marker  (Fig.  1B).  In  the  REGULAR  condition,  the  next  trial  is \nstarted from the marker position at the end of the previous trial. Therefore, in order to \nmaximize the reward acquired in a long run, the subject has to select an action by \ntaking into account both the immediate reward and the future reward expected from \nthe  subsequent  state.  The  optimal  behavior  is  to  receive  small  negative  rewards  at \nstates  s2,  s3,  and  s4  to  obtain  a  large  positive  reward  at  state  s1  (Fig.  1C).  In  the \nRANDOM condition, next trial is started from a random marker position so that the \nsubject has to consider only immediate reward. Thus, the optimal behavior is to col-\nlect a larger reward at each state (Fig. 1D). In the baseline condition (NO condition), \nthe reward is always zero.  \nIn order to learn the optimal behaviors, the discount factor \u03b3  has to be larger than \n0.3425 in REGULAR condition, while it can be arbitrarily small in RANDOM con-\ndition. \n\n2.2 \n\nfMRI imaging \n\nEighteen healthy, right-handed volunteers (13 males and 5 females), gave informed \nconsent to take part in the study, with the approval of the ethics and safety committees \nof ATR and Hiroshima University. \n\n\f \n\nA\n\nTime\n\n0\n\n1.0\n\n2.0\n\n2.5\n\n3.0\n\n4.0 (s)\n\n 100\n\n 100\n\nB\n\ns2\n\n-r2\n\n+r 1\n\ns4\n\n+r 2\n\n-r1\n\n-r1\n\n-r1\n\ns1\n\n+r 1\n\n-r2\n\na1\n\na2\n\ns3\n\nr1  = 20 10 yen\nr2 = 100 10 yen\n\nC\n\nREGULAR condition\n\nD\n\nRANDOM condition\n\ns2\n\ns4\n\n+r 2\n\n-r1\n\n-r1\n\n-r1\n\ns1\n\ns2\n\n+r 1\n\ns3\n\ns4\n\n+r 2\n\n-r1\n\n-r1\n\ns1\n\ns3\n\nFig. 1. (A) Sequence of stimulus and response events in the Markov decision \ntask. First, one of four squares representing present state turns green (0s). As \nthe fixation point turns green (1s), the subject presses either the right or left \nbutton within 1 second. After 1s delay, the green square changes its position \n(2s), and then a reward for the current action is presented by a number (2.5s) \nand  a  bar  graph  showing  cumulative  reward  during  the  block  is  updated \n(3.0s). One trial takes four seconds. Subjects performed five trials in the NO \ncondition, 32 trials in the RANDOM condition, five trials in the NO condi-\ntion, and 32 trials in the REGULAR condition in one block. They repeated \nfour blocks; thus, the entire experiment consisted of 312 trials, taking about \n20 minutes.   (B) The rule of the reward and marker movement. (C) In the \nREGULAR condition, the optimal behavior is to receive small negative re-\nwards \u2013r1 (-10, -20, or -30 yen) at states s2, s3, and s4 to obtain a large positive \nreward +r2 (90, 100, or 110 yen) at state s1. (D) In the RANDOM condition, \nthe next trial is started from random state. Thus, the optimal behavior is to \nselect a larger reward at each state. \n\nA 1.5-Tesla scanner (Marconi, MAGNEX ECLIPSE, Japan) was used to acquire both \nstructural T1-weighted images (TR = 12 s, TE = 450 ms, flip angle = 20 deg, matrix = \n256 \u00d7 256, FoV = 256 mm, thickness = 1 mm, slice gap = 0 mm ) and T2*-weighted \necho planar images (TR = 4 s, TE = 55 msec, flip angle = 90 deg, 38 transverse slices, \nmatrix = 64 \u00d7 64, FoV = 192 mm, thickness = 4 mm, slice gap = 0 mm, slice gap = 0 \nmm) with blood oxygen level-dependent (BOLD) contrast. \n\n2.3  Data analysis \n\n The data were preprocessed and analyzed with SPM99 (Friston et al., 1995; Well-\ncome Department of Cognitive Neurology, London, UK). The first three volumes of \nimages were discarded to avoid T1 equilibrium effects. The images were realigned to \nthe  first  image  as  a  reference,  spatially  normalized  with  respect  to  the  Montreal \nNeurological Institute EPI template, and spatially smoothed with a Gaussian kernel (8 \nmm, full-width at half-maximum). \n\n\f \n\nA\n\nlarger\nreward\n\nn\no\n\ni\nt\nc\na\n\nsmaller\nreward\n\n1\n\nB\noptimal\n\nn\no\n\ni\nt\nc\na\n\nnon-\n\noptimal\n\n1\n\nRANDOM condition\n\n32\n\n32\n\n64\ntrial\n\nREGULAR condition\n\n96\n\n128\n\n64\ntrial\n\n96\n\n128\n\nFig. 2. The selected action \nof a representative single \nsubject (solid line) and the \ngroup average ratio of \nselecting optimal action \n(dashed line) in (A) RAN-\nDOM and (B) REGULAR \nconditions. \n\nImages of parameter estimates for the contrast of interest were created for each subject. \nThese  were  then  used  for  a  second-level  group  analysis  using  a  one-sample  t-test \nacross the subjects (random effects analysis). \nWe  conducted  two  types  of  analysis.  One  was  block  design  analysis  using  three \nboxcar regressors convolved with a hemodynamic response function as the reference \nwaveform  for  each  condition  (RANDOM,  REGULAR,  and  NO).  The  other  was \nmultivariate  regression  analysis  using  explanatory  variables,  representing  the  time \ncourse of the reward prediction V(t) and reward prediction error \u03b4(t) estimated from \nsubjects\u2019 performance data (described below), in addition to three regressors repre-\nsenting the condition of the block.  \n\n2.4  Estimation of predicted reward V(t) and prediction error \u03b4(t) \n\nThe time course of reward prediction V(t) and reward prediction error \u03b4(t) were es-\ntimated from each subject\u2019s performance data, i.e. state s(t), action a(t), and reward \nr(t), as follows. \nIf the subject starts from a state s(t) and comes back to the same state after k steps, the \nexpected cumulative reward V(t) should satisfy the consistency condition \n\nV(t) = r(t + 1) + \u03b3 r(t + 2) + \u2026 + \u03b3 k-1r(t + k) + \u03b3 kV(t).                 (3) \nThus, for each time t of the data file, we calculated the weighted sum of the rewards \nacquired until the subject returned to the same state and estimated the value function \nfor that episode as \n\n(\nr t\n\n)\n1\n+ +\n\n(\nr t\n\u03b3\n\n\uf8ee\n\uf8f0\n\n\u02c6\n( )\nV t\n\n=\n\n)\n2\n...\n+\n+ +\nk\n1\n\u03b3\n\u2212\n\nk\n\u03b3\n\n(\nr t\n1\n\u2212\n\n+\n\nk\n\n)\n\n\uf8f9\n\uf8fb\n\n \n\n.\n\n(4) \n\nThe estimate of the value function V(t) at time t was given by the average of all pre-\nvious episodes from the same state as at time t \n\n( )\nV t\n\nL\n\n1\n= \u2211\nL =\n\nl\n\n1\n\n\u02c6\n( )\nV t\n\nl\n\n,                                              (5) \n\nwhere {t1, \u2026, tL} are the indices of time visiting the same state as s(t), i.e. s(t1) = \u2026 = \ns(tL) = s(t). The TD error was given by the difference between the actual reward r(t) \nand the temporal difference of the value function V(t) according to equation (2). \nAssuming  that  different  brain  areas  are  involved  in  reward  prediction  on  different \ntime scales, we varied the discount factor \u03b3  as 0, 0.3, 0.6, 0.8, 0.9, and 0.99. \n\n\f \n\nFig.  3.  (A)  In  REGULAR  vs.  RANDOM  comparison,  significant  activation \nwas  observed  in  DLPFC  ((x,  y,  z)  =  (46,  45,  9),  peak  t  =  4.06)  (p  <  0.001 \nuncorrected). (B) In RANDOM vs. REGULAR comparison, significant acti-\nvation was observed in lateral OFC ((x, y, z) = (-32, 9, -21), peak t = 4.90) (p < \n0.001 uncorrected). \n\n3  Results \n\n3.1  Behavioral results \n\nFigure 2 summarizes the learning performance of a representative single subject (solid \nline) and group average (dashed line) during fMRI measurement. Fourteen subjects \nsuccessfully learned to take larger immediate rewards in the RANDOM condition (Fig. \n2A) and a large positive reward at s1 after small negative rewards at s2, s3 and s4 in the \nREGULAR condition (Fig. 2B). \n\n3.2  Block-design analysis \n\nIn  REGULAR  vs.  RANDOM  contrast,  we  observed  a  significant  activation  in  the \ndorsolateral  prefrontal  cortex  (DLPFC)  (Fig.  3A)  (p  <  0.001  uncorrected).  In \nRANDOM  vs.  REGULAR  contrast,  we  observed  a  significant  activation  in  lateral \norbitofrontal cortex (lOFC) (Fig. 3B) (p < 0.001 uncorrected). \nThe  result  of  block-design  analysis  suggests  differential  involvement  of  neural \npathways in reward prediction on long and short time scales. The result in RANDOM \nvs. REGULAR contrast was consistent with previous studies that the OFC is involved \nin reward prediction within a short delay and reward outcome [14-20].  \n\n3.3  Regression analysis  \n\nWe observed significant correlation with reward prediction V(t) in the MFC, DLPFC \n(all \u03b3 ), ventromedial insula (small \u03b3 ), dorsal striatum, amygdala, hippocampus, and \nparahippocampal gyrus (large \u03b3 ) (p < 0.001 uncorrected) (Fig. 4A). We also found \nsignificant correlation with reward prediction error \u03b4(t) in the IPC, PMd, cerebellum \n(all \u03b3 ), ventral striatum (small \u03b3 ), and lateral OFC (large \u03b3 ) (p < 0.001 uncorrected) \n(Fig. 4B). As we changed the time scale parameter \u03b3 of reward prediction, we found \nrostro-caudal maps of correlation to V(t) in MFC with increasing \u03b3.  \n\n\f \n\nFig. 4. Voxels with a significant correlation (p < 0.001 uncorrected) with reward \nprediction V(t) and prediction error \u03b4(t) are shown in different colors for dif-\nferent settings of the time scale parameter (\u03b3  = 0 in red, \u03b3  = 0.3 in orange, \u03b3  = \n0.6 in yellow, \u03b3  = 0.8 in green, \u03b3  = 0.9 in cyan, and \u03b3 = 0.99 in blue). Voxels \ncorrelated with two or  more regressors are shown by a  mosaic of colors. (A) \nSignificant correlation with reward prediction V(t) was observed in the MFC, \nDLPFC, dorsal striatum, insula, and hippocampus. Note the anterior-ventral to \nposterior-dorsal  gradient  with  the  increase  in \u03b3   in  the  MFC.  (B)  Significant \ncorrelation with reward prediction error \u03b4(t) on \u03b3  = 0 was observed in the ventral \nstriatum.  \n\n4  Discussion \nIn  the  MFC,  anterior  and  ventral  part  was  involved  in  reward  prediction  V(t)  on \nshorter time scales (0 \u2264 \u03b3 \u2264 0.6), whereas posterior and dorsal part was involved in \nreward prediction V(t) on longer time scales (0.6 \u2264 \u03b3 \u2264 0.99). The ventral striatum \ninvolved  in  reward  prediction  error \u03b4(t)  on  shortest  time  scale  (\u03b3  =  0),  while  the \ndorsolateral striatum correlated with reward prediction V(t) on longer time scales (0.9 \n\u2264  \u03b3  \u2264  0.99).  These  results  are  consistent  with  the  topographic  organization  of \nfronto-striatal connection; the rostral part of the MFC project to the ventral striatum, \nwhereas the dorsal and posterior part of the cingulate cortex project to the dorsolateral \nstriatum [21]. \nIn  the  MFC and  the  striatum, no significant  difference in activity  was observed  in \nblock-design  analysis  while  we  did  find  graded  maps  of  activities  with  different \nvalues of \u03b3. A possible reason is that different parts of the MFC and the striatum are \nconcurrently involved with reward prediction on different time scales, regardless of \nthe task context. Activities of the DLPFC and lOFC, which show significant differ-\nences in block-design analysis (Fig. 3), may be regulated according to the necessity \nfor the task;  \n\n\f \n\nFrom  these  results,  we  propose  the  following  mechanism  of  reward  prediction  on \ndifferent  time  scales.  The  parallel  cortico-basal  ganglia  loops  are  responsible  for \nreward prediction on various time scales. The \u2018limbic loop\u2019 via the ventral striatum \nspecializes in immediate reward prediction, whereas the \u2018cognitive and motor loop\u2019 \nvia  the  dorsal  striatum  specialises  in  future  reward  prediction.  Each  loop  learns  to \npredict rewards on its specific time scale. To perform an optimal action under a given \ntime scale, the output of the loop with an appropriate time scale is used for actual \naction selection.  \nPrevious studies in brain damages and serotonergic functions suggest that the MFC \nand the dorsal raphe, which are reciprocally connected [22, 23], play an important role \nin  future  reward  prediction.  The  cortico-cortico  projections  from  the  MFC,  or  the \nserotonergic projections from the dorsal raphe to the cortex and the striatum may be \ninvolved in the modulation of these parallel loops. \nIn present study, using a novel regression analysis based on subjects\u2019 performance \ndata and reinforcement learning model, we revealed the maps of time scales in reward \nprediction, which could not be found by conventional block-design analysis. Future \nstudies  using  this  method  under  pharmacological  manipulation  of  the  serotonergic \nsystem  would  clarify  the  role  of  serotonin  in  regulating  the  time  scale  of  reward \nprediction. \n\nAcknowledgments \nWe  thank  Nicolas  Schweighofer,  Kazuyuki  Samejima,  Masahiko  Haruno,  Hiroshi \nImamizu, Satomi Higuchi, Toshinori Yoshioka, and Mitsuo Kawato for helpful dis-\ncussions and technical advice. \n\nReferences \n[1] Rogers, R.D., et al. (1999) Dissociable deficits in the decision-making cognition \nof chronic amphetamine abusers, opiate abusers, patients with focal damage to pre-\nfrontal  cortex,  and  tryptophan-depleted  normal  volunteers:  evidence  for  mono-\naminergic mechanisms. Neuropsychopharmacology 20(4):322-339. \n[2] Evenden, J.L. & Ryan, C.N. (1996) The pharmacology of impulsive behaviour in \nrats: the effects of drugs on response choice with varying delays of reinforcement. \nPsychopharmacology (Berl) 128(2):161-170. \n[3]  Mobini,  S.,  et  al.  (2000)  Effects  of  central  5-hydroxytryptamine  depletion  on \nsensitivity  to  delayed  and  probabilistic  reinforcement.  Psychopharmacology  (Berl) \n152(4):390-397. \n[4] Bechara, A., et al. (1994) Insensitivity to future consequences following damage \nto human prefrontal cortex. Cognition 50(1-3):7-15. \n[5]  Bechara,  A.,  Tranel,  D.  &  Damasio,  H.  (2000)  Characterization  of  the  deci-\nsion-making  deficit  of  patients  with  ventromedial  prefrontal  cortex  lesions.  Brain \n123:2189-2202. \n[6] Mobini, S., et al. (2002) Effects of lesions of the orbitofrontal cortex on sensitivity \nto  delayed \n(Berl) \n160(3):290-298. \n[7]  Doya,  K. \n15(4-6):495-506. \n\n(2002)  Metalearning  and  neuromodulation.  Neural  Netw \n\nand  probabilistic \n\nreinforcement.  Psychopharmacology \n\n\f \n\n[8] Sutton, R.S., Barto, A. G. (1998) Reinforcement learning.  Cambridge, MA: MIT \npress. \n[9] Houk, J.C., Adams, J.L. & Barto, A.G., A model of how the basal ganglia generate \nand use neural signals that predict reinforcement, in Models of information processing \nin the basal ganglia, J.C. Houk, J.L. Davis, and D.G. Beiser, Editors. 1995, MIT Press: \nCambridge, Mass. p. 249-270. \n[10] Schultz, W., Dayan, P. & Montague, P.R. (1997) A neural substrate of prediction \nand reward. Science 275(5306):1593-1599. \n[11]  Doya,  K.  (2000)  Complementary  roles  of  basal  ganglia  and  cerebellum  in \nlearning and motor control. Curr Opin Neurobiol 10(6):732-739. \n[12]  Berns,  G.S.,  et  al.  (2001)  Predictability  modulates  human  brain  response  to \nreward. J Neurosci 21(8):2793-2798. \n[13]  O'Doherty,  J.P.,  et  al.  (2003)  Temporal  difference  models  and  reward-related \nlearning in the human brain. Neuron 38(2):329-337. \n[14] Koepp, M.J., et al. (1998) Evidence for striatal dopamine release during a video \ngame. Nature 393(6682):266-268. \n[15] Rogers, R.D., et al. (1999) Choosing between small, likely rewards and large, \nunlikely  rewards  activates  inferior  and  orbital  prefrontal  cortex.  J  Neurosci \n19(20):9029-9038. \n[16] Elliott, R., Friston, K.J. & Dolan, R.J. (2000) Dissociable neural responses in \nhuman reward systems. J Neurosci 20(16):6159-6165. \n[17] Breiter, H.C., et al. (2001) Functional imaging of neural responses to expectancy \nand experience of monetary gains and losses. Neuron 30(2):619-639. \n[18] Knutson, B., et al. (2001) Anticipation of increasing monetary reward selectively \nrecruits nucleus accumbens. J Neurosci 21(16):RC159. \n[19] O'Doherty, J.P., et al. (2002) Neural responses during anticipation of a primary \ntaste reward. Neuron 33(5):815-826. \n[20] Pagnoni, G., et al. (2002) Activity in human ventral striatum locked to errors of \nreward prediction. Nat Neurosci 5(2):97-98. \n[21] Haber, S.N., et al. (1995) The orbital and medial prefrontal circuit through the \nprimate basal ganglia. J Neurosci 15(7 Pt 1):4851-4867. \n[22]  Celada,  P.,  et  al.  (2001)  Control  of  dorsal  raphe  serotonergic  neurons  by  the \nmedial  prefrontal  cortex:  Involvement  of  serotonin-1A,  GABA(A),  and  glutamate \nreceptors. J Neurosci 21(24):9917-9929. \n[23]  Martin-Ruiz,  R.,  et  al.  (2001)  Control  of  serotonergic  function  in  medial  pre-\nfrontal cortex by serotonin-2A receptors through a glutamate-dependent mechanism. J \nNeurosci 21(24):9856-9866. \n \n \n\n\f", "award": [], "sourceid": 2534, "authors": [{"given_name": "Saori", "family_name": "Tanaka", "institution": null}, {"given_name": "Kenji", "family_name": "Doya", "institution": null}, {"given_name": "Go", "family_name": "Okada", "institution": null}, {"given_name": "Kazutaka", "family_name": "Ueda", "institution": null}, {"given_name": "Yasumasa", "family_name": "Okamoto", "institution": null}, {"given_name": "Shigeto", "family_name": "Yamawaki", "institution": null}]}