{"title": "An exploration-exploitation model based on norepinepherine and dopamine activity", "book": "Advances in Neural Information Processing Systems", "page_first": 867, "page_last": 874, "abstract": "", "full_text": " \n\n \n\nAn exploration-exploitation model based \n\non norepinepherine and dopamine \n\nactivity \n\nSamuel M. McClure*, Mark S. Gilzenrat, and Jonathan D. Cohen \n\nCenter for the Study of Brain, Mind, and Behavior \n\nsmcclure@princeton.edu; mgilzen@princeton.edu; jdc@princeton.edu \n\nPrinceton University \nPrinceton, NJ 08544 \n\nAbstract \n\nWe propose a model by which dopamine (DA) and norepinepherine \n(NE) combine to alternate behavior between relatively exploratory \nand  exploitative  modes.  The  model  is  developed  for  a  target \ndetection  task  for  which  there  is  extant  single  neuron  recording \ndata  available  from \nlocus  coeruleus  (LC)  NE  neurons.  An \nexploration-exploitation trade-off is elicited by regularly switching \nwhich  of  the  two  stimuli  are  rewarded.  DA  functions  within  the \nmodel  to  change  synaptic  weights  according  to  a  reinforcement \nlearning  algorithm.  Exploration  is  mediated  by  the  state  of  LC \nfiring,  with  higher  tonic  and  lower  phasic  activity  producing \ngreater  response  variability.  The  opposite  state  of  LC  function, \nwith lower baseline firing rate and greater phasic responses, favors \nexploitative  behavior.  Changes  in  LC  firing  mode  result  from \ncombined  measures  of  response  conflict  and  reward  rate,  where \nresponse  conflict  is  monitored  using  models  of  anterior  cingulate \ncortex (ACC). Increased long-term response conflict and decreased \nreward  rate,  which  occurs  following  reward  contingency  switch, \nfavors  the  higher  tonic  state  of  LC  function  and  NE  release.  This \nincreases exploration, and facilitates discovery of the new target. \n\n1  Introduction \n\nA central problem in reinforcement learning is determining how to adaptively move \nbetween  exploitative  and  exploratory  behaviors  in  changing  environments.    We \npropose a  set of neurophysiologic  mechanisms whose  interaction  may mediate  this \nbehavioral  shift.  Empirical  work  on  the  midbrain  dopamine  (DA)  system  has \nsuggested  that  this  system  is  particularly  well  suited  for  guiding  exploitative \nbehaviors.  This hypothesis has been reified by  a number of studies showing  that  a \ntemporal  difference  (TD)  learning  algorithm  accounts  for  activity  in  these  neurons \nin  a  wide  variety  of  behavioral  tasks  [1,2].  DA  release  is  believed  to  encode  a \nreward  prediction  error  signal  that  acts  to  change  synaptic  weights  relevant  for \nproducing  behaviors  [3].  Through  learning,  this  allows  neural  pathways  to  predict \nfuture  expected  reward  through  the  relative  strength  of  their  synaptic  connections \n\n\f \n\n[1].  Decision-making  procedures  based  on  these  value  estimates  are  necessarily \ngreedy.  Including  reward  bonuses  for  exploratory  choices  supports  non-greedy \nactions [4] and accounts for additional data derived from DA neurons [5]. We show \nthat combining a DA learning algorithm with models of response conflict detection \n[6]  and  NE  function  [7]  produces  an  effective  annealing  procedure  for  alternating \nbetween exploration and exploitation.  \nNE neurons within the LC alternate between two firing modes [8]. In the first mode, \nknown as the phasic mode, NE neurons fire at a low baseline rate but have relatively \nrobust phasic responses to behaviorally salient stimuli. The second mode, called the \ntonic  mode,  is  associated  with  a  higher  baseline  firing  and  absent  or  attenuated \nphasic responses. The effects of NE on efferent areas are modulatory in nature, and \nare  well  captured  as  a  change  in  the  gain  of  efferent  inputs  so  that  neuronal \nresponses are potentiated in the presence of NE [9]. Thus, in phasic mode,  the  LC \nprovides  transient  facilitation  in  processing,  time-locked  to  the  presence  of \nbehaviorally  salient  information  in  motor  or  decision  areas.  Conversely,  in  tonic \nmode, higher overall LC discharge rate increases gain generally and hence increases \nthe  probability  of  arbitrary  responding.  Consistent  with  this  account,  for  periods \nwhen  NE  neurons  are  in  the  phasic  mode,  monkey  performance  is  nearly  perfect. \nHowever, when NE neurons are in the tonic mode, performance is more erratic, with \nincreased  response  times  and  error  rate  [8].  These  findings  have  led  to  a  recent \ncharacterization  of  the  LC  as  a  dynamic  temporal  filter,  adjusting  the  system's \nrelative responsivity to salient and irrelevant information [8]. In this way, the LC is \nideally  positioned  to  mediate  the  shift  between  exploitative  and  exploratory \nbehavior. \nThe parameters that underlie changes in LC firing mode remain largely unexplored. \nBased on data from a target detection task by Aston-Jones and colleagues [10], we \npropose that LC firing mode is determined in part by measures of response conflict \nand reward rate as calculated by the ACC and OFC, respectively [8]. Together, the \nACC and OFC are the principle sources of cortical input to the LC [8]. Activity  in \nthe ACC is known, largely through human neuroimaging experiments, to change in \naccord  with  response  conflict  [6].  In  brief,  relatively  equal  activity  in  competing \nbehavioral  responses  (reflecting  uncertainty)  produces  high  conflict.  Low  conflict \nresults  when  one  behavioral  response  predominates.  We  propose  that  increased \nlong-term  response  conflict  biases  the  LC  towards  a  tonic  firing  mode.  Increased \nconflict  necessarily  follows  changes  in  reward  contingency.  As  the  previously \nrewarded  target  no  longer  produces  reward,  there  will  be  a  relative  increase  in \nresponse  ambiguity  and  hence  conflict.  This  relationship  between  conflict  and  LC \nfiring is analogous to other modeling work [11], which proposes that increased tonic \nfiring reflects increased environmental uncertainty. \nAs  a  final  component  to  our  model,  we  hypothesize  that  the  OFC  maintains  an \nongoing estimate in reward rate, and that this estimate of reward rate also influences \nLC firing mode. As reward rate increases, we assume that the OFC tends to bias the \nLC in favor of phasic firing to target stimuli. \nWe  have  aimed  to  fix  model  parameters  based  on  previous  work  using  simpler \nnetworks. We use parameters derived primarily from a previous model of the LC by \nGilzenrat  and  colleagues  [7].  Integration  of  response  conflict  by  the  ACC  and  its \ninfluence  on  LC  firing  was  borrowed  from  unpublished  work  by  Gilzenrat  and \ncolleagues  in  which  they  fit  human  behavioral  data  in  a  diminishing  utilities  task. \nGiven this approach, we interpret our observed improvement in model performance \nwith combined NE and DA function as validation of a mechanism for automatically \nswitching between exploitative and exploratory action selection. \n\n\f \n\n2  Go-No-Go Task and Core Model \n\nWe  have  modeled  an  experiment  in  which  monkeys  performed  a  target  detection \ntask [10]. In the task, monkeys were shown either a vertical bar or a horizontal bar \nand  were  required  to  make or  omit  a  motor  response  appropriately.  Initially,  the \nvertical  bar  was  the  target  stimulus  and  correctly  responding  was  rewarded  with  a \nsquirt  of  fruit  juice  (r=1  in  the  model).  Responding  to  the non-target  horizontal \nstimulus resulted in time out punishment (r=-.1; Figure 1A). No responses to either \nthe target or non-target gave zero reward. \nAfter  the  monkeys  had  fully acquired  the  task,  the  experimenters  periodically \nswitched the reward contingency such that the previously rewarded stimulus (target) \nbecame  the  distractor, and  vice  versa.  Following  such  reversals,  LC  neurons  were \nobserved to change from emitting phasic bursts of firing to the target, to tonic firing \nfollowing the switch, and slowly back to phasic firing for the new target as the new \nresponse criteria was obtained [10]. \n  \n\n \n\nFigure 1: Task and model design. (A) Responses were required for targets  in order \nto  obtain  reward.  Responses  to  distractors  resulted  in  a  minor  punishment.  No \nresponses gave zero reward. (B) In the model, vertical and horizontal bar inputs (I1 \nand I2) fed to integrator neurons (X1 and X2) which then drove response units (Y1 and \nY2).  Responses  were  made  if  Y1  or  Y2  crossed  a  threshold  while  input  units  were \nactive. \n \nWe  have  previously  modeled  this  task  [7,12]  with  a  three-layer  connectionist \nnetwork  in  which  two  input  units,  I1  and  I2,  corresponding  to  the  vertical  and \nhorizontal  bars,  drive  two  mutually  inhibitory  integrator  units,  X1  and  X2.  The \nintegrator  units  subsequently  feed  two  response  units,  Y1  and  Y2  (Figure  1B). \nResponses  are  made  whenever  output  from  Y1  or  Y2  crosses  a  threshold  level  of \nactivity, \u03b8.  Relatively weak  cross  connections from  each input unit  to  the opposite \nintegrator unit (I1 to X2 and I2 to X1) are intended to model stimulus similarity. \nBoth the integrator and response units were modeled as noisy, leaky accumulators: \n \n\n\u02d9 X i = \"X i +wX i I i\n\nI i + wX i I j\n\nI j \" wX i X j\n\nf (X j ) +#i \n\n(1) \n\n \n\n\u02d9 Y i = \"Yi +wYi X i\n\nf (X i) \" wYiY j\n\nf (Y j ) +#i. \n\n(2) \n\n! \n\nThe  \u03bei  terms  represent  stochastic  noise  variables.  The  response  function  for  each \nunit is sigmoid with gain, gt, determined by current LC activity (Eq. 9, below) \n \n\n! \n\nf (X) = 1 + e\"g t X \" b\n\n(\n\n(\n\n\"1. \n\n)\n\n)\n\n(3) \n\nResponse units, Y, were given a positive bias, b, and integrator units were unbiased. \nAll weight values, biases, and variance of noise are as reported in [7]. \n\n! \n\n\f \n\nIntegration  was  done  with  a  Euler  method  at  time  steps  of  0.02.  Simulation  of \nstimulus presentations involved setting one of the input units to a value of 1.0 for 20 \nunits  of  model  time.  Activation  of  I1  and  I2  were  alternated  and  20  units  of  model \ntime  were  allowed  between  presentations  for  the  integrator  and  response  units  to \nrelax to baseline levels of activity. Input 1 was initially set to be the target and input \n2  the  distractor.  After  50  presentations  of  I1  and  I2  the  reward  contingencies  were \nswitched;  the  model  was  run  through  6  such  blocks  and  reversals.  The  response \nduring  each  stimulus  presentation  was  determined  by  which  of  the  two  response \nunits first crossed a threshold of output activity (i.e. f(Y1) > \u03b8), or was a no response \nif neither unit crossed threshold. \n\n3  Performance of model with DA-mediated learning \n\nIn  order  to  obtain  a  benchmark  level  of  performance  to  compare  against,  we  first \ndetermined  how  learning  progresses  with  DA-mediated  reinforcement  learning \nalone. A reward unit,  r,  was included that had  activity  0 except  at the  end of each \nstimulus  presentation  when  its  activity  was  set  equal  to  the  obtained  reward \noutcome. Inhibitory inputs from  the response units served as  measures of  expected \nreward. At the end of every trial, the DA unit, \u03b4, obtained a value given by \n \n\n(4) \n\n \n\nwhere Z(Y) is a threshold function that is 1 if f(Y)\u2265\u03b8 and is 0 otherwise. \nThe output of dopamine neurons was used to update the weights along the pathway \nthat  lead  to  the  response.  Thus,  at  the  end  of  every  stimulus  presentation,  the \nweights between response units and DA neurons were updated according to \n \n\n(5) \n\n \n\nwhere  the  learning  rate,  \u03bb,  was  set  to  0.3  for  all  simulations.  This  learning  rule \nallowed the weights to converge to the expected reward for selecting each of the two \nactions. Weights between integrator and response units were updated using the same \nrule as in Eq. 5, except the weights were restricted to a minimum value of 0.8. When \nthe  weight  values  were  allowed  to  decrease  below  0.8,  sufficient  activity  never \naccumulated in the response units to allow discovery to new reward contingencies. \nAs  the  model  learned,  the  weights  along  the  target  pathway  obtained  a  maximum \nvalue  while  those  along  the  distractor  pathway  obtained  a  minimum  value.  After \nreversals,  the  model  initially  adapted  by  reducing  the  weights  along  the  pathway \nassociated with the previous target. The only way the model was able to obtain the \nnew  target  was  by  noise  pushing  the  new  target  response  unit  above  threshold. \nBecause of this, the performance of the model was greatly dependent of the value of \nthe  threshold  used  in  the  simulation  (Figure  2B).  When  the  threshold  was  low \nrelative  to  noise,  the  model  was  able  to  quickly  adapt  to  reversals.  However,  this \nalso resulted in a high rate of responding to non-target stimuli even after learning. In \norder  to  reduce  responding  to  the  distractor,  the  threshold  had  to  be  raised,  which \nalso increased the time required to adapt following reward reversals. \nThe  network  was  initialized  with  equal  preference  for  responding  to  input  1  or  2, \nand generally  acquired  the initial  target faster  than  after reversals (see Figure 2B). \nBecause  of  this,  all  subsequent  analyses  ignore  this  first  learning  period.  For  each \nvalue  of  threshold  studied,  we  ran  the  model  100  times.  Plots  shown  in  Figures  2 \nand  3  show  the  probability  that  the  model  responded,  when  each  input  was \nactivated, as a function of trial number (i.e. P(f(Yi)\u2265\u03b8 | Ii=1)). \n \n\n! \"(t)=r(t)#w\"Y1Z(Y1(t))#w\"Y2Z(Y2(t))! w\"Yi(t+1)=w\"Yi(t)+#\"(t)Z(Yi)\f \n\nFigure 2: Model performance with DA alone. (A) DA neurons, \u03b4, modulated weights \nfrom integrator to response units in order to modulate the probability of responding \nto  each  input.  (B)  The  model  successfully  increases  and  decreases  responding  to \ninputs  1  and  2  as  reward  contingencies  reverse.  However,  the  model  is  unable  to \nsimultaneously obtain the new response quickly and maintain a low error rate once \nthe  response  is  learned.  When  threshold  is  relatively  low  (left  plot),  the  model \nadapts quickly but  makes frequent responses  to  the distractor. At higher  threshold, \nresponses  are  correctly  omitted  to  the  distractor,  but  the  model  acquires  the  new \nresponse slowly. \n\n4  Improvement with NE-mediated annealing \n\nWe  used  the  FitzHugh-Nagumo  set  of  differential  equations  to  model  LC  activity. \n(These equations are generally used to model individual neurons, but we use them to \nmodel the  activity  in  the nucleus  as a whole.)  Previous  work has  shown that  these \nequations,  with simple modifications,  capture  the fundamental aspects of  tonic  and \nphasic  mode  activity  in  the  LC  [7].  The  FitzHugh-Nagumo  equations  involve  two \ninteracting  variables  v  and  u,  where  v  is  an  activity  term  and  u  is  an  inhibitory \ndampening  term.  The  output  of  the  LC  is  given  by  the  value  of  u,  which \nconveniently  captures  the  fact  that  the  LC  is  self-inhibitory  and  that  the  post-\nsynaptic effect of NE release is somewhat delayed [7]. \nThe model included two inputs to the LC from the integrator units (X1 and X2) with \nmodifiable weights. The state of the LC is then given by \n \n\n(6) \n\n \n\n \n\n \n\n(7) \n\n \nwhere the function h is defined by \n \n(8) \nand  governs  the  firing  mode  of  the  LC.  In  order  to  change  firing  mode,  h  can  be \nmodified so that the dynamics of u depend entirely on the state of the LC or so that \nthe dynamics are independent of state. This alternation is governed by the parameter \nC.  When  C  is  equal  to  1.0,  the  model  is  appropriately  dampened  and  can  burst \nsharply and return to a relatively low baseline level of activity (phasic mode). When \nC is small, the LC receives a fixed level of inhibition, which simultaneously reduces \nbursting activity and increases baseline activity (tonic mode) [7]. \nThe  primary  function  of  the  LC  in  the  model  is  to  modify  the  gain,  g,  of  the \nresponse function of the integrator and response units as in equation 3. We let gain \nbe a linear function of u with base value G and dependency on u given by k \n \n\n(9) \n\n. \n\n! \"v\u02d9 v =v(#$v)(v$1)$u+wvX1f(X1)+wvX2f(X2)! \"u\u02d9 u =h(v)#u! h(v)=Cv+(1\"C)d! gt=G+kut\fThe value of  C  was updated  after  every  trial by  measures of response conflict and \nreward rate. Response conflict was calculated as a normalized measure of the energy \nin the response units during the trial. For convenience,  define Y1 to be a vector of \nthe activity in unit Y1 at each point of time during a trial, f(Y1(t)). Let Y2 be defined \nsimilarly. The conflict during the trial is \n \n\n \n\n \n\n(10) \n\nwhich correctly measures energy since Y1 and Y2 are connected with weight \u20131. This \nnormalization procedure was necessary  to account for changes  in the  magnitude of \nY1 and Y2 activity due to learning. \nBased on previous work [8], we  let conflict modify  C separately based on  a short-\nterm,  KS,  and  long-term,  KL,  measure.  The  variable  KS  was  updated  at  the  end  of \nevery Tth trial according to \n \n(11) \nwhere  \u03b5S  was  0.2  and  KS(T+1)  was  used  to  calculate  the  value  of  C  used  for  the \nT+1th  trial.  KL  was  update  with  the  same  rule  as  KS  except  \u03b5L  was  0.05.  We  let \nshort-  and  long-term  conflict  have  opposing  effect  on  the  firing  mode  of  the  LC. \nThis was developed previously to capture human behavior in a diminishing utilities \ntask.  When  short-term  conflict  increases,  the  LC  is  biased  towards  phasic  firing \n(increased  C).  This  allows  the  model  to  recover  from  occasional  errors.  However, \nwhen long-term conflict increases this is taken to indicate that the  current decision \nstrategy is not working. Therefore, increased long-term conflict biases the LC to the \ntonic mode so as to increase response volatility. \n \n\n. \n\nFigure  3:  Model  performance  with  DA  and  NE.  (A)  The  full  model  includes  a \nconflict detection unit,  K, and a reward rate  measure, R, which  combine  to modify \nactivity in the LC. The LC modifies the gain in the integrator and response units. (B) \nThe  benefit  of  including  the  LC  in  the  model  is  insignificant  when  the  response \nthreshold is regularly crossed by noise alone, and hence when the error rate is high. \n(C)  However,  when  the  threshold  is  greater  and  error  rate  lower,  NE  dramatically \nimproves the rate at which the new reward contingencies are learned after reversal. \n \nReward rate, R, was updated at the end of every trial according to \n \n(12) \nwhere r is the reward earned on the Tth trial. Increased reward rate was assumed to \nbias the LC to phasic firing. \nReward rate, short-term conflict, and long-term conflict updated C according to \n \n\n(13) \n\n \n\n \n\n! K=Y1\"Y2Y1Y2! KS(T+1)=(1\"#S)KS(T)+#SK(T)! R(T+1)=(1\"#R)R(T)+#Rr! C=\"(KS)1#\"(KL)()\"(R)\f \n\nwhere each \u03c3 is a sigmoid function with a gain of 6.0 and no bias as determined by \nfitting to behavior with previous models. \nAs  with  the  model  with  DA  alone,  the  effect  of  NE  depended  significantly  on  the \nvalue  of  the  threshold \u03b8.  When \u03b8  was  small,  the  improvement  afforded  by  the  LC \nwas negligible (Figure 3B). However, when  the  threshold was significantly greater \nthan noise, the improvement was substantial (Figure 3C). \nMonkeys  were  able  to  perform  this  task  with  accuracy  greater  than  90%  and \nsimultaneously  were  able  to  adapt  to  reversals  within 50  trials  [10].  While  it  is \nimpossible to compare the output of our model with monkey behavior, we can make \nthe  qualitative  assertion  that,  as  with  monkeys,  our  NE-based  annealing  model \nallows  for  high  accuracy  (and  high  threshold)  decision-making while  preserving \nadaptability to changes in reward contingencies. In order to better demonstrate this \nimprovement,  we  fit  single  exponential  curves  to  the  plots  of  probability  of \naccurately responding to the new target by trial number (as in Figure 3B,C). Shown \nin  Figure  4  is  the  time  constant  for  these exponential  fits,  which  we  term  the \ndiscovery  time  constant,  for  different  values  of  the  threshold. As  can  be  seen,  the \nmodel with NE-mediated annealing  maintains a relatively fast discovery  time even \nas the threshold becomes relatively large. \n  \n\nFigure 4: Summary of model performance with and without NE. \n\n \n\n5  Discussion\n\nWe  have  demonstrated  that  a  model  incorporating  behavioral  and  learning  effects \npreviously  ascribed to DA  and  NE  produces  an  adaptive  mechanism  for  switching \nbetween exploratory and exploitative decision-making. Our model uses measures of \nresponse  conflict  and  reward  rate  to  modify LC  firing  mode,  and  hence  to  change \nnetwork dynamics in favor of more or less volatile behavior. In essence, combining \nprevious  models  of  DA  and  NE  function  produces  a  performance-based  auto-\nannealing algorithm. \nThere  are  several  limitations  to  this  model  that  can  be  remedied  by  greater \nsophistication  in  the  learning  algorithm.  The  primary  limitation  is  that  the  model \nvaries between more or less volatile action selection only over the range of reward \nrelevant  to  our  studied  task.  Model  parameters  could  be  altered  on  a  task-by-task \nbasis to correct this; however, a more general scheme may be accomplished with a \nmean reward learning algorithm [13]. It has previously been argued that DA neurons \nmay actually  emit  an  average  reward  TD  error  [14].  This  change  may  require \nallowing both short- and long-term reward rate control the LC firing mode (Eq. 13). \nAnother  limitation  of  this  model  is  that,  while  exploration  is  increased  as \nperformance measures wane, exploration is not managed intelligently. This does not \nsignificantly affect the performance of our model since there are only two available \nactions.  As  the  number  of  alternatives  increases,  rapid  learning  may  require \nsomething akin to reward bonuses [4,5]. \n\n\f \n\nUnderstanding the interplay between DA and NE function in learning and decision-\nmaking  is  also  relevant  for  understanding  disease.  Numerous  psychiatric  disorders \nare  known  to  involve  dysregulation  of  NE  and  DA  release.  Furthermore,  hallmark \nfeatures of ADHD and schizophrenia include cognitive disorders in which behavior \nappears  either  too  volatile  (ADHD)  or  too  inflexible  (schizophrenia)  [15,16]. \nImproved models of DA-NE interplay during learning and decision-making, coupled \nwith  empirical  data,  may  simultaneously  improve  knowledge  of  how  the  brain \nhandles the exploration-exploitation dilemma and how this goes awry in disease. \n\nAckno wl edg ments \nThis work was supported by NIH grants P50 MH62196 and MH065214. \n\nRefer en ces \n[1]  Montague,  P.R.  Dayan,  P.,  Sejnowski,  T.J.  (1996)  A  framework  for  mesencephalic \ndopamine systems based on predictive Hebbian learning. J. Neurosci. 16: 1936-1947. \n[2]  Schultz,  W.  Dayan,  P.  &  Montague,  P.R.  (1997)  A  neural  substrate  for  prediction  and \nreward. Science 275: 1593-1599. \n[3]  Reynolds,  J.N.,  Hyland,  B.I.,  Wickens,  J.R.  (2001)  A  cellular  mechanism  of  reward-\nrelated learning. Nature 413: 67-70. \n[4] Sutton, R.S. (1990) Integrated architectures for learning, planning, and reacting based on \napproximated dynamic programming. Mach. Learn., Proc. 7th International Conf. 216-224. \n[5]  Kakade,  S.,  Dayan,  P.  (2002)  Dopamine:  generalization  and  bonuses.  Neural  Networks \n15: 549-559. \n[6]  Botvinick,  M.M.,  Braver,  T.S.,  Barch,  D.M.,  Carter,  C.S.,  Cohen,  J.D.  (2001)  Conflict \nmonitoring and cognitive control. Psychol. Rev. 108: 624-652. \n[7]  Gilzenrat,  M.S.,  Holmes,  B.D.,  Rajkowski,  J.,  Aston-Jones,  G.,  Cohen,  J.D.  (2002) \nSimplified  dynamics  in  a  model  of  noradrenergic  modulation  of  cognitive  performance. \nNeural Networks 15: 647-663. \n[8]  Aston-Jones,  G.,  Cohen,  J.D.  (2005)  An  integrative  theory  of  locus  coeruleus-\nnorepinepherine function. Ann. Rev. Neurosci. 28: 403-450. \n[9] Servan-Schreiber, D., Printz, H., Cohen, J.D. (1990) A network model of catecholamine \neffects: gain, signal-to-noise ratio and behavior. Science 249: 892-895. \n[10]  Aston-Jones,  G.,  Rajkowski,  J.,  Kubiak,  P.  (1997)  Conditioned  responses  of  monkey \nlocus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. \nNeuroscience 80: 697-715. \n[11] Yu, A., Dayan, P. (2005) Uncertainty, neuromodulation and attention. Neuron 46: 681-\n92. \n[11]  Usher,  M.,  Cohen,  J.D.,  Rajkowski,  J.,  Aston-Jones,  G.  (1999)  The  role  of  the  locus \ncoeruleus in the regulation of cognitive performance. Science 283: 549-554. \n[12]  Schwartz,  A.  (1993)  A  reinforcement  learning  method  for  maximizing  undiscounted \nrewards.  In:  Proc.  10th  International  Conf.  Mach.  Learn.  (pp.  298-305).  San  Mateo,  CA: \nMorgan Kaufmann. \n[13] Daw, N.D., Touretzky, D.S.  (2002) Long-term  reward prediction in TD models of the \ndopamine system. Neural Computation 14: 2567-2583. \n[14]  Goldberg,  T.E.,  Weinberger,  D.R.,  Berman,  K.F.,  Pliskin,  N.H.,  Podd,  M.H.  (1987) \nFurther  evidence  for  dementia  of  the  prefrontal  type  in  schizophrenia?  A  controlled  study \nteaching the Wisconsin Card Sorting Test. Arch. Gen. Psychiatry 44: 1008-1014. \n[15]  Barkley,  R.A.  (1997)  Behavioural  inhibition,  sustained  attention,  and  executive \nfunctions: constructing a unified theory of AD/HD. Psychol. Bull. 121: 65-94. \n\n\f", "award": [], "sourceid": 2950, "authors": [{"given_name": "Samuel", "family_name": "McClure", "institution": null}, {"given_name": "Mark", "family_name": "Gilzenrat", "institution": null}, {"given_name": "Jonathan", "family_name": "Cohen", "institution": null}]}