{"title": "Integrated accounts of behavioral and neuroimaging data using flexible recurrent neural network models", "book": "Advances in Neural Information Processing Systems", "page_first": 4228, "page_last": 4237, "abstract": "Neuroscience studies of human decision-making abilities commonly involve\nsubjects completing a decision-making task while BOLD signals are\nrecorded using fMRI. Hypotheses are tested about which brain regions\nmediate the effect of past experience, such as rewards, on future\nactions. One standard approach to this is model-based fMRI data\nanalysis, in which a model is fitted to the behavioral data, i.e., a\nsubject's choices, and then the neural data are parsed to find brain\nregions whose BOLD signals are related to the model's internal\nsignals. However, the internal mechanics of such purely behavioral\nmodels are not constrained by the neural data, and therefore might miss\nor mischaracterize aspects of the brain. To address this limitation, we\nintroduce a new method using recurrent neural network models that are\nflexible enough to be jointly fitted to the behavioral and neural\ndata. We trained a model so that its internal states were suitably\nrelated to neural activity during the task, while at the same time its\noutput predicted the next action a subject would execute. We then used\nthe fitted model to create a novel visualization of the relationship\nbetween the activity in brain regions at different times following a\nreward and the choices the subject subsequently made. Finally, we\nvalidated our method using a previously published dataset. We found that\nthe model was able to recover the underlying neural substrates that were\ndiscovered by explicit model engineering in the previous work, and also\nderived new results regarding the temporal pattern of brain activity.", "full_text": "Integrated accounts of behavioral and neuroimaging\ndata using \ufb02exible recurrent neural network models\n\nAmir Dezfouli12# Richard Morris3\u2020 Fabio Ramos3\u2021 Peter Dayan4\u00a7 Bernard W. Balleine1\u21e4\n\n1UNSW Sydney 2Data61, CSIRO 3University of Sydney 4Gatsby Unit, UCL\n\n#akdezfuli@gmail.com \u2020richardumorris@gmail.com \u00a7p.dayan@ucl.ac.uk\n\n\u2021fabio.ramos@sydney.edu.au \u21e4bernard.balleine@unsw.edu.au\n\nAbstract\n\nNeuroscience studies of human decision-making abilities commonly involve sub-\njects completing a decision-making task while BOLD signals are recorded using\nfMRI. Hypotheses are tested about which brain regions mediate the effect of past\nexperience, such as rewards, on future actions. One standard approach to this is\nmodel-based fMRI data analysis, in which a model is \ufb01tted to the behavioral data,\ni.e., a subject\u2019s choices, and then the neural data are parsed to \ufb01nd brain regions\nwhose BOLD signals are related to the model\u2019s internal signals. However, the\ninternal mechanics of such purely behavioral models are not constrained by the\nneural data, and therefore might miss or mischaracterize aspects of the brain. To\naddress this limitation, we introduce a new method using recurrent neural network\nmodels that are \ufb02exible enough to be jointly \ufb01tted to the behavioral and neural data.\nWe trained a model so that its internal states were suitably related to neural activity\nduring the task, while at the same time its output predicted the next action a subject\nwould execute. We then used the \ufb01tted model to create a novel visualization of\nthe relationship between the activity in brain regions at different times following\na reward and the choices the subject subsequently made. Finally, we validated\nour method using a previously published dataset. We found that the model was\nable to recover the underlying neural substrates that were discovered by explicit\nmodel engineering in the previous work, and also derived new results regarding the\ntemporal pattern of brain activity.\n\n1\n\nIntroduction\n\nDecision-making circuitry in the brain enables humans and animals to learn from the consequences\nof their past actions to adjust their future choices. The role of different brain regions in this circuitry\nhas been the subject of extensive research in the past [Gold and Shadlen, 2007, Doya, 2008], with one\nof the main challenges being that decisions \u2013 and thus the neural activity that causes them \u2013 are not\nonly affected by the immediate events in the task, but are also affected by a potentially long history\nof previous inputs, such as rewards, actions and environmental cues. As an example, assume that\nsubjects make choices in a bandit task while their brain activity is recorded using fMRI, and we seek\nto determine which brain regions are involved in reward processing. Key signals, such as reward\nprediction errors, are not only determined by the current reward, but also a potentially extensive\nhistory of past inputs. Thus, it is inadequate merely to \ufb01nd brain regions showing marked BOLD\nchanges just in response to reward.\nAn in\ufb02uential approach to address the above problem has been to use model-based analysis of fMRI\ndata [e.g., O\u2019Doherty et al., 2007, Cohen et al., 2017], which involves training a computational model\nusing behavioral data and then searching the brain for regions whose BOLD activity is related to the\ninternal signals and variables of the model. Examples include \ufb01tting a reinforcement-learning model\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fto the choices of subjects (with learning-rates etc. as the model parameters) and then \ufb01nding the\nbrain regions that are related to the estimated value of each action or other variables of interest [e.g.,\nDaw et al., 2006]. One major challenge for this approach is that, even if the model produces actions\nsimilar to the subjects, the variables and summary statistics that the brain explicitly tracks might not\ntransparently represent the ones the hypothetical model represents. In this case, either the relevant\nsignals in the brain will be missed in the analysis, or the model will have to be altered manually in\nthe hope that the new signals in the model resemble neural activity in the brain.\nIn contrast, here, we propose a new approach using a recurrent neural network as a type of model that\nis suf\ufb01ciently \ufb02exible [Siegelmann and Sontag, 1995] to represent the potentially complex neural\ncomputations in the brain, while also closely matching subjects\u2019 choice behavior. In this way, the\nmodel learns to learn the task such that (a) its output matches subjects\u2019 choices; and (b) its internal\nmechanism tracks subjects\u2019 brain activity. A model trained using this approach ideally provides an\nend-to-end model of neural decision-making circuitry that does not bene\ufb01t from manual engineering,\nbut describes how past inputs are translated to future actions through a successive set of computations\noccurring in different brain regions.\nHaving introduced the architecture of this recurrent neural network meta-learner, we show how\nto interpret it by unrolling it over space and time to determine the role of each brain region at\neach time slice in the path from reward processing to action selection. We show that experimental\nresults obtained using our method are consistent with the previous literature on the neural basis of\ndecision-making and provide novel insights into the temporal dynamics of reward processing in the\nbrain.\n\n2 Related work\n\nThere are at least four types of previous approach. In type one, which includes model-based fMRI\nanalysis and some work on complex non-linear recurrent dynamical systems [Sussillo et al., 2015],\nthe models are trained on the behavioral data and are only then applied to the neural data. By contrast,\nwe include neural data at the outset. In a second type recurrent neural networks are trained to perform\na task [e.g., to maximize reward; Song et al., 2017], but without the attention that we give to both the\npsychological and neural data. A third type aims to uncover the dynamics of the interaction between\ndifferent brain regions by approximating the underlying neural activity (see Breakspear [2017] for\nreview). However, unlike our protocol, these models are not trained on behavioral data. A fourth\ntype relies on two separate models for the behavioral and neural data but, unlike model-based fMRI\nanalyses, the free parameters of the two models are jointly modeled and estimated, e.g., by assuming\nthat they follow a joint distribution [Turner et al., 2013, Halpern et al., 2018]. Nevertheless, similar to\nmodel-based fMRI, this approach requires manual model engineering and is limited by how well the\nhypothesized behavioral model characterizes its underlying neural processes.\n\n3 The model\n\n3.1 Data\nWe consider a typical neuroscience study of decision-making processes in humans, in which the data\ninclude the actions of a set of subjects while they are making choices and receiving rewards (DBEH)\nin a decision-making task, while their brain activity in the form of fMRI images is recorded (DfMRI).\nBehavioral data include the states of the environment (described by set S), choices executed by the\nsubjects in each state, and the rewards they receive. At each time t 2 T i subject i observes state\nt 2S as an input, calculates and then executes action ai\nt (e.g., presses a button on a computer\nsi\nkeyboard; ai\nt (e.g., a monetary reward;\nt 2 <). The behavioral data can be described as,\nri\n\nt 2A and A is a set of actions) and receives a reward ri\n\nDBEH = {(si\n\nti) |i = 1...NSUBJ, ti 2 T i}.\n\nti, ai\n\nti, ri\n\n(1)\n\nThe second component of the data is the recorded brain activity in the form of 3D images taken by\nthe scanner during the task. Each image can be divided into a set of voxels (NVOX voxels; e.g., 3mm\nx 3mm x 3mm cubes), each of which has an intensity (a scalar number) which represents the neural\nactivity of the corresponding brain region at the time of image acquisition by the scanner. Images\n\n2\n\n\fat1\nrt1\nst\n\nrnn\n(GRU)\n\nW\n.....\n\nh1\nt\nh2\nt\nh3\nt\n. . .\n. . .\nhNcells\nt\n\nut\n\nyt\n\n.....\n\nP (at = eye)\n\nP (at = hand)\n\nLfMRI(\u21e5)\n\nL(\u21e5)\n\nLBEH(\u21e5)\n\nFigure 1: Architecture of the model. The model has a RNN layer which consists of a set of GRU cells,\nand receives previous actions, rewards and the current state of the environment as inputs (NCELLS is\nthe number of cells in the RNN layer). The outputs/states of the RNN layer (ht) are connected to a\nmiddle layer (shown by red circles) with the same number of units as there are voxels (NVOX); the\noutputs of the units ut are weighted sums over their inputs. Each component of ut is convolved\nwith the HRF signal and is compared to the bias-adjusted intensity of its corresponding voxel in\nfMRI recordings (yt). Voxels are shown by the squares overlaying the brain, and three of them are\nhighlighted (in blue) as an example of how they are connected to the units in the middle layer. The\noutputs of the GRU cells are also connected to a softmax layer (the green lines), which outputs the\nprobability of selecting each action on the next trial (in this case, EYE and HAND are the available\nactions). LBEH refers to the behavioral loss function and LfMRI refers to the fMRI loss function. The\n\ufb01nal loss function is denoted by L(\u21e5), which is a weighted sum of the fMRI and the behavioral loss\nfunctions. \u21e5 contains all the parameters.\n\nare acquired at times 0, TR, 2TR, . . . , (NACQ  1)TR, where TR refers to the repetition time of the\nscanner (time between image acquisitions), and NACQ is the total number of images. Let yi,v\nt denote\nthe intensity of voxel v recorded at time t for subject i. The fMRI data will take the following form:\n\nDfMRI = {yi,v\n\nt }, t = 0, TR, 2TR, . . . , (NACQ  1)TR, i = 1 . . . NSUBJ, v = 1 . . . NVOX.\n\n(2)\n\n3.2 Network architecture\nActions taken by a subject at each point in time are affected by the history of previous rewards,\nactions and states experienced by the subject. Aspects of this history are encoded in neural activity in\na persistent, albeit mutating, form, and enable subjects\u2019 future choices to bene\ufb01t from past experience.\nThis process constitutes learning in the task; we aim to recover it by jointly modeling the behavioral\nand neural data. We \ufb01rst describe the network architecture and then explain how it can be interpreted\nto answer the questions of interest.\nRNN layer. The model (Figure 1) is a speci\ufb01c form of recurrent neural network (RNN). The recurrent\nlayer consists of a set of NCELLS GRU cells [Gated recurrent unit; Cho et al., 2014]; cell c outputs\nits state hc\n]>).\nThis state summarizes the past history of the inputs to the network and is updated as new inputs are\nreceived according to a function that we denote by f,\n\nt at time t. We de\ufb01ne ht as the state of the whole RNN network (ht = [h1\n\nt , . . . , hNCELLS\n\nt\n\nht = f (at1, rt1, st, ht1;\u21e5) ,\n\n(3)\ndepending on parameters \u21e5. We aim to train the parameters of this dynamical system to approximate\nthe underlying neural computations in the brain that translate previous inputs to future actions during\nthe task.\nfMRI layer. To establish a correspondence between the underlying RNN and neural activity, one\ntraining signal for \u21e5 comes from requiring the activity of each voxel at each point in time to be\ndescribed as a (noisy) linear combination of GRU cell states (shown by the red connections in Figure 1).\nWe denote the weights of this linear combination as W 2 <NVOX\u21e5NCELLS, and ut as a vector of size\nNVOX representing predicted neural activity at each voxel at time t. Thus,\n(4)\n\nut = W ht.\n\n3\n\n\fFor training the model, the predicted neural activity is compared with the actual activity recorded by\nthe scanner. However, neural activity is not instantly re\ufb02ected in the intensity recorded by the scanner,\nbut is delayed according to the haemodynamic response function (HRF; Figure S3). To correct for\nthis delay, elements of ut are \ufb01rst convolved with HRF [Henson and Friston, 2007], and after adding\na bias term b, are compared with the intensities of the corresponding voxels, to form the following\nloss function,\n\nLfMRI(\u21e5) =Xt kut ~ HRF + b  ytk2 , t 2{ 0, TR, . . . (NACQ  1)TR},\n\n(5)\n\nin which \u21e5 is the model parameters (W and RNN parameters), yt is a vector of size NVOX containing\nthe recorded activity of each voxel at time t. Symbol ~ is the convolution operator. The above\nloss function can be thought of as the logarithm of a Gaussian likelihood function. Note that in this\ncase the convolution operator acts on the output of the network, and so this is not a conventional\nconvolutional neural network, in which convolutions act on the input.\nBehavioral layer. To ensure that the RNN also captures the behavioral data, a second training signal\nfor \u21e5 comes from requiring it to produce actions similar to those of humans. This is achieved by\nconnecting the output of the RNN network to a softmax layer in Figure 1 (shown by the green lines),\nin which the weights of the connections determine the in\ufb02uence of each cell on the probability of\nselecting actions. Denoting by \u21e1t(a) the predicted probability of taking action a at time t, we de\ufb01ne\nthe behavioral loss function as:\n\nLBEH(\u21e5) = Xt2T 0\n\nlog \u21e1t(at),\n\n(6)\n\nin which T 0 refers to the timesteps at which the subject was allowed to execute an action.\nTraining. We de\ufb01ne the overall loss function as the weighted sum of the behavioral and fMRI loss\nfunctions,\n\nL(\u21e5) =\n\nLBEH(\u21e5;Di) + LfMRI(\u21e5;Di),\n\n(7)\n\nNSUBJXi=1\n\nwith parameter  determining the contribution of the fMRI loss function, and Di denoting the data of\nsubject i. Note the above loss function can be thought of as the logarithm of the multiplication of a\nGaussian likelihood function (for the fMRI part) \u2013 with  being related to the level of noise/variance\nin the likelihood function \u2013 and a multinomial likelihood function (for the behavioral part).\n\nInterpreting the model\n\n3.3\nWe seek to understand how the inputs to the network (previous rewards, actions, states) affect future\nactions through the medium of the brain\u2019s neural activity. Although different methods have been\nsuggested for investigating the way the inputs to a neural network determine its outputs, the most\nfundamental quantity is the gradient of the output with respect to the input, which represents how\nmuch the output changes by changing the input (as used, for instance, by Simonyan et al. [2013] in\nthe context of an image classi\ufb01cation task).\nInspired by this, we de\ufb01ned two differential quantities relating rewards, actions and brain activity to\neach other. There are at least two \u2018layers\u2019 to this: off- and on-policy. In the off-policy setting, which\nis conventionally studied in model-based imaging, there is a \ufb01xed sequence of inputs, whose effects\non future predicted probabilities and neural activities we determine. In the on-policy setting, which is\nused in settings such as approximate Bayesian computation [Sunn\u00e5ker et al., 2013], future choices,\nand thus future inputs are also affected by past inputs. For the present, we consider the simpler,\noff-policy setting. This allows us to look, for instance, at the brain regions involved in mediating the\neffect of the reward that subject i actually received at, say, time t1 on the predicted probability of\nthe action that the subject actually executed at, say, time t2. For convenience, we drop notation for\nthe \ufb01xed inputs for the subject; and indeed for the subject number (since we \ufb01t a single model to the\nwhole group).\nThe \ufb01rst measure represents the behavioral effects of reward on future actions, which can be calculated\nas the gradient of the predicted probabilities of actions at each time t2 with respect to the input\nreceived at time t1. For the case of binary choices, which are the focus of the current experiment, with\n\n4\n\n\fEYE and HAND as the two available actions in the task, we only needed to calculate the probability\nfor one of the actions. Let \u21e1t2 denote the probability of taking action EYE at time t2. The effect of\nreward at time t1 on the action at time t2 can be calculated as follows,\n\nd\u21e1r\nt1,t2 =\n\n@\u21e1t2\n@rt1\n\n.\n\nThis is a straightforward application of backpropagation (calculated using automatic differentiation),\nnoting again that we consider the inputs received by the network between t1 and t2 to be \ufb01xed. d\u21e1r\nt1,t2\ncan be thought as capturing how much the probability of taking action EYE at time t2 increases as the\nresults of increasing the magnitude of reward earned at time t1.\nThe second measure relates behavioral and fMRI data by exploiting the informational association\nbetween the predicted neural activity ut and the state of RNN, ht. First, note that, at each time t, ht\nis a Markov state for the RNN, in that given ht, RNN outputs after time t are independent of their past.\nThus, we can decompose:\n\n@\u21e1t2\n@rt1\n\n=\n\nNCELLSXk=1\n\n@\u21e1t2\n@hk\nt\n\n@hk\nt\n@rt1\n\n,\n\nfor any\n\nt 2{ t1 + 1 . . . t2},\n\n(8)\n\nas the effect changing rt1 has on the predicted RNN state hk\nt at time t, times the effect that a change\nin hk\nt has on the action probability \u21e1t2 at t2. Now, consider the case that W >W is non-singular (note\nthat NVOX  NCELLS). This implies that there is a one-to-one mapping between the RNN state and\npredicted neural activity:\n\nht = (W >W )1W >ut .\n\n(9)\n\nThus, we can rewrite equation 8 in terms of the effect changing rt1 has on the predicted neural\nactivity uv\nt implies about a change in \u21e1t2, operating\nimplicitly via what the change in uv\n\nt in each voxel at time t times what a change in uv\n\nt tells us about a change in ht. We can write this as,\n\n@\u21e1t2\n@rt1\n\n=\n\nNVOXXv=1\n\n@\u21e1t2\n@uv\nt\n\n@uv\nt\n@rt1\n\n, t = t1 + 1 . . . t2.\n\n(10)\n\nNote that this is a correlational relationship \u2013 the direction of causality is from ht to ut. Nevertheless\nthe individual terms in this sum:\n\n@uv\nt\n@rt1\n\n(11)\n\nt /@rt1),\nt ). This\n\nd\u21e1ur\nv,t1,t,t2 =\n\n, for any t 2{ t1 + 1 . . . t2},\n\n@\u21e1t2\n@uv\nt\ncombine the in\ufb02uence that voxel uv\nt at time t receives from the reward at time t1 (which is @uv\nwith the covariation between the voxel activity and the action at time t2 (which is @\u21e1t2/@uv\nquanti\ufb01es the intermediation of voxel uv\nt between the reward at t1 and the action at time t2.\nWe make two remarks: (i) The joint \ufb01tting of the model to both the behavioral and fMRI data was\nimportant that, if there are behaviorally equivalent solutions, then the one that can \ufb01t the neural data\nshould be chosen; and (ii) for equation 10 to hold it is necessary for the state of the network to be\nfully determined by the neural activity (ut). This can hold in the case of GRU cells (provided that the\nhidden units do not partition into separate behavioral and neural groups). In contrast, in LSTM cells\n[Long short-term memory; Hochreiter and Schmidhuber, 1997], the cell states and cell outputs are\ndifferent and are both required to determine the outputs in the next time-step, and therefore in the\ncase of LSTM cells equation 10 does not hold.\n\n4 Results\n\nIn this section we aim to show how the above measures can be used to study the neural substrates of\ndecision-making in the brain.\n\n5\n\n\fA\n\nmake choice\n\n2.5 sec\n\ndelay\n\n3.5 sec\n\noutcome\nrevealed\n1 sec\n\nITI\n\n1-8 sec\n\nA\n\nVe\nVh\n\nSMA\n(0, -12, 78)\n\npreSEF\n(-6, 9, 60)\n\n1C\n\n \n\n0.5\n\n \n\n1D\n0.6\n0.2\n\nj\n\ni\n\nB\n\nt\n\n0.6\n\n6\n\n4\n\n0\n\n50\n\n0\n\n50\n\n)\n.\n\n.\n\n100\n\n150\n\n100\n\n150\n\nd\nr\na\nw\ne\nr\n \nP\n\n1\n0.5\n\nFigure 2: The task. Each trial started with the presentation of a screen (the left most square in the\n\ufb01gure) and the subjects had 2.5 seconds to make an eye saccade to the red target circle or press\na button with their right hand. After a delay (3.5 second) during which the screen showed only a\n\ufb01xation point, subjects received the outcome of their choice which indicated whether their choice was\nrewarded. The next trial started after an inter-trial interval (ITI) that varied between 1 and 8 seconds.\nFigure reprinted with permission from Wunderlich et al. [2009]. Copyright (2009) National Academy\nof Sciences.\n\n0 0.2\n1\nmodel choice prob.\n\ntime (choice trial)\nchoice eye\nchoice hand\n\nmodel choice probability\nsubject choice\n\ns\ne\nc\no\nh\nc\n \n\u2019\ns\ns\n\ntime (choice trial)\n\ne\ny\ne\ne\ns\no\no\nh\nc\nP\n\nu\na\n(\n \ne\n\nx = -6\n\nB\n\na\nm\n\n2\n\ni\nt\ns\nE\n\nn\no\nC\n\n \nt\ns\na\nr\nt\n\n4.2 Model settings\n\n4.1 Task and subjects\n\nFig. 1.\nExperimental Design and Behavior. (A) Subjects were presented with a\nchoice cue after which they had to respond within 2.5 s by performing a saccade\nto the red target circle or a right handed button press. Once a response was\nregistered the screen was immediately cleared for a short delay and subsequently\nthe outcome was revealed (6 s after trial onset) indicating either receipt of reward\nor no reward. Inter-trial-intervals varied between 1 and 8 s. (B) Example reward\nprobabilities for saccades and button presses as a function of the trial number.\nThe probability of being rewarded following choice of either the hand or eye\nmovement was varied across the experiment independently for each movement.\n(C) Fitted model choice probability (red) and actual choice behavior (blue) shown\nfor a single subject. (D) Actual choice behavior versus model predicted choice\nprobability. Data are pooled across subjects, the regression slope is shown as a\nline, vertical bars, SEM.\n\n0\nThe data used here were previously published in Wunderlich et al. [2009]. The structure of the\ndecision-making task is shown in Figure 2. In each trial subjects had a choice between making\na saccade (EYE) or pressing a button (HAND). Choices were rewarded with varying probabilities\n\u22122\nacross the experiment. There were four trial types in the task: (i) free-choice trials (150 trials), in\nwhich subjects could choose between EYE and HAND; (ii) forced-choice trials in which subjects\nwere instructed to choose EYE (50 trials) or (iii) HAND (50 trials); (iv) null trials in which no\nreward was received irrespective of the action selected (50 trials). Forced-choices and null trails\nwere randomly inserted between the free-choice trials. The environment consisted of two actions\n(EYE and HAND) and \ufb01ve states corresponding to the four trial types and one state when the choice\noutcomes were shown (reward or no-reward). Actions and states are assumed to be coded using\none-hot representations. Since time was discretized (see below), there were time points at which no\naction was taken or no visual stimulus was shown, in which case states and actions were coded using\nzero vectors.\nThe total number of subjects was NSUBJ = 22, and in total NACQ = 1136 images were acquired by\nthe scanner each containing NVOX = 63191 voxels. Therefore, the fMRI data can be summarized as\na tensor of size 22 \u21e5 1136 \u21e5 63191. Each subject made \u21e0 300 choices. See Supplementary Material\nfor the details of fMRI preprocessing and model settings.\n\nTo look for neural correlates of action values we had to estimate\nthe value of taking each action in every trial. We calculated the\naction values using a computational reinforcement-learning (RL)\nmodel in which the value of each action, Veye and Vhand, was\nupdated in proportion to a prediction error on each trial (see Table\nS1 for a summary of how the different types of value signals relate\nto the components of the experiment). The model also assumed\nthat action selection in every trial followed a soft-max probability\nrule based on the difference of the estimated action values (8). To\ntest for the presence of action value signals in the brain we took the\nmodel predicted trial-by-trial estimates of the two action values and\nentered these into a regression analysis against the fMRI data. In\naddition to a whole brain screening for the presence of action-value\nsignals, we specifically looked for them in areas known to be\ninvolved in the planning of motor actions, including supplementary\nmotor cortex (18\u201321) and lateral parietal cortex (22, 23). Given that\nboth of these areas have previously been shown to contain value-\nrelated signals for movements in nonhuman primates, and that they\nare closely interconnected with the area of motor cortex involved\nin carrying out motor actions (24\u201326), we considered these areas\nprime candidates for containing action-value representations that\ncould then be used to guide action-based choices. It is important to\nemphasize, however, that the tasks used in previous studies did not\nmake it possible to determine if the value signals identified were\nchosen values or action values.\n\nFigure 3(a,b) shows two sets of off-policy simulations. In each simulation there are four choice states,\nthe times of which are shown by the vertical gray patches in the top panels. The red patch following\neach grey ribbon shows the time at which the outcome was revealed following the choice. The \ufb01rst\nchoice was rewarded (shown by \u2018R\u2019 in the graph), but the rest were not. In panel (a) action HAND\nwas selected in all choice states whereas in panel (b) it was action EYE. Based on this, since in\npanel (a) the reward was earned when HAND was selected, we expected that choice to decrease the\nprobability of selecting action EYE on the next choice. This is shown by the blue bars which illustrate\nthe gradient of the probability of selecting the EYE action at each subsequent choice with respect to\nthe amount of reward earned after the \ufb01rst choice (d\u21e1r). For panel (b), since the reward was earned\nas a consequence of choosing EYE in the \ufb01rst choice, we expected the reward to have a positive effect\non the probability of selecting EYE on the next trials, which is consistent with the graph.\nNext we asked about the intermediation of each brain region between the reward earned after the \ufb01rst\nchoice (t1) and the next choice (t2), shown by the red arrow in Figure 3(a). To answer this question,\n\nWe also looked for areas that are involved in comparing the\naction values to make a choice. Two areas of a priori interest were\nthe anterior cingulate cortex (ACC) and the dorsal striatum. ACC\nhas been previously implicated in action-based choice, both in the\ncontext of a human imaging study reporting activity in this area\nduring a task involving choices between different actions compared\nto a situation involving responses guided by instruction (27), and in\na monkey lesion study where ACC lesions produced an impairment\nin action-outcome based choice but not in mediating changes in\nresponses following errors (28). Dorsal striatum has been impli-\ncated in both goal-directed and habitual instrumental responding\nfor reward in rodents (29, 30). Moreover, human fMRI studies\nreveal increased activity in both of these regions when subjects\n\nAll the methods were implemented in Tensor\ufb02ow [Abadi et al., 2016] and gradients (for both\noptimization and interpretation of the model) were calculated using automatic differentiation methods\navailable in this package. See Supplementary Material for the model settings.\n\n4.3 From reward to action\n\n6\n\nz = +60\n\npreSEF\n\nVe\n\nVh\n\nVe\n\nFig. 2. Action values. (A) Region of supplementary motor area showing cor-\nrelations with action values for hand movement (Vh/green) and a region of\npre-SEF showing correlations with action-values for eye movements (Ve/red).\nT-maps are shown from a whole brain analysis thresholded at P \u2b0d 0.001 uncor-\nrected (see Fig. S1 for a version with color bars relating to t stats). (B) Average\neffect sizes of Ve (red) and Vh (green) extracted from SEF and SMA. The effects\nshown here were calculated from trials independent of those used to functionally\nidentify the ROI. Note that only Ve but not Vh modulate the signal in preSEF, and\nthat activity in SMA shows the opposite pattern. Vertical lines, SEM.\n\nmake choices to obtain reward compared to an otherwise analogous\nsituation in which the rewards are obtained without the need to\nmake a choice (31\u201334).\n\nThe most simple type of comparison process would be to\ncompute a difference between the two action values. We tested for\nsuch a difference, but as we had no a priori hypothesis about the\ndirectionality of the computation, we tested for both the difference\nbetween the value of the action chosen and the value of action not\nchosen (Vchosen \u2afa Vunchosen), and one involving the opposite\ndifference (Vunchosen \u2afa Vchosen). As we found evidence for such an\naction-value comparison signal in the brain, we then proposed a\nsimple computational model to provide a conceptual explanation as\nto how such a signal could reflect the output of a computationally\nplausible decision mechanism.\n\nResults\nRL Model Fits to Behavioral Choice Data. A comparison of the choice\nprobabilities predicted by the RL model and the soft-max proce-\ndure to subjects\u2019 actual behavior suggests that the model matches\nsubjects behavior well. Fig. 1C compares both variables for a typical\nsubject. Fig. 1D compares the predicted choice probability (binned)\nagainst the actual choice probabilities for the group. A similar linear\nregression analysis at the individual level generated an average R2\nacross subjects of 0.83 and regression coefficients that were signif-\nicant at P \u2b0d 0.001 in each subject.\n\nAction Values. We found neural activity correlating with the action\nvalues for making a hand movement in left supplementary motor\narea (SMA; Fig. 2A and Table S2). A region of interest (ROI)\nanalysis showed that activity in this area satisfied the properties of\na hand action value: it was sensitive to the value of hand movements,\nand it showed no response selectivity to the value of eye movements\n(Fig. 2B). Activity in lateral parietal cortex, ACC, and right dorsal\n\n\fwe calculated d\u21e1ur for every voxel and every time-step between t1 and t2, and masked out the voxels\nthat were not in the top one percent. By focusing only on the 99th percentile of |d\u21e1ur|, we hoped to\nlimit our analysis to the circuitry known to be involved in decision-making. The resulting voxel maps\nare shown in Figure 3(c) for the case of HAND action corresponding to the inputs shown in panel\n(a), and Figure 3(d) shows the time-course of changes in d\u21e1ur. See Figure S1(c,d) for EYE action\ncorresponding to the inputs shown in panel (b).\nThe results show that, for each action, the top 1% of voxels contain three key cortical and subcortical\nbrain regions known to be critically involved in reward-processing and decision-making, i.e., (i)\nstriatum (associative aStr; or ventral, vStr), (ii) anterior cingulate cortex (ACC) and (iii) supplementary\nmotor area (SMA) [Rangel and Hare, 2010, Wunderlich et al., 2009]. We \ufb01rst note that these\nanatomical regions are among the same anatomical regions that Wunderlich et al. [2009] also\nidenti\ufb01ed as involved in decision-making in this task (see Figure S4 for the time course of changes in\nd\u21e1ur for the voxel coordinates reported in Wunderlich et al. [Table S3; 2009]).\nSecondly, we can see that not only are the identi\ufb01ed regions consistent with the neural substrates\nof decision-making based on previous work, but the temporal order of engagement of these regions\nis also consistent with their functional role in decision-making. It has been argued that activity in\nsubregions of the striatum re\ufb02ect reward prediction-errors [O\u2019Doherty et al., 2004] and that these\nerrors serve to update action-values in the ACC [Dayan and Balleine, 2002, Wunderlich et al., 2009,\nSeo and Lee, 2007, Walton et al., 2004], which in turn must be compared in the SMA to determine\nthe best action before a decision can be made [Wunderlich et al., 2009]. Such prior work has argued\nthat these different decision-making signals are carried by separate regions in a corticostriatal loop,\nwhich is assumed to participate in a time course of events leading to action-selection [Balleine and\nO\u2019Doherty, 2010, Hare et al., 2011].\nHere we show for the \ufb01rst time the temporal dynamics between these critical regions in the striatum,\nanterior cingulate cortex and motor areas leading to action-selection. Figure 3(d) shows the time\ncourse of each region\u2019s d\u21e1ur between the reward at 9.2 s (t1) and the next response at 12.8 s (t2).\nNote that since we took the probability of taking the EYE action as the reference, negative values\nof d\u21e1ur indicate a region\u2019s role in selecting the HAND action. At reward receipt (9.2 s), d\u21e1ur of the\nventral striatum begins below the zero baseline and then (negatively) peaks at 9.8 s, as it mediates\nthe effect of reward prediction-errors on the subsequent hand response. The value of d\u21e1ur for the\nanterior cingulate then (negatively) peaks after 10.4 s, consistent with its role in updating action\nvalues with the new errors before the next response. Finally d\u21e1ur for the large cluster in the motor\narea (including the supplementary motor area) controlling motor responses such as the HAND action,\nnegatively peaks at the time of the action (12.8 s), which marks the end of the decision process in the\ncurrent task.\nAs part of our supplementary material, Figure S1(d) shows the time dynamics between the striatum,\nanterior cingulate and motor areas controlling EYE choices \u2013 corresponding to the inputs shown in\npanel (b). Here positive values of d\u21e1ur indicate a region\u2019s role in selecting the EYE action. At reward\nreceipt (9.2 s) the associative striatum is involved immediately in mediating the effect of reward on\nthe subsequent action-selection. Then at 11 s the involvement of the anterior cingulate peaks before\na region in the motor area nearest the supplementary eye \ufb01eld peaks at the time of action (12.8 s).\nIn sum, changes in d\u21e1ur over this time period mirror those for the HAND action, and are consistent\nwith the hypothesized roles of these regions in the varying decision stages of the reward-learning task\nused here.\n\n5 Discussion\n\nWe have introduced a new neural architecture for investigating the neural substrates of decision-\nmaking in the brain. Unlike previous methods, our approach does not require manual engineering and\nis able to learn computational processes directly from the data. We further showed that the model can\nbe interpreted to uncover the temporal engagement of different brain regions in choice and reward\nprocessing. Besides being used as a standalone analysis tool, this approach can inform model-based\nfMRI analyses to investigate whether the model correctly tracks the brain\u2019s internal mechanism. That\nis, if a brain region is found to be important in the current analysis, but not using the model-based\nfMRI analysis, this could mean that the model used to extract neural information is not representing\nall of the relevant neural signals involved in decision-making and requires further modi\ufb01cation.\n\n7\n\n\fa) HAND action\n\nto how such a signal could reflect the output of a computationally\nplausible decision mechanism.\n\nto how such a signal could reflect the output of a computationally\nplausible decision mechanism.\n\nrelated signals for movements in nonhuman primates, and that they\nare closely interconnected with the area of motor cortex involved\nin carrying out motor actions (24\u201326), we considered these areas\nprime candidates for containing action-value representations that\ncould then be used to guide action-based choices. It is important to\nemphasize, however, that the tasks used in previous studies did not\nmake it possible to determine if the value signals identified were\nchosen values or action values.\n\nrelated signals for movements in nonhuman primates, and that they\nare closely interconnected with the area of motor cortex involved\nin carrying out motor actions (24\u201326), we considered these areas\nprime candidates for containing action-value representations that\ncould then be used to guide action-based choices. It is important to\nemphasize, however, that the tasks used in previous studies did not\nmake it possible to determine if the value signals identified were\nchosen values or action values.\n\nFigure 3(a,b) shows two sets of off-policy simulations. In each simulation there are four choice states,\nthe times of which are shown by the vertical gray patches in the top panels. The red patch following\neach grey ribbon shows the time at which the outcome was revealed following the choice. The \ufb01rst\nchoice was rewarded (shown by \u2018R\u2019 in the graph), but the rest were not. In panel (a) action HAND\nwas selected in all choice states whereas in panel (b) it was action EYE. Based on this, since in\nb) EYE action\npanel (a) the reward was earned when HAND was selected, we expected that choice to decrease the\nprobability of selecting action EYE on the next choice. This is shown by the blue bars which illustrate\nthe gradient of the probability of selecting the EYE action at each subsequent choice with respect to\nthe amount of reward earned after the \ufb01rst choice (d\u21e1r). For panel (b), since the reward was earned\nas a consequence of choosing EYE in the \ufb01rst choice, we expected the reward to have a positive effect\non the probability of selecting EYE on the next trials, which is consistent with the graph.\nNext we asked about the intermediation of each brain region between the reward earned after the \ufb01rst\nchoice (t1) and the next choice (t2), shown by the red arrow in Figure 3(a). To answer this question,\n\nFigure 3(a,b) shows two sets of off-policy simulations. In each simulation there are four choice states,\nthe times of which are shown by the vertical gray patches in the top panels. The red patch following\neach grey ribbon shows the time at which the outcome was revealed following the choice. The \ufb01rst\nchoice was rewarded (shown by \u2018R\u2019 in the graph), but the rest were not. In panel (a) action HAND\nwas selected in all choice states whereas in panel (b) it was action EYE. Based on this, since in\npanel (a) the reward was earned when HAND was selected, we expected that choice to decrease the\nprobability of selecting action EYE on the next choice. This is shown by the blue bars which illustrate\nthe gradient of the probability of selecting the EYE action at each subsequent choice with respect to\nthe amount of reward earned after the \ufb01rst choice (d\u21e1r). For panel (b), since the reward was earned\nas a consequence of choosing EYE in the \ufb01rst choice, we expected the reward to have a positive effect\non the probability of selecting EYE on the next trials, which is consistent with the graph.\nNext we asked about the intermediation of each brain region between the reward earned after the \ufb01rst\nchoice (t1) and the next choice (t2), shown by the red arrow in Figure 3(a). To answer this question,\n\nResults\nRL Model Fits to Behavioral Choice Data. A comparison of the choice\nprobabilities predicted by the RL model and the soft-max proce-\ndure to subjects\u2019 actual behavior suggests that the model matches\n0.15\nsubjects behavior well. Fig. 1C compares both variables for a typical\nWe also looked for areas that are involved in comparing the\n0.10\nsubject. Fig. 1D compares the predicted choice probability (binned)\naction values to make a choice. Two areas of a priori interest were\nagainst the actual choice probabilities for the group. A similar linear\nH\n0.05\nthe anterior cingulate cortex (ACC) and the dorsal striatum. ACC\nregression analysis at the individual level generated an average R2\n0.00\nhas been previously implicated in action-based choice, both in the\nacross subjects of 0.83 and regression coefficients that were signif-\n\u22120.05\ncontext of a human imaging study reporting activity in this area\nicant at P \u2b0d 0.001 in each subject.\n~26\ntime (s)\nduring a task involving choices between different actions compared\nto a situation involving responses guided by instruction (27), and in\nAction Values. We found neural activity correlating with the action\nc)\na monkey lesion study where ACC lesions produced an impairment\nvalues for making a hand movement in left supplementary motor\nin action-outcome based choice but not in mediating changes in\narea (SMA; Fig. 2A and Table S2). A region of interest (ROI)\n6\nresponses following errors (28). Dorsal striatum has been impli-\nanalysis showed that activity in this area satisfied the properties of\ncated in both goal-directed and habitual instrumental responding\na hand action value: it was sensitive to the value of hand movements,\nfor reward in rodents (29, 30). Moreover, human fMRI studies\nand it showed no response selectivity to the value of eye movements\nreveal increased activity in both of these regions when subjects\n(Fig. 2B). Activity in lateral parietal cortex, ACC, and right dorsal\n\nWe also looked for areas that are involved in comparing the\naction values to make a choice. Two areas of a priori interest were\nthe anterior cingulate cortex (ACC) and the dorsal striatum. ACC\nhas been previously implicated in action-based choice, both in the\ncontext of a human imaging study reporting activity in this area\n~26\nduring a task involving choices between different actions compared\nto a situation involving responses guided by instruction (27), and in\na monkey lesion study where ACC lesions produced an impairment\nin action-outcome based choice but not in mediating changes in\n10.4 s\nresponses following errors (28). Dorsal striatum has been impli-\ncated in both goal-directed and habitual instrumental responding\nfor reward in rodents (29, 30). Moreover, human fMRI studies\nreveal increased activity in both of these regions when subjects\n\nResults\nRL Model Fits to Behavioral Choice Data. A comparison of the choice\nprobabilities predicted by the RL model and the soft-max proce-\ndure to subjects\u2019 actual behavior suggests that the model matches\nsubjects behavior well. Fig. 1C compares both variables for a typical\nE\nsubject. Fig. 1D compares the predicted choice probability (binned)\nagainst the actual choice probabilities for the group. A similar linear\nregression analysis at the individual level generated an average R2\nacross subjects of 0.83 and regression coefficients that were signif-\nicant at P \u2b0d 0.001 in each subject.\ntime (s)\nAction Values. We found neural activity correlating with the action\nvalues for making a hand movement in left supplementary motor\narea (SMA; Fig. 2A and Table S2). A region of interest (ROI)\nanalysis showed that activity in this area satisfied the properties of\na hand action value: it was sensitive to the value of hand movements,\nand it showed no response selectivity to the value of eye movements\n(Fig. 2B). Activity in lateral parietal cortex, ACC, and right dorsal\n\n~0 ~3 ~6 ~9\n\n~0 ~3 ~6 ~9\n\n0.3\n0.2\n0.1\n0.0\n\n11.0 s\n\n11.6 s\n\n12.2 s\n\n12.8 s\n\nr\n \n\u03c0\nd\n\nr\n \n\u03c0\nd\n\n9.8 s\n\n9.2 s\n\n~13\n\n~16\n\n~19\n\n~23\n\n~13\n\n~16\n\n~19\n\n~23\n\n~29\n\n~29\n\nR\n\nH\n\nR\n\nH\n\nH\n\n6\n\nE\n\nE\n\nE\n\n,\n\nm\n17200 \u5169 www.pnas.org\u517ecgi\u517edoi\u517e10.1073\u517epnas.0901077106\nu\nt\na\ni\nr\nt\ns\n\nm\nu\nt\na\ni\nr\nt\ns\n\ne\nh\nt\n\nr\no\nf\n\nf\no\n\ni\nt\n\ns\ne\nu\nl\na\nv\n\nn\no\ni\nt\nc\na\n\ne\nm\n\ne\nh\nt\n\ne\nh\nt\n\ns\n\ns\ne\nt\na\ni\nd\ne\nm\n\n.\n)\n2\nt\n(\n\n,\n\n,\n\nn\ni\n\n.\n)\ns\n\nk\ns\na\nt\n\ne\nh\nt\n\ne\nr\no\nf\ne\nb\n\nd\nr\na\nw\ne\nr\n\nn\ni\nn\nw\no\nh\ns\n\nn\no\nd\nr\na\nw\ne\nr\n\nd\nn\nn\n17200 \u5169 www.pnas.org\u517ecgi\u517edoi\u517e10.1073\u517epnas.0901077106\no\na\ni\nt\ng\nc\na\nn\ni\nD\nr\ne\nN\ne\nA\nn\ni\nH\ng\nn\ne\ne\nh\nt\n\nt\nn\ne\nt\ns\ni\ns\nn\no\nc\n\ns\ns\ne\nc\no\nr\np\n\nn\no\ni\nt\nc\na\n\ns\nt\nu\np\nn\ni\n\ns\nk\na\ne\np\n\n8\n.\n2\n1\n(\n\ne\nh\nt\n\n-\nn\no\ni\ns\ni\nc\ne\nd\n\nf\no\n\nA\n\nt\n\n.\n\nn\no\ni\nt\nc\na\n\nr\no\nt\no\nm\ne\nh\nt\nn\ni\n\nr\ne\nt\ns\nu\nl\nc\n\ng\nn\ni\nt\na\nd\np\nu\n\nn\ni\n\ne\nl\no\nr\n\ns\nt\ni\n\nh\nt\ni\n\nw\n\nt\nn\ne\nt\ns\ni\ns\nn\no\nc\n\n,\ns\n\n4\n.\n0\n1\n\nr\ne\nt\nf\na\n\nn\no\ni\ns\ni\nc\ne\nd\n\ne\nh\nt\n\nd\nn\ne\n\ne\nh\nt\n\ns\nk\nr\na\nm\n9.2\nh\nc\ni\nh\nw\n\n,\n)\ns\n\ns\na\n\ne\ng\nr\na\nl\n\nd)\n40\n\ne\nh\nt\n\nr\no\nf\n\nh\nc\nu\ns\n\ns\ne\ns\nn\no\np\ns\ne\nr\n\n0\n\nr\nu\nr\n\u21e1\no\nd\nt\no\ny\nm\nl\nl\n-40\na\ng\nn\nn\ni\ni\nF\nl\nl\no\nr\nt\nn\no\nc\n\n.\ne\ns\nn\no\np\ns\ne\nr\n\nf\no\n\nx10-5\n\ng\nn\ni\nn\nr\na\ne\nl\n-\nd\nr\na\nw\ne\nr\n\ne\nh\nt\n\nf\no\n\ns\ne\ng\na\nt\ns\n\nn\no\ni\ns\ni\nc\ne\nd\n\ne\nr\na\nd\nn\na\n\n,\nn\no\ni\nt\nc\na\nD\nN\nA\nH\ne\nh\nt\n\nr\no\nf\n\ne\ns\no\nh\nt\n\n10.4\n\ng\nn\ni\ny\nr\na\nv\n\nvStr\n\ne\nh\nt\n\nr\no\nr\nr\ni\n\nm\nd\no\ni\nr\ne\np\ne\nm\n\ne\nt\na\nl\nu\ng\nn\ni\nc\n\nr\no\ni\nr\ne\nt\nn\na\n\ne\nh\nt\n\nf\no\n\nt\nn\ne\nm\ne\nv\nl\no\nv\nn\ni\n\ne\nh\nt\n\nf\no\nt\nc\ne\nf\nf\ne\n\ne\nh\nt\ng\nn\ni\nt\na\ni\nd\ne\nm\nn\ni\ny\nl\ne\nt\na\ni\nd\ne\nm\nm\n\ni\nd\ne\nv\nl\no\nv\nn\ni\n\nf\no\n\ne\nm\n\ni\nt\n\ne\nh\nt\n\nt\na\n\ns\nk\na\ne\np\n\nd\nl\ne\n\ufb01\ne\ny\ne\n\ny\nr\na\nt\nn\ne\nm\ne\nl\np\np\nu\ns\n\nE\nY\nE\n\ne\nh\nt\n\ng\nn\ni\nt\nc\ne\nl\ne\ns\n\nn\ni\n\ne\nl\no\nr\n\ne\nh\nt\no\nt\ng\nn\ni\nd\nn\no\np\ns\ne\nr\nr\no\nc\n\u2013\ns\ne\nc\ni\no\nh\nc\n\n9.8\n\ns\n\u2019\nn\no\ni\ng\ne\nr\n\na\n\nE\nY\nE\n\nn\ne\ne\nw\nt\ne\nb\n\ns\nc\ni\nm\na\nn\ny\nd\n\ne\nm\n\ni\nt\n\ne\nh\nt\n\ns\nw\no\nh\ns\n\n)\nd\n(\n1\nS\ne\nr\nu\ng\ni\nF\n\n11.0\nACC\n\nt\na\nh\nT\n\nd\ne\ns\na\nb\n-\nl\ne\nd\no\nm\nm\nr\no\nf\nn\ni\n\nn\na\nc\n\nh\nc\na\no\nr\np\np\na\n\ns\ni\nh\nt\n\n,\nl\no\no\nt\n\n.\n\nm\n\ns\ni\nn\na\nh\nc\ne\nm\n\nl\na\nn\nr\ne\nt\nn\ni\n\ns\n\u2019\nn\ni\na\nr\nb\n\ne\nh\nt\n\ns\nk\nc\na\nr\nt\n\ns\ni\ns\ny\nl\na\nn\na\n\ny\nl\nt\nc\ne\nr\nr\no\nc\n\nd\ne\ns\na\nb\n-\nl\ne\nd\no\nm\ne\nh\nt\ng\nn\ni\ns\nu\nt\no\nn\nt\nu\nb\n,\ns\ni\ns\ny\nl\na\nn\na\n\nt\nn\ne\nr\nr\nu\nc\n\ng\nn\ni\nt\nn\ne\ns\ne\nr\np\ne\nr\n\nt\no\nn\n\ns\ni\n\nn\no\ni\nt\na\nm\nr\no\nf\nn\ni\n\nl\na\nr\nu\ne\nn\n\nt\nc\na\nr\nt\nx\ne\n\no\nt\n\nd\ne\ns\nu\n\n.\n\nn\no\ni\nt\na\nc\n\ufb01\n\ni\nd\no\nm\n\nr\ne\nh\nt\nr\nu\nf\n\ns\ne\nr\ni\nu\nq\ne\nr\nd\nn\na\n\ng\nn\ni\nk\na\nm\n-\nn\no\ni\ns\ni\nc\ne\nd\n\ne\nh\nt\nn\ni\n\n12.2\n\nn\na\nc\n\nl\ne\nd\no\nm\ne\nh\nt\n\nt\na\nh\nt\n\nd\ne\nw\no\nh\ns\n\nr\ne\nh\nt\nr\nu\nf\n\ne\n\nW\n\n.\na\nt\na\nd\n\ne\nh\nt\n\nm\no\nr\nf\n\nd\nr\na\nw\ne\nr\nd\nn\na\n\ne\nc\ni\no\nh\nc\nn\ni\n\ns\nn\no\ni\ng\ne\nr\nn\ni\na\nr\nb\nt\nn\ne\nr\ne\nf\nf\ni\nd\nf\no\nt\nn\ne\nm\ne\ng\na\ng\nn\ne\n\ns\ne\nt\na\nr\nt\ns\nb\nu\ns\n\nl\na\nr\nu\ne\nn\n\ne\nh\nt\n\ng\nn\ni\nt\na\ng\ni\nt\ns\ne\nv\nn\ni\n\nr\no\nf\n\nl\na\nu\nn\na\nm\ne\nr\ni\nu\nq\ne\nr\n\nt\no\nn\ns\ne\no\nd\nh\nc\na\no\nr\np\np\na\n\n11.6\nPMC\n\nr\nu\no\n\nWunderlich et al.\n\n12.8\n\n7\n\ne\nh\nt\n\nn\ni\n\ns\nw\no\nh\ns\n\n8\n.\n2\n1\n\nt\na\n\ne\ns\nn\no\np\ns\ne\nr\n\nt\nx\ne\nn\n\ne\nh\nt\n\nd\nn\na\n\ne\nv\ni\nt\na\ng\ne\nn\n\n,\ne\nc\nn\ne\nr\ne\nf\ne\nr\n\ne\nh\nt\n\ns\na\n\nn\no\ni\nt\nc\na\n\n)\n1\nt\n(\n\ns\n\n2\n.\n9\n\nt\na\n\nE\nY\nE\n\ne\nh\nt\n\n)\nd\n(\n3\n\ne\nr\nu\ng\ni\n\nF\n\n.\nn\no\ni\nt\nc\ne\nl\ne\ns\n-\nn\no\ni\nt\nc\na\n\no\nt\n\ng\nn\ni\nd\na\ne\nl\n\nd\nr\na\nw\ne\nr\n\ng\nn\ni\nk\na\nt\n\ns\nn\no\ni\ng\ne\nr\n\nl\na\nc\ni\nt\ni\nr\nc\n\ne\ns\ne\nh\nt\n\nn\ne\ne\nw\nt\ne\nb\n\ns\nc\ni\nm\na\nn\ny\nd\n\nl\na\nr\no\np\nm\ne\nt\n\ne\nh\nt\n\ne\nm\n\ni\nt\n\ns\na\ne\nr\na\n\nr\no\nt\no\nm\nd\nn\na\n\nt\ns\nr\n\ufb01\ne\nh\nt\n\nx\ne\nt\nr\no\nc\n\nr\no\nf\n\nw\no\nh\ns\n\ne\nt\na\nl\nu\ng\nn\ni\nc\n\ne\nh\nt\n\nn\ne\ne\nw\nt\ne\nb\n\nr\nu\n\u21e1\nd\n\ns\n\u2019\nn\no\ni\ng\ne\nr\n\nh\nc\na\ne\n\nf\no\n\nf\no\n\ny\nt\ni\nl\ni\nb\na\nb\no\nr\np\n\ne\nh\nt\n\nk\no\no\nt\n\ne\nw\ne\nc\nn\ni\ns\n\nt\na\nh\nt\n\ne\nt\no\nN\n\ne\nw\ne\nr\ne\nH\n\nr\no\ni\nr\ne\nt\nn\na\n\ne\ns\nr\nu\no\nc\n\n.\n]\n1\n1\n0\n2\n\n,\n.\nl\na\n\nt\ne\n\ne\nr\na\nH\n\n,\n\n0\n1\n0\n2\n\n,\n\ny\nt\nr\ne\nh\no\nD\nO\n\n\u2019\n\nr\nu\n\u21e1\nd\n\nr\nu\n\u21e1\nd\n\nt\ni\n\ns\na\n\n,\ns\n8\n.\n9\nt\na\n\ns\nk\na\ne\np\n)\ny\nl\ne\nv\ni\nt\na\ng\ne\nn\n(\nn\ne\nh\nt\nd\nn\na\n\ne\nn\ni\nl\ne\ns\na\nb\no\nr\ne\nz\n\ne\nh\nt\n\nw\no\nl\ne\nb\ns\nn\ni\ng\ne\nb\nm\nu\nt\na\ni\nr\nt\ns\n\n,\n)\ns\n2\n\n.\n\n9\n(\n\nt\np\ni\ne\nc\ne\nr\nd\nr\na\nw\ne\nr\n\nt\n\nA\n\n.\n\nn\no\ni\nt\nc\na\nD\nN\nA\nH\ne\nh\nt\ng\nn\ni\nt\nc\ne\nl\ne\ns\nn\ni\n\ne\nl\no\nr\n\ns\n\u2019\nn\no\ni\ng\ne\nr\n\na\n\ne\nt\na\nc\ni\nd\nn\ni\n\nr\nu\n\u21e1\nd\nf\no\n\nl\na\nr\nt\nn\ne\nv\n\nf\no\n\ne\nu\nl\na\nv\n\ne\nh\nT\n\n.\ne\ns\nn\no\np\ns\ne\nr\n\nd\nn\na\nh\n\nt\nn\ne\nu\nq\ne\ns\nb\nu\ns\n\ne\nh\nt\n\nn\no\n\ns\nr\no\nr\nr\ne\n-\nn\no\ni\nt\nc\ni\nd\ne\nr\np\n\nd\nr\na\nw\ne\nr\n\nf\no\n\nt\nc\ne\nf\nf\ne\n\ne\nh\nt\n\ns\n\ni\nt\n\ns\ni\n\nt\na\n\nn\ni\n\nn\ni\n\n1\n1\n\ne\nh\nt\n\ne\nh\nt\n\n)\na\ne\nr\na\n\n8\n.\n2\n1\n(\n\nn\ne\nh\nT\n\ns\na\ne\nr\na\n\nn\no\ni\nt\nc\na\n\ns\nn\no\ni\ng\ne\nr\n\nr\nu\n\u21e1\nd\nf\no\n\ny\nl\nt\nc\ne\nr\ni\nd\n\ne\nt\na\nc\ni\nd\nn\ni\n\nt\nx\ne\nn\ne\nh\nt\n\nm\nu\nt\na\ni\nr\nt\ns\n\n,\ns\nd\no\nh\nt\ne\nm\n\nt\nn\na\nt\nr\no\np\nm\n\nl\ne\nd\no\nm\ne\nh\nt\n\nl\ne\nd\no\nm\ne\nh\nt\n\ne\nn\no\nl\na\nd\nn\na\nt\ns\n\ng\nn\ni\nl\nl\no\nr\nt\nn\no\nc\n\ne\nr\nu\nt\nc\ne\nt\ni\nh\nc\nr\na\n\nr\no\nt\no\nm\ny\nr\na\nt\nn\ne\nm\ne\nl\np\np\nu\ns\n\nFigure 3: (a,b). The graphs show the effect of reward on actions in terms of d\u21e1r. The choice states\n(between EYE and HAND actions) are shown by the grey shaded area. In the left panel, action HAND\n(shown by \u2018H\u2019) was selected and in the right panel action EYE (shown by \u2018E\u2019) was selected at all\nof the choice states. The outcome of each choice (reward/no reward) was delivered in red shaded\narea. The \ufb01rst choice was rewarded, as shown by \u2018R\u2019 in the graph, but the other choices were not\nfollowed by any reward. The blue bars show the effect of reward received after the \ufb01rst choice on\nthe subsequent choices (d\u21e1r). (c). Voxel maps and the time-course of changes in d\u21e1ur in cortical\ne\nr\nand subcortical brain regions between reward of the HAND action at 9.2 s and the response at 12.8 s\no\nf\nshown by the red arrow in panel (a). Voxels below the 99th percentile of voxels were masked to reveal\ne\nb\nonly the top one percent of voxels shown here. (d) The time courses of each region calculated from\ns\nr\nthe maximum voxel in that region at each time point (smoothed), selected within an anatomical mask\no\nr\nfrom wfu_pickatlas. y-axis represents d\u21e1ur. ACC: anterior cingulate cortex; vStr: ventral striatum;\nr\ne\nPMC: primary motor cortex. See Figure S1 for the voxel maps and time course changes relating to\nw\nthe EYE action.\ne\nn\ne\nh\nt\nh\nt\ni\n\n,\nl\na\ni\nr\ne\nt\na\nm\ny\nr\na\nt\nn\ne\nm\ne\nl\np\np\nu\ns\n\nr\ne\nh\nt\ne\nh\nw\ne\nt\na\ng\ni\nt\ns\ne\nv\nn\ni\n\n.\nn\no\ni\nt\nc\ne\nl\ne\ns\n-\nn\no\ni\nt\nc\na\n\ns\nu\no\ni\nv\ne\nr\np\ne\nk\ni\nl\nn\nU\n\nl\na\nn\no\ni\nt\na\nt\nu\np\nm\no\nc\n\ne\nb\no\nt\nd\nn\nu\no\nf\n\nn\na\ne\nm\nd\nl\nu\no\nc\n\nl\na\nr\nu\ne\nn\nw\ne\nn\n\ne\nv\ni\nt\na\ni\nc\no\ns\ns\na\n\nd\ne\nc\nu\nd\no\nr\nt\nn\ni\n\nr\no\nt\no\nm\nd\nn\na\n\nr\no\nt\no\nm\ne\nh\nt\n\ns\ne\ns\ns\ne\nc\no\nr\np\n\nl\na\nr\no\np\nm\ne\nt\n\nd\ne\nv\nl\no\nv\nn\ni\n\nt\nn\na\nv\ne\nl\ne\nr\n\ns\ne\nd\ni\ns\ne\nB\n\ne\nv\ni\nt\ni\ns\no\np\n\nr\nu\n\u21e1\nd\nn\ni\n\nt\ns\ne\nr\na\ne\nn\n\ns\nl\na\nn\ng\ni\ns\n\nl\na\nr\nu\ne\nn\n\ns\ne\nu\nl\na\nv\n\ns\nk\na\ne\np\n\ng\nn\ni\ne\nb\n\ne\ns\ne\nh\nt\n\nn\nr\na\ne\nl\n\ne\nr\ne\nH\n\ns\ne\nl\no\nr\n\nd\ne\ns\nu\n\nr\ne\nv\no\n\na\ne\nr\na\n\nd\ne\nz\ni\ns\ne\nh\nt\no\np\ny\nh\n\ne\nt\na\nl\nu\ng\nn\ni\nc\n\nt\na\nh\nt\n\n)\ns\n2\n\ns\ni\nh\nt\n\ns\ni\nh\nt\n\nr\nu\no\n\ne\nh\nt\n\ne\nh\nt\n\ne\nh\nt\n\ne\nh\nt\n\ne\nm\n\ns\na\n\nf\no\n\nf\no\n\nn\ni\n\no\nt\n\nt\na\n\ni\nt\n\na\n\na\n\ni\n\ns\ne\ng\nn\na\nh\nc\n\n.\n\ns\nk\na\ne\np\n\n)\ny\nl\ne\nv\ni\nt\na\ng\ne\nn\n(\n\nn\ne\nh\nt\n\ne\nt\na\nl\nu\ng\nn\ni\nc\n\nr\ne\nv\no\nc\nn\nu\no\nt\nd\ne\nt\ne\nr\np\nr\ne\nt\nn\ni\n\ns\ni\nn\no\ni\ng\ne\nr\nn\ni\na\nr\nb\na\n\nf\ni\n\n,\ns\ni\n\n,\ns\ni\ns\ny\nl\na\nn\na\n\ne\nh\nt\n\nI\n\nR\nM\n\nf\n\nf\no\nl\nl\na\n\ns\ne\ns\ny\nl\na\nn\na\n\nI\n\nR\nM\n\nf\n\n.\ng\nn\ni\ns\ns\ne\nc\no\nr\np\n\n.\nn\ni\na\nr\nb\ne\nh\nt\nn\ni\ng\nn\ni\nk\na\nm\n\ne\nv\na\nh\n\ne\n\nW\n\no\nt\n\ne\nl\nb\na\n\ns\ni\n\ne\nb\n\nn\no\ni\ns\ns\nu\nc\ns\ni\nD\n8\n\n5\n\ng\nn\ni\nd\nu\nl\nc\nn\ni\n(\n\na\ne\nr\na\n\nk\ns\na\nt\n\nt\nn\ne\nr\nr\nu\nc\n\ny\nl\ne\nv\ni\nt\na\ng\ne\nn\n\nr\no\ni\nr\ne\nt\nn\na\n\nw\ns\ne\nu\nl\na\nv\n\nt\nn\ne\nu\nq\ne\ns\nb\nu\ns\n\ne\nh\nt\n\nf\no\n\n.\n9\n(\n\n.\n)\nb\n(\n\nt\nr\na\np\n\ns\nA\n\nr\no\ni\nr\ne\nt\nn\na\n\nt\np\ni\ne\nc\ne\nr\n\nl\ne\nn\na\np\n\nn\no\ni\ng\ne\nr\n\na\n\n,\n\nm\nu\ns\nn\nI\n\n.\ne\nr\ne\nh\n\ne\nh\nt\n\nh\nt\ni\n\nw\n\nd\ne\ns\nu\n\n\fAcknowledgments\nAD and BWB were supported by funding from UNSW Sydney and the National Health and Medical\nResearch Council of Australia GNT1079561. PD was funded by the Gatsby Charitable Foundation.\nPart of this work was conducted whilst PD was at Uber Technologies. Neither body played a part in\nits design, execution or communication. PD is af\ufb01liated with Max Planck Institute for Biological\nCybernetics, T\u00fcbingen, Germany (peter.dayan@tuebingen.mpg.de).\n\nReferences\nJoshua I Gold and Michael N Shadlen. The neural basis of decision making. Annual review of neuroscience, 30,\n\n2007.\n\nKenji Doya. Modulators of decision making. Nature neuroscience, 11(4):410\u20136, apr 2008.\n\nJohn P O\u2019Doherty, Alan Hampton, and Hackjin Kim. Model-based fMRI and its application to reward learning\n\nand decision making. Annals of the New York Academy of sciences, 1104(1):35\u201353, 2007.\n\nJonathan D Cohen, Nathaniel Daw, Barbara Engelhardt, Uri Hasson, Kai Li, Yael Niv, Kenneth A Norman,\nJonathan Pillow, Peter J Ramadge, Nicholas B Turk-Browne, and Others. Computational approaches to fMRI\nanalysis. Nature neuroscience, 20(3):304, 2017.\n\nNathaniel D Daw, John P O\u2019Doherty, Peter Dayan, Ben Seymour, and Raymond J. Dolan. Cortical substrates for\n\nexploratory decisions in humans. Nature, 441(7095):876\u20139, jun 2006.\n\nHava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. Journal of computer\n\nand system sciences, 50(1):132\u2013150, 1995.\n\nDavid Sussillo, Mark M. Churchland, Matthew T. Kaufman, and Krishna V. Shenoy. A neural network that \ufb01nds\na naturalistic solution for the production of muscle activity. Nature Neuroscience, 18(7):1025\u20131033, 2015.\n\nH. Francis Song, Guangyu R. Yang, and Xiao Jing Wang. Reward-based training of recurrent neural networks\n\nfor cognitive and value-based tasks. eLife, 6:1\u201324, 2017.\n\nMichael Breakspear. Dynamic models of large-scale brain activity. Nature neuroscience, 20(3):340, 2017.\n\nBrandon M Turner, Birte U Forstmann, Eric-Jan Wagenmakers, Scott D Brown, Per B Sederberg, and Mark\nSteyvers. A Bayesian framework for simultaneously modeling neural and behavioral data. NeuroImage, 72:\n193\u2013206, 2013.\n\nDavid Halpern, Shannon Tubridy, Hong Yu Wang, Camille Gasser, Pamela Osborn Popp, Lila Davachi, and\nTodd M Gureckis. Knowledge Tracing Using the Brain. In Proceedings of the 11th International Conference\non Educational Data Mining, EDM, 2018.\n\nKyunghyun Cho, Bart Van Merri\u00ebnboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk,\nand Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine\ntranslation. arXiv preprint arXiv:1406.1078, 2014.\n\nRichard Henson and Karl J Friston. CHAPTER 14 - Convolution Models for fMRI. In Karl Friston, John\nAshburner, Stefan Kiebel, Thomas Nichols, and William Penny, editors, Statistical Parametric Mapping,\npages 178\u2013192. Academic Press, London, 2007. ISBN 978-0-12-372560-8.\n\nKaren Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising\n\nimage classi\ufb01cation models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.\n\nMikael Sunn\u00e5ker, Alberto Giovanni Busetto, Elina Numminen, Jukka Corander, Matthieu Foll, and Christophe\n\nDessimoz. Approximate Bayesian Computation. PLOS Computational Biology, 9(1):1\u201310, 2013.\n\nSepp Hochreiter and J\u00fcrgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735\u20131780,\n\n1997.\n\nKlaus Wunderlich, Antonio Rangel, and John P O\u2019Doherty. Neural computations underlying action-based\ndecision making in the human brain. Proceedings of the National Academy of Sciences, 106(40):17199\u201317204,\n2009.\n\nMart\u00edn Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy\nDavis, Jeffrey Dean, Matthieu Devin, and Others. Tensor\ufb02ow: Large-scale machine learning on heterogeneous\ndistributed systems. arXiv preprint arXiv:1603.04467, 2016.\n\n9\n\n\fAntonio Rangel and Todd Hare. Neural computations associated with goal-directed choice. Current opinion in\n\nneurobiology, 20(2):262\u2013270, 2010.\n\nJohn P O\u2019Doherty, Peter Dayan, Johannes Schultz, Ralf Deichmann, Karl J Friston, and Raymond J. Dolan.\nDissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304(5669):452\u20134, apr\n2004.\n\nPeter Dayan and Bernard W Balleine. Reward, motivation, and reinforcement learning. Neuron, 36(2):285\u201398,\n\n2002. ISSN 0896-6273.\n\nHyojung Seo and Daeyeol Lee. Temporal \ufb01ltering of reward signals in the dorsal anterior cingulate cortex during\n\na mixed-strategy game. Journal of neuroscience, 27(31):8366\u20138377, 2007.\n\nMark E Walton, Joseph T Devlin, and Matthew F S Rushworth. Interactions between decision making and\n\nperformance monitoring within prefrontal cortex. Nature neuroscience, 7(11):1259, 2004.\n\nBernard W Balleine and John P O\u2019Doherty. Human and rodent homologies in action control: corticostriatal\n\ndeterminants of goal-directed and habitual action. Neuropsychopharmacology, 35(1):48\u201369, jan 2010.\n\nTodd A Hare, Wolfram Schultz, Colin F Camerer, John P O\u2019Doherty, and Antonio Rangel. Transformation of\nstimulus value signals into motor commands during simple choice. Proceedings of the National Academy of\nSciences, 108(44):18120\u201318125, 2011.\n\nTijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its\n\nrecent magnitude. COURSERA: Neural networks for machine learning, 4(2):26\u201331, 2012.\n\nDiederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization.\n\narXiv:1412.6980, 2014.\n\narXiv preprint\n\n10\n\n\f", "award": [], "sourceid": 2077, "authors": [{"given_name": "Amir", "family_name": "Dezfouli", "institution": "Data61, CSIRO"}, {"given_name": "Richard", "family_name": "Morris", "institution": "U Sydney"}, {"given_name": "Fabio", "family_name": "Ramos", "institution": "University of Sydney"}, {"given_name": "Peter", "family_name": "Dayan", "institution": "Gatsby Unit, UCL"}, {"given_name": "Bernard", "family_name": "Balleine", "institution": "UNSW"}]}