{"title": "Prediction and Change Detection", "book": "Advances in Neural Information Processing Systems", "page_first": 1281, "page_last": 1288, "abstract": null, "full_text": "Prediction and Change Detection \n\n \n \n \n \n \n\nMark Steyvers \n\nmsteyver@uci.edu \n\nScott Brown \n\nscottb@uci.edu \n\nUniversity of California, Irvine University of California, Irvine \n\nIrvine, CA 92697 \n\nIrvine, CA 92697 \n\nAbstract \n\nWe measure the ability of human observers to predict the next datum \nin a sequence that is generated by a simple statistical process \nundergoing change at random points in time. Accurate performance \nin this task requires the identification of changepoints. We assess \nindividual differences between observers both empirically, and \nusing two kinds of models: a Bayesian approach for change detection \nand a family of cognitively plausible fast and frugal models. Some \nindividuals detect \ntoo many changes and hence perform \nsub-optimally due to excess variability. Other individuals do not \ndetect enough changes, and perform sub-optimally because they fail \nto notice short-term temporal trends. \n\n1 I n t r o d u c t i o n \n\nDecision-making often requires a rapid response to change. For example, stock \nanalysts need to quickly detect changes in the market in order to adjust investment \nstrategies. Coaches need to track changes in a player\u2019s performance in order to adjust \nstrategy. When tracking changes, there are costs involved when either more or less \nchanges are observed than actually occurred. For example, when using an overly \nconservative change detection criterion, a stock analyst might miss important \nshort-term trends and interpret them as random fluctuations instead. On the other \nhand, a change may also be detected too readily. For example, in basketball, a player \nwho makes a series of consecutive baskets is often identified as a \u201chot hand\u201d player \nwhose underlying ability is perceived to have suddenly increased [1,2]. This might \nlead to sub-optimal passing strategies, based on random fluctuations. \n\nWe are interested in explaining individual differences in a sequential prediction task. \nObservers are shown stimuli generated from a simple statistical process with the task \nof predicting the next datum in the sequence. The latent parameters of the statistical \nprocess change discretely at random points in time. Performance in this task depends \non the accurate detection of those changepoints, as well as inference about future \noutcomes based on the outcomes that followed the most recent inferred changepoint. \nThere is much prior research in statistics on the problem of identifying changepoints \n[3,4,5]. In this paper, we adopt a Bayesian approach to the changepoint identification \nproblem and develop a simple inference procedure to predict the next datum in a \nsequence. The Bayesian model serves as an ideal observer model and is useful to \ncharacterize the ways in which individuals deviate from optimality. \n\n\f \n\nThe plan of the paper is as follows. We first introduce the sequential prediction task \nand discuss a Bayesian analysis of this prediction problem. We then discuss the results \nfrom a few individuals in this prediction task and show how the Bayesian approach \ncan capture individual differences with a single \u201ctwitchiness\u201d parameter that \ndescribes how readily changes are perceived in random sequences. We will show that \nsome individuals are too twitchy: their performance is too variable because they base \ntheir predictions on too little of the recent data. Other individuals are not twitchy \nenough, and they fail to capture fast changes in the data. We also show how behavior \ncan be explained with a set of fast and frugal models [6]. These are cognitively \nrealistic models that operate under plausible computational constraints. \n\n2 A p r e d i c t i o n t a s k w i t h m u l t i p l e c h a n g e p o i n t s \n\nIn the prediction task, stimuli are presented sequentially and the task is to predict the \nnext stimulus in the sequence. After t trials, the observer has been presented with \nstimuli y1, y2, \u2026, yt and the task is to make a prediction about yt+1. After the prediction \nis made, the actual outcome yt+1 is revealed and the next trial proceeds to the \nprediction of yt+2. This procedure starts with y1 and is repeated for T trials. \n\nThe observations yt are D-dimensional vectors with elements sampled from binomial \ndistributions. The parameters of those distributions change discretely at random \npoints in time such that the mean increases or decreases after a change point. This \ngenerates a sequence of observation vectors, y1, y2, \u2026, yT, where each yt = {yt,1 \u2026 \nyt,D}. Each of the yt,d is sampled from a binomial distribution Bin(\u03b8t,d,K), so 0 \u2264 yt,d \u2264 \nK. The parameter vector \u03b8t ={\u03b8t,1 \u2026 \u03b8t,D} changes depending on the locations of the \nchangepoints. At each time step, \nx is a binary indicator for the occurrence of a \n\nt\n\nchangepoint occurring at time t+1. The parameter \u03b1 determines the probability of a \nchange occurring in the sequence. The generative model is specified by the following \nalgorithm: \n\n \n\n1. For d=1..D sample \u03b81,d from a Uniform(0,1) distribution \n\n2. For t=2..T, \n\n(a) Sample xt-1 from a Bernoulli(\u03b1) distribution \n\n(b) If xt-1=0, then \u03b8t=\u03b8t-1, else \n\nfor d=1..D sample \u03b8t,d from a Uniform(0,1) distribution \n\n(c) for d=1..D, sample yt from a Bin(\u03b8t,d,K) distribution \n\n \n\nTable 1 shows some data generated from the changepoint model with T=20, \u03b1=.1,and \nD=1. In the prediction task, y will be observed, but x and \u03b8 are not. \n\n \n\nt\n\nx\n\n\u03b8\n\ny\n\nTable 1: Example data \n\n1\n\n0\n\n2\n\n0\n\n3\n\n0\n\n4\n\n1\n\n5\n\n0\n\n6\n\n0\n\n7\n\n1\n\n8\n\n0\n\n9 10 11 12 13 14 15 16 17 18 19 20\n\n0\n\n0\n\n0\n\n0\n\n1\n\n0\n\n1\n\n0\n\n0\n\n0\n\n0\n\n0\n\n.68 .68 .68 .68 .48 .48 .48 .74 .74 .74 .74 .74 .74 .19 .19 .87 .87 .87 .87 .87\n8 \n\n7\n\n9\n\n1\n\n8\n\n3\n\n6\n\n7\n\n4\n\n9\n\n7\n\n8\n\n2\n\n9\n\n8\n\n8\n\n8\n\n4\n\n4\n\n9\n\n \n\n\f \n\n3 A B a y e s i a n p r e d i c t i o n m o d e l \n\nIn both our Bayesian and fast-and-frugal analyses, the prediction task is decomposed \ninto two inference procedures. First, the changepoint locations are identified. This is \nfollowed by predictive inference for the next outcome based on the most recent \nchangepoint locations. Several Bayesian approaches have been developed for \nchangepoint problems involving single or multiple changepoints [3,5]. We apply a \nMarkov Chain Monte Carlo (MCMC) analysis to approximate the joint posterior \ndistribution over changepoint assignments x while integrating out \u03b8. Gibbs sampling \nwill be used to sample from this posterior marginal distribution. The samples can then \nbe used to predict the next outcome in the sequence. \n\n3 . 1 \n\nI n f e r e n c e f o r c h a n g e p o i n t a s s i g n m e n t s . \n\nTo apply Gibbs sampling, we evaluate the conditional probability of assigning a \nchangepoint at time i, given all other changepoint assignments and the current \u03b1 value. \nBy integrating out \u03b8, the conditional probability is \n\n \n\nP x x\n\n|\n\n(\n\ni\n\n\u2212\n\n,\n\n,\ny\n\n\u03b1\n\ni\n\n)\n\n= \u222b\n\n(\nP x\n\ni\n\n,\n\n\u03b8 \u03b1\n\n,\n\n|\n\nx\n\n\u2212\n\n,\n\ny\n\n)\n\n \n\ni\n\n\u03b8\n\n(1) \n\n \n\nwhere \n\nx\n\n\u2212\n\ni\n\n represents all switch point assignments except xi. This can be simplified by \n\nconsidering the location of the most recent changepoint preceding and following time \ni and the outcomes occurring between these locations. Let\nn be the number of time \n\nL\n\ni\n\nsteps from the last changepoint up to and including the current time step i such that \n\nx\n\nL\ni n\ni\n\n\u2212\n\n=1 and \n\nx\n\ni n\n\u2212\n\nj\n\n+\n\nL\n\ni\n\n=0 for 0<j< L\n\nn . Similarly, let \n\ni\n\nn be the number of time steps that \n\nR\n\ni\n\nfollow time step i up to the next changepoint such that \n\nx\n\nR\ni n\ni\n\n+\n\n=1 and \n\nx\n\ni n\n+\n\nj\n\n\u2212\n\nR\n\ni\n\n=0 for \n\n0<j<\n\nR\n\nn . Let\n\ni\n\nL\n\ny\ni\n\n= \u2211\n\ny\n\nk\n\nand \n\nR\n\ny\ni\n\n= \u2211\n\nk i\n< \u2264\n\nk k i n\ni\n\n< \u2264 +\n\nR\n\ny\nk\n\n. The update equation for the \n\nL\n\n\u2212\n\ni n\ni\n\nchangepoint assignment can then be simplified to \n\n \n\nP x m x\n\n|\n\n=\n\n(\n\ni\n\n)\n\n\u221d\n\ni\n\n\u2212\n\n \n\n \n\n\u23a7\n\u23aa\n\u23aa\u23aa\n\u23a8\n\u23aa\n\u23aa\n\u23aa\u23a9\n\n\u03b1\n\n\u0393 +\n\n(\n1\n\nL\ny\ni j\n,\n\n+\n\n(\n1\n\n\u2212\n\n\u03b1\n\nD\n\n\u220f\n\n)\n\nj\n\n1\n\n=\n\ny\n\n(\n\u0393 +\n1\n\n)\n\nL\ny\ni j\n,\n\nD\n\n\u220f\n\nj\n\n1\n\n=\n\n\u0393 +\n\n(\n1\n\nL\nKn\ni\n(\n\u0393 +\n\n2\n\n2\n\n)\n(\n\u0393 +\nR\n1\ni j\n,\n(\n\u0393 +\n)\n)\n\nL\nKn\ni\n\nL\ny\ni j\n,\n\n\u2212\n\nL\nKn\ni\n+\n\nL\nKn\ni\n(\n\u0393 +\n1\n(\n\u0393 +\n\n2\n\n+\n\nR\nKn\ni\n)\n\nR\nKn\ni\n)\nR\nKn\ni\n\nR\ny\ni j\n,\n\n(\n1\n\n\u0393 +\n)\n\n\u2212\n\nL\ny\ni j\n,\n\n\u2212\n\nR\ny\ni j\n,\n\n)\n\nR\nKn\ni\n\n\u2212\n\nR\ny\ni j\n,\n\n)\n\n \n\n(2) \n\nm\n\n=\n\n0\n\nm\n\n=\n\n1\n\nWe initialize the Gibbs sampler by sampling each xt from a Bernoulli(\u03b1) distribution. \nAll changepoint assignments are then updated sequentially by the Gibbs sampling \nequation above. The sampler is run for M iterations after which one set of changepoint \nassignments is saved. The Gibbs sampler is then restarted multiple times until S \nsamples have been collected. \n\nAlthough we could have included an update equation for \u03b1, in this analysis we treat \u03b1 \nas a known constant. This will be useful when characterizing the differences between \nhuman observers in terms of differences in \u03b1. \n\n\f3 . 2 P r e d i c t i v e i n f e r e n c e \n\nThe next latent parameter value \u03b8t+1 and outcome yt+1 can be predicted on the basis of \nobserved outcomes that occurred after the last inferred changepoint: \n\n\u03b8\n\nt\n\n1,\n\n+\n\nj\n\n=\n\nt\n\n\u2211\n\ny\n,\ni j\n\n/\n\nK\n\ny\n, \n\nt\n\n1,\n\n+\n\nj\n\n=\n\nround\n\n(\n\u03b8\n\n)\n\nK\n\nt\n\n1,\n\n+\n\nj\n\n(3) \n\n \n\ni t\n= +\n\n*\n\n1\n\nwhere t* is the location of the most recent change point. By considering multiple \nGibbs samples, we get a distribution over outcomes yt+1. We base the model \npredictions on the mean of this distribution. \n\n3 . 3 \n\nI l l u s t r a t i o n o f m o d e l p e r f o r m a n c e \n\nFigure 1 illustrates the performance of the model on a one dimensional sequence \n(D=1) generated from the changepoint model with T=160, \u03b1=0.05, and K=10. The \nGibbs sampler was run for M=30 iterations and S=200 samples were collected. The \ntop panel shows the actual changepoints (triangles) and the distribution of \nchangepoint assignments averaged over samples. The bottom panel shows the \nobserved data y (thin lines) as well as the \u03b8 values in the generative model (rescaled \nbetween 0 and 10). \n\nAt locations with large changes between observations, the marginal changepoint \nprobability is quite high. At other locations, the true change in the mean is very small, \nand the model is less likely to put in a changepoint. The lower right panel shows the \ndistribution over predicted \u03b8t+1 values. \n\n \n\nt\n\nx\n\n1\n\n0.5\n\n0\n\n10\n\nt\n\ny\n\n5\n\n0\n\n\u03b8\n\nt+1\n\n1\n\n0.5\n\n0\n\n160\n\n \n\n20\n\n40\n\n60\n\n80\n\n100\n\n120\n\n140\n\nFigure 1. Results of model simulation. \n\n \n\n4 P r e d i c t i o n e x p e r i m e n t \n\nWe tested performance of 9 human observers in the prediction task. The observers \nincluded the authors, a visitor, and one student who were aware of the statistical \nnature of the task as well as na\u00efve students. The observers were seated in front of an \nLCD touch screen displaying a two-dimensional grid of 11 x 11 buttons. The \nchangepoint model was used to generate a sequence of T=1500 stimuli for two \nbinomial variables y1 and y2 (D=2, K=10). The change probability \u03b1 was set to 0.1. \nThe two variables y1 and y2 specified the two-dimensional button location. The same \nsequence was used for all observers. \n\nOn each trial, the observer touched a button on the grid displayed on the touch screen. \nFollowing each button press, the button corresponding to the next {y1,y2} outcome in \nthe sequence was highlighted. Observers were instructed to press the button that best \npredicted the next location of the highlighted button. The 1500 trials were divided into \n\n \n \n\f \n\nthree blocks of 500 trials. Breaks were allowed between blocks. The whole \nexperiment lasted between 15 and 30 minutes. Figure 2 shows the first 50 trials from \nthe third block of the experiment. The top and bottom panels show the actual \noutcomes for the y1 and y2 button grid coordinates as well as the predictions for two \nobservers (SB and MY). The figure shows that at trial 15, the y1 and y2 coordinates \nshow a large shift followed by an immediate shift in observer\u2019s MY predictions (on \ntrial 16). Observer SB waits until trial 17 to make a shift. \n\n \n\n10\n\n5\n\n0\n\n10\n\n5\n\n0\n\noutcomes\n\nSB predictions\nMY predictions\n\n0\n\n5\n\n10\n\n15\n\n20\n\n25\nTrial\n\n30\n\n35\n\n40\n\n45\n\n50\n\n \n\nFigure 2. Trial by trial predictions from two observers. \n\n \n\n4 . 1 T a s k e r r o r \n\nWe assessed prediction performance by comparing the prediction with the actual \noutcome in the sequence. Task error was measured by normalized city-block distance \n\ntask error=\n\n1\n\n(\nT\n\n1)\n\n\u2212\n\nT\n\n\u2211\n\nt\n\n=\n\n2\n\ny\n\nt\n\n,1\n\n\u2212\n\ny\n\nO\n,1\n\nt\n\n+\n\ny\n\nt\n\n,2\n\n\u2212\n\ny\n\nO\n,2\n\nt\n\n(4) \n\nwhere yO represents the observer\u2019s prediction. Note that the very first trial is excluded \nfrom this calculation. Even though more suitable probabilistic measures for prediction \nerror could have been adopted, we wanted to allow comparison of observer\u2019s \nperformance with both probabilistic and non-probabilistic models. Task error ranged \nfrom 2.8 (for participant MY) to 3.3 (for ML). We also assessed the performance of \nfive models \u2013 their task errors ranged from 2.78 to 3.20. The Bayesian models \n(Section 3) had the lowest task errors, just below 2.8. This fits with our definition of \nthe Bayesian models as \u201cideal observer\u201d models \u2013 their task error is lower than any \nother model\u2019s and any human observer\u2019s task error. The fast and frugal models \n(Section 5) had task errors ranging from 2.85 to 3.20. \n\n5 M o d e l i n g R e s u l t s \n\nWe will refer to the models with the following letter codes: B=Bayesian Model, \nLB=limited Bayesian model, FF1..3=fast and frugal models 1..3. We assessed model \nfit by comparing the model\u2019s prediction against the human observers\u2019 predictions, \nagain using a normalized city-block distance \n\n \n \n\f \n\n(5) \n\n\u2212\n\ny\n\nO\n,2\n\nt\n\n\u2212\n\ny\n\nO\n,1\n\nt\n\n+\n\ny\n\nM\n,2\n\nt\n\nmodel error=\n\n1\n\nT\n(\n\n1)\n\n\u2212\n\nT\n\n\u2211\n\nt\n\n=\n\n2\n\ny\n\nM\n,1\n\nt\n\nwhere yM represents the model\u2019s prediction. The model error for each individual \nobserver is shown in Figure 3. It is important to note that because each model is \nassociated with a set of free parameters, the parameters optimized for task error and \nmodel error are different. For Figure 3, the parameters were optimized to minimize \nEquation (5) for each individual observer, showing the extent to which these models \ncan capture the performance of individual observers, not necessarily providing the \nbest task performance. \n\nB\n\nLB\n\nFF1\n\nFF2\n\nFF3\n\nr\no\nr\nr\n\nE\n\n \nl\n\ne\nd\no\nM\n\n2\n\n1.5\n\n1\n\n0.5\n\n0\n\nMY\n\nMS\n\nMM\n\nEJ\n\nPH\n\nNP\n\nDN\n\nSB\n\nML\n\n \n\nFigure 3. Model error for each individual observer.1 \n\n5 . 1 B a y e s i a n p r e d i c t i o n m o d e l s \n\n \n\nAt each trial t, the model was provided with the sequence of all previous outcomes. \nThe Gibbs sampling and inference procedures from Eq. (2) and (3) were applied with \nM=30 iterations and S=200 samples. The change probability \u03b1 was a free parameter. In \nthe full Bayesian model, the whole sequence of observations up to the current trial is \navailable for prediction, leading to a memory requirement of up to T=1500 trials \u2013 a \npsychologically unreasonable assumption. We therefore also simulated a limited \nBayesian model (LB) where the observed sequence was truncated to the last 10 \noutcomes. The LB model showed almost no decrement in task performance compared \nto the full Bayesian model. Figure 3 also shows that it fit human data quite well. \n\n5 . 2 \n\nI n d i v i d u a l D i f f e r e n c e s \n\nThe right-hand panel of Figure 4 plots each observer\u2019s task error as a function of the \nmean city-block distance between their subsequent button presses. This shows a clear \nU-shaped function. Observers with very variable predictions (e.g., ML and DN) had \nlarge average changes between successive button pushes, and also had large task \nerror: These observers were too \u201ctwitchy\u201d. Observers with very small average button \nchanges (e.g., SB and NP) were not twitchy enough, and also had large task error. \nObservers in the middle had the lowest task error (e.g., MS and MY). The left-hand \npanel of Figure 4 shows the same data, but with the x-axis based on the Bayesian \nmodel fits. Instead of using mean button change distance to index twitchiness (as in \n\n \n\n1 Error bars indicate bootstrapped 95% confidence intervals. \n\n \n \n\f \n\nthe right-hand panel), the left-hand panel uses the estimated \u03b1 parameters from the \nBayesian model. A similar U-shaped pattern is observed: individuals with too large or \ntoo small \u03b1 estimates have large task errors. \n\nNP\n\n \n\n3.3\n\nSB\n\nr\no\nr\nr\nE\n\n \nk\ns\na\nT\n\n3.2\n\n3.1\n\n3\n\n2.9\n\n2.8\n\nSB\n\nNP\n\nEJ\n\nML\n\nDN\n\nPH\n\nMM\n\nMS\n\nMY\n\nr\no\nr\nr\nE\n\n \nk\ns\na\nT\n\n3.3\n\n3.2\n\n3.1\n\n3\n\n2.9\n\n2.8\n\nML\n\nDN\n\nPH\n\nMM\n\nMS\n\nMY\n\nEJ\n\nB\n\n-4\n\n10\n\n-3\n\n10\n\n-2\n\n10\n\n\u03b1\n\n-1\n\n10\n\n0\n\n10\n\n0.5\n\n1\n\n1.5\n\n2\n\n2.5\n\n3\n\nMean Button Change\n\n \n\nFigure 4. Task error vs. \u201ctwitchiness\u201d. Left-hand panel indexes twitchiness using \nestimated \u03b1 parameters from Bayesian model fits. Right-hand panel uses mean \ndistance between successive predictions. \n\n \n\n5 . 3 F a s t - a n d - F r u g a l ( F F ) p r e d i c t i o n m o d e l s \n\n \n\nThese models perform the prediction task using simple heuristics that are cognitively \nplausible. The FF models keep a short memory of previous stimulus values and make \npredictions using the same two-step process as the Bayesian model. First, a decision is \nmade as to whether the latent parameter \u03b8 has changed. Second, remembered stimulus \nvalues that occurred after the most recently detected changepoint are used to generate \nthe next prediction. \n\nA simple heuristic is used to detect changepoints: If the distance between the most \nrecent observation and prediction is greater than some threshold amount, a change is \ninferred. We defined the distance between a prediction (p) and an observation (y) as \nthe difference between the log-likelihoods of y assuming \u03b8=p and \u03b8=y. Thus, if fB(.|\u03b8, \nK) is the binomial density with parameters \u03b8 and K, the distance between observation \ny and prediction p is defined as d(y,p)=log(fB(y|y,K))-log(fB(y|p,K)). A changepoint on \ntime step t+1 is inferred whenever d(yt,pt)>C. The parameter C governs the \ntwitchiness of the model predictions. If C is large, only very dramatic changepoints \nwill be detected, and the model will be too conservative. If C is small, the model will \nbe too twitchy, and will detect changepoints on the basis of small random fluctuations. \n\nPredictions are based on the most recent M observations, which are kept in memory, \nunless a changepoint has been detected in which case only those observations \noccurring after the changepoint are used for prediction. The prediction for time step \nt+1 is simply the mean of these observations, say p. Human observers were reticent to \nmake predictions very close to the boundaries. This was modeled by allowing the FF \nmodel to change its prediction for the next time step, yt+1, towards the mean prediction \n(0.5). This change reflects a two-way bet. If the probability of a change occurring is \n\u03b1, the best guess will be 0.5 if that change occurs, or the mean p if the change does not \noccur. Thus, the prediction made is actually yt+1=1/2 \u03b1+(1-\u03b1)p. Note that we do not \nallow perfect knowledge of the probability of a changepoint, \u03b1. Instead, an estimated \nvalue of \u03b1 is used based on the number of changepoints detected in the data series up \nto time t. \n\n\f \n\nThe FF model nests two simpler FF models that are psychologically interesting. If the \ntwitchiness threshold parameter C becomes arbitrarily large, the model never detects a \nchange and instead becomes a continuous running average model. Predictions from \nthis model are simply a boxcar smooth of the data. Alternatively, if we assume no \nmemory the model must based each prediction on only the previous stimulus (i.e., \nM=1). Above, in Figure 3, we labeled the complete FF model as FF1, the boxcar \nmodel as FF2 and the memoryless model FF3. \n\nFigure 3 showed that the complete FF model (FF1) fit the data from all observers \nsignificantly better than either the boxcar model (FF2) or the memoryless model \n(FF3). Exceptions were observers PH, DN and ML, for whom all three FF model fit \nequally well. This result suggests that our observers were (mostly) doing more than \njust keeping a running average of the data, or using only the most recent observation. \nThe FF1 model fit the data about as well as the Bayesian models for all observers \nexcept MY and MS. Note that, in general, the FF1 and Bayesian model fits are very \ngood: the average city block distance between the human data and the model \nprediction is around 0.75 (out of 10) buttons on both the x- and y-axes. \n\n6 C o n c l u s i o n \n\nWe used an online prediction task to study changepoint detection. Human observers \nhad to predict the next observation in stochastic sequences containing random \nchangepoints. We showed that some observers are too \u201ctwitchy\u201d: They perform \npoorly on the prediction task because they see changes where only random fluctuation \nexists. Other observers are not twitchy enough, and they perform poorly because they \nfail to see small changes. We developed a Bayesian changepoint detection model that \nperformed the task optimally, and also provided a good fit to human data when \nsub-optimal parameter settings were used. Finally, we developed a fast-and-frugal \nmodel that showed how participants may be able to perform well at the task using \nminimal information and simple decision heuristics. \n\nA c k n o w l e d g m e n t s \n\nWe thank Eric-Jan Wagenmakers and Mike Yi for useful discussions related to this \nwork. This work was supported in part by a grant from the US Air Force Office of \nScientific Research (AFOSR grant number FA9550-04-1-0317). \n\nR e f e r e n c e s \n\n[1] Gilovich, T., Vallone, R. and Tversky, A. (1985). The hot hand in basketball: on the \nmisperception of random sequences. Cognitive Psychology17, 295-314. \n\n[2] Albright, S.C. (1993a). A statistical analysis of hitting streaks in baseball. Journal of the \nAmerican Statistical Association, 88, 1175-1183. \n\n[3] Stephens, D.A. (1994). Bayesian retrospective multiple changepoint identification. Applied \nStatistics 43(1), 159-178. \n\n[4] Carlin, B.P., Gelfand, A.E., & Smith, A.F.M. (1992). Hierarchical Bayesian analysis of \nchangepoint problems. Applied Statistics 41(2), 389-405. \n\n[5] Green, P.J. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian \nmodel determination. Biometrika 82(4), 711-732. \n\n[6] Gigerenzer, G., & Goldstein, D.G. (1996). Reasoning the fast and frugal way: Models of \nbounded rationality. Psychological Review, 103, 650-669. \n\n\f", "award": [], "sourceid": 2946, "authors": [{"given_name": "Mark", "family_name": "Steyvers", "institution": null}, {"given_name": "Scott", "family_name": "Brown", "institution": null}]}