{"title": "OnACID: Online Analysis of Calcium Imaging Data in Real Time", "book": "Advances in Neural Information Processing Systems", "page_first": 2381, "page_last": 2391, "abstract": "Optical imaging methods using calcium indicators are critical for monitoring the activity of large neuronal populations in vivo. Imaging experiments typically generate a large amount of data that needs to be processed to extract the activity of the imaged neuronal sources. While deriving such processing algorithms is an active area of research, most existing methods require the processing of large amounts of data at a time, rendering them vulnerable to the volume of the recorded data, and preventing real-time experimental interrogation. Here we introduce OnACID, an Online framework for the Analysis of streaming Calcium Imaging Data, including i) motion artifact correction, ii) neuronal source extraction, and iii) activity denoising and deconvolution. Our approach combines and extends previous work on online dictionary learning and calcium imaging data analysis, to deliver an automated pipeline that can discover and track the activity of hundreds of cells in real time, thereby enabling new types of closed-loop experiments. We apply our algorithm on two large scale experimental datasets, benchmark its performance on manually annotated data, and show that it outperforms a popular offline approach.", "full_text": "OnACID: Online Analysis of Calcium Imaging Data\n\nin Real Time\n\nAndrea Giovannucci\u20201\n\nJohannes Friedrich\u2020\u22171\n\nMatthew Kaufman\u2021\n\nAnne K. Churchland\u2021\n\nDmitri Chklovskii\u2020\n\nLiam Paninski\u2217\n\nEftychios A. Pnevmatikakis\u20202\n\n\u2021 Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724\n\n\u2020 Flatiron Institute, New York, NY 10010\n\u2217 Columbia University, New York, NY 10027\n\n{agiovannucci, jfriedrich, dchklovskii, epnevmatikakis}@flatironinstitute.org\n\n{mkaufman, churchland}@cshl.edu\n\nliam@stat.columbia.edu\n\nAbstract\n\nOptical imaging methods using calcium indicators are critical for monitoring the\nactivity of large neuronal populations in vivo. Imaging experiments typically\ngenerate a large amount of data that needs to be processed to extract the activity\nof the imaged neuronal sources. While deriving such processing algorithms is\nan active area of research, most existing methods require the processing of large\namounts of data at a time, rendering them vulnerable to the volume of the recorded\ndata, and preventing real-time experimental interrogation. Here we introduce\nOnACID, an Online framework for the Analysis of streaming Calcium Imaging\nData, including i) motion artifact correction, ii) neuronal source extraction, and iii)\nactivity denoising and deconvolution. Our approach combines and extends previous\nwork on online dictionary learning and calcium imaging data analysis, to deliver\nan automated pipeline that can discover and track the activity of hundreds of cells\nin real time, thereby enabling new types of closed-loop experiments. We apply our\nalgorithm on two large scale experimental datasets, benchmark its performance on\nmanually annotated data, and show that it outperforms a popular of\ufb02ine approach.\n\n1\n\nIntroduction\n\nCalcium imaging methods continue to gain traction among experimental neuroscientists due to their\ncapability of monitoring large targeted neuronal populations across multiple days or weeks with\ndecisecond temporal and single-neuron spatial resolution. To infer the neural population activity\nfrom the raw imaging data, an analysis pipeline is employed which typically involves solving the\nfollowing problems (all of which are still areas of active research): i) correcting for motion artifacts\nduring the imaging experiment, ii) identifying/extracting the sources (neurons and axonal or dendritic\nprocesses) in the imaged \ufb01eld of view (FOV), and iii) denoising and deconvolving the neural activity\nfrom the dynamics of the expressed calcium indicator.\n\n1These authors contributed equally to this work.\n2To whom correspondence should be addressed.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fThe \ufb01ne spatiotemporal resolution of calcium imaging comes at a data rate cost; a typical two-photon\n(2p) experiment on a 512\u00d7512 pixel large FOV imaged at 30Hz, generates \u223c50GB of data (in 16-bit\ninteger format) per hour. These rates can be signi\ufb01cantly higher for other planar and volumetric\nimaging techniques, e.g., light-sheet [1] or SCAPE imaging [4], where the data rates can exceed 1TB\nper hour. The resulting data deluge poses a signi\ufb01cant challenge.\nOf the three basic pre-processing problems described above, the problem of source extraction faces\nthe most severe scalability issues. Popular approaches reshape the data movies into a large array\nwith dimensions (#pixels) \u00d7 (#timesteps), that is then factorized (e.g., via independent component\nanalysis [20] or constrained non-negative matrix factorization (CNMF) [26]) to produce the locations\nin the FOV and temporal activities of the imaged sources. While effective for small or medium\ndatasets, direct factorization can be impractical, since a typical experiment can quickly produce\ndatasets larger than the available RAM. Several strategies have been proposed to enhance scalability,\nincluding parallel processing [9], spatiotemporal decimation [10], dimensionality reduction [23], and\nout-of-core processing [13]. While these approaches enable ef\ufb01cient processing of larger datasets,\nthey still require signi\ufb01cant storage, power, time, and memory resources.\nApart from recording large neural populations, optical methods can also be used for stimulation\n[5]. Combining optogenetic methods for recording and perturbing neural ensembles opens the\ndoor to exciting closed-loop experiments [24, 15, 8], where the pattern of the stimulation can be\ndetermined based on the recorded activity during behavior. In a typical closed-loop experiment, the\nmonitored/perturbed regions of interest (ROIs) have been preselected by analyzing of\ufb02ine a previous\ndataset from the same FOV. Monitoring the activity of a ROI, which usually corresponds to a soma,\ntypically entails averaging the \ufb02uorescence over the corresponding ROI, resulting in a signal that is\nonly a proxy for the actual neural activity and which can be sensitive to motion artifacts and drifts, as\nwell as spatially overlapping sources, background/neuropil contamination, and noise. Furthermore, by\npreselecting the ROIs, the experimenter is unable to detect and incorporate new sources that become\nactive later during the experiment, which prevents the execution of truly closed-loop experiments.\nIn this paper, we present an Online, single-pass, algorithmic framework for the Analysis of Calcium\nImaging Data (OnACID). Our framework is highly scalable with minimal memory requirements,\nas it processes the data in a streaming fashion one frame at a time, while keeping in memory a set\nof low dimensional suf\ufb01cient statistics and a small minibatch of the last data frames. Every frame\nis processed in four sequential steps: i) The frame is registered against the previous denoised (and\nregistered) frame to correct for motion artifacts. ii) The \ufb02uorescence activity of the already detected\nsources is tracked. iii) Newly appearing neurons and processes are detected and incorporated to the\nset of existing sources. iv) The \ufb02uorescence trace of each source is denoised and deconvolved to\nprovide an estimate of the underlying spiking activity.\nOur algorithm integrates and extends the online NMF algorithm of [19], the CNMF source extraction\nalgorithm of [26], and the near-online deconvolution algorithm of [11], to provide a framework\ncapable of real time identi\ufb01cation and processing of hundreds of neurons in a typical 2p experiment\n(512\u00d7512 pixel wide FOV imaged at 30Hz), enabling novel designs of closed-loop experiments.\nWe apply OnACID to two large-scale (50 and 65 minute long) mouse in vivo 2p datasets; our\nalgorithm can \ufb01nd and track hundreds of neurons faster than real-time, and outperforms the CNMF\nalgorithm of [26] benchmarked on multiple manual annotations using a precision-recall framework.\n\n2 Methods\n\nWe illustrate OnACID in process in Fig. 1. At the beginning of the experiment (Fig. 1-left), only\na few components are active, as shown in the panel A by the max-correlation image3, and these\nare detected by the algorithm (Fig. 1B). As the experiment proceeds more neurons activate and are\nsubsequently detected by OnACID (Fig. 1 middle, right) which also tracks their activity across time\n(Fig. 1C). See also Supplementary Movie 1 for an example in simulated data.\nNext, we present the steps of OnACID in more detail.\n\n3The correlation image (CI) at every pixel is equal to the average temporal correlation coef\ufb01cient between\nthat pixel and its neighbors [28] (8 neighbors were used for our analysis). The max-correlation image is obtained\nby computing the CI for each batch of 1000 frames, and then taking the maximum over all these images.\n\n2\n\n\fFigure 1: Illustration of the online data analysis process. Snapshots of the online analysis after\nprocessing 1000 frames (left), 6000 frames (middle), and 90000 frames (right). A) \"Max-correlation\"\nimage of registered data at each snapshot point (see text for de\ufb01nition). B) Spatial footprints (shapes)\nof the components (neurons and processes) found by OnACID up to each point. C) Examples of\nneuron activity traces (marked by contours in panel A and highlighted in red in panel B). As the\nexperiment proceeds, OnACID detects newly active neurons and tracks their activity.\n\nMotion correction: Our online approach allows us to employ a very simple yet effective motion\ncorrection scheme: each denoised dataframe can be used to register the next incoming noisy dataframe.\nTo enhance robustness we use the denoised background/neuropil signal (de\ufb01ned in the next section)\nas a template to align the next dataframe. We use rigid, sub-pixel registration [16], although piecewise\nrigid registration can also be used at an additional computational cost. This simple alignment process\nis not suitable for of\ufb02ine algorithms due to noise in the raw data, leading to the development of\nvarious algorithms based on template matching [14, 23, 25] or Hidden Markov Models [7, 18].\nSource extraction: A standard approach for source extraction is to model the \ufb02uorescence within\na matrix factorization framework [20, 26]. Let Y \u2208 Rd\u00d7T denote the observed \ufb02uorescence across\nspace and time in a matrix format, where d denotes the number of imaged pixels, and T the length\nof the experiment in timepoints. If the number of imaged sources is K, then let A \u2208 Rd\u00d7K denote\nthe matrix where column i encodes the \"spatial footprint\" of the source i. Similarly, let C \u2208 RK\u00d7T\ndenote the matrix where each row encodes the temporal activity of the corresponding source. The\nobserved data matrix can then be expressed as\n\nY = AC + B + E,\n\n(1)\nwhere B, E \u2208 Rd\u00d7T denote matrices for background/neuropil activity and observation noise, re-\nspectively. A common approach, introduced in [26], is to express the background matrix B as a\nlow rank matrix, i.e., B = bf, where b \u2208 Rd\u00d7nb and f \u2208 Rnb\u00d7T denote the spatial and temporal\ncomponents of the low rank background signal, and nb is a small integer, e.g., nb = 1, 2. The CNMF\nframework of [26] operates by alternating optimization of [A, b] given the data Y and estimates of\n[C; f ], and vice versa, where each column of A is constrained to be zero outside of a neighborhood\naround its previous estimate. This strategy exploits the spatial locality of each neuron to reduce the\ncomputational complexity. This framework can be adapted to a data streaming setup using the online\nNMF algorithm of [19], where the observed \ufb02uorescence at time t can be written as\n\nyt = Act + bft + \u03b5t.\n\n(2)\nProceeding in a similar alternating way, the activity of all neurons at time t, ct, and temporal\nbackground ft, given yt and the spatial footprints and background [A, b], can be found by solving\na nonnegative least squares problem, whereas [A, b] can be estimated ef\ufb01ciently as in [19] by only\nkeeping in memory the suf\ufb01cient statistics (where we de\ufb01ne \u02dcct = [ct; ft])\nt Mt\u22121 + 1\n\nMt = t\u22121\n\nt \u02dcct\u02dcc(cid:62)\nt ,\n\n(3)\n\nWt = t\u22121\n\nt Wt\u22121 + 1\n\nt yt\u02dcc(cid:62)\nt ,\n\n3\n\n5s5min30sABC1000 frames6000 frames90000 framesMax-CorrelationImageFoundComponents\fwhile at the same time enforcing the same spatial locality constraints as in the CNMF framework.\nDeconvolution: The online framework presented above estimates the demixed \ufb02uorescence traces\nc1, . . . , cK of each neuronal source. The \ufb02uorescence is a \ufb01ltered version of the underlying neural\nactivity that we want to infer. To further denoise and deconvolve the neural activity from the\ndynamics of the indicator we use the OASIS algorithm [11] that implements the popular spike\ndeconvolution algorithm of [30] in a nearly online fashion by adapting the highly ef\ufb01cient Pool\nAdjacent Violators Algorithm used in isotonic regression [3]. The calcium dynamics is modeled with\ni=1 \u03b3ict\u2212i + st. We use p = 1 here, but can extend\n\na stable autoregressive process of order p, ct =(cid:80)p\n\nto p = 2 to incorporate the indicator rise time [11]. OASIS solves a modi\ufb01ed LASSO problem\n\n1\n\n2(cid:107)\u02c6c \u2212 y(cid:107)2 + \u03bb(cid:107)\u02c6s(cid:107)1\n\nsubject to\n\n\u02c6st = \u02c6ct \u2212 \u03b3\u02c6ct\u22121 \u2265 smin or \u02c6st = 0\n\n(4)\n\nminimize\n\n\u02c6c,\u02c6s\n\nwhere the (cid:96)1 penalty on \u02c6s or the minimal spike size smin can be used to enforce sparsity of the neural\nactivity. The algorithm progresses through each time series sequentially from beginning to end and\nbacktracks only to the most recent spike. We can further restrict the lag to few frames, to obtain a\ngood approximate solution applicable for real-time experiments.\nDetecting new components: The approach explained above enables tracking the activity of a \ufb01xed\nnumber of sources, and will ignore neurons that become active later in the experiment. To account\nfor a variable number of sources in an online NMF setting, [12] proposes to add a new random\ncomponent when the correlation coef\ufb01cient between each data frame and its representation in terms of\nthe current factors is lower than a threshold. This approach is insuf\ufb01cient here since the footprint of a\nnew neuron in the whole FOV is typically too small to modify the correlation coef\ufb01cient signi\ufb01cantly.\nWe approach the problem by introducing a buffer that contains the last lb instances of the residual\nsignal rt = yt \u2212 Act \u2212 bft, where lb is a reasonably small number, e.g., lb = 100. On this buffer,\nsimilarly to [26], we perform spatial smoothing with a Gaussian kernel with radius similar to the\nexpected neuron radius, and then search for the point in space that explains the maximum variance.\nNew candidate components anew, and cnew are estimated by performing a local rank-1 NMF of the\nresidual matrix restricted to a \ufb01xed neighborhood around the point of maximal variance.\nTo limit false positives, the candidate component is screened for quality. First, to prevent noise\nover\ufb01tting, the shape anew must be signi\ufb01cantly correlated (e.g., \u03b8s \u223c 0.8\u2212 0.9) to the residual buffer\naveraged over time and restricted to the spatial extent of anew. Moreover, if anew signi\ufb01cantly overlaps\nwith any of the existing components, then its temporal component cnew must not be highly correlated\nwith the corresponding temporal components; otherwise we reject it as a possible duplicate of an\nexisting component. Once a new component is accepted, [A, b], [C; f ] are augmented with anew and\ncnew respectively, and the suf\ufb01cient statistics are updated as follows:\n\nWt =\n\nWt,\n\nYbufc(cid:62)\n\nnew\n\n1\nt\n\n,\n\nMt =\n\n1\nt\n\ntMt\ncnew \u02dcC(cid:62)\n\nbuf\n\n\u02dcCbufc(cid:62)\n(cid:107)cnew(cid:107)2\n\nnew\n\n,\n\n(5)\n\n(cid:20)\n\n(cid:21)\n\n(cid:20)\n\n(cid:21)\n\nwhere Ybuf, \u02dcCbuf denote the matrices Y, [C; f ], restricted to the last lb frames that the buffer stores.\nThis process is repeated until no new components are accepted, at which point the next frame is read\nand processed. The whole online procedure is described in Algorithm 1; the supplement includes\npseudocode description of all the referenced routines.\nInitialization: To initialize our algorithm we use the CNMF algorithm on a short initial batch of\ndata of length Tb, (e.g., Tb = 1000). The suf\ufb01cient statistics are initialized from the components that\nthe of\ufb02ine algorithm \ufb01nds according to (3). To ensure that new components are also initialized in\nthe darker parts of the FOV, each data frame is normalized with the (running) mean for every pixel,\nduring both the of\ufb02ine and the online phases.\nAlgorithmic Speedups: Several algorithmic and computational schemes are employed to boost the\nspeed of the algorithm and make it applicable to real-time large-scale experiments. In [19] block\ncoordinate descent is used to update the factors A, warm started at the value from the previous\niteration. The same trick is used here not only for A, but also for C, since the calcium traces are\ncontinuous and typically change slowly. Moreover, the temporal traces of components that do not\nspatially overlap with each other can be updated simultaneously in vector form; we use a simple\ngreedy scheme to partition the components into spatially non-overlapping groups.\nSince neurons\u2019 shapes are not expected to change at a fast timescale, updating their values (i.e.,\nrecomputing A and b) is not required at every timepoint; in practice we update every lb timesteps.\n\n4\n\n\ftimestep t(cid:48), rest of parameters.\n\n(cid:46) Initialize suf\ufb01cient statistics\n(cid:46) Alg. S1-S2\n(cid:46) Initialize residual buffer\n\n(cid:46) [16]\n(cid:46) Alg. S3\n(cid:46) [11]\n\nAlgorithm 1 ONACID\nRequire: Data matrix Y , initial estimates A, b, C, f , S, current number of components K, current\n1: W = Y [:, 1 : t(cid:48)]C(cid:62)/t(cid:48)\n2: M = CC(cid:62)/t(cid:48)\n3: G = DETERMINEGROUPS([A, b], K)\n4: Rbuf = [Y \u2212 [A, b][C; f ]][:, t(cid:48) \u2212 lb + 1 : t(cid:48)]\n5: t = t(cid:48)\n6: while there is more data do\n7:\n8:\n9:\n10:\n11:\n12:\n13:\n14:\n15:\n16:\n17: return A, b, C, f , S\n\nt \u2190 t + 1\nyt \u2190 ALIGNFRAME(yt, bft\u22121)\n[ct; ft] \u2190 UPDATETRACES([A, b], [ct\u22121; ft\u22121], yt,G)\nC, S \u2190 OASIS(C, \u03b3, smin, \u03bb)\n[A, b], [C, f ], K,G, Rbuf, W, M \u2190\nRbuf \u2190 [Rbuf[:, 2 : lb], yt \u2212 Act \u2212 bft]\nif mod (t \u2212 t(cid:48), lb) = 0 then\n\nDETECTNEWCOMPONENTS([A, b], [C, f ], K,G, Rbuf, yt, W, M )\n\nW, M \u2190 UPDATESUFFSTATISTICS(W, M, yt, [ct; ft])\n[A, b] \u2190 UPDATESHAPES[W, M, [A, b]]\n\n(cid:46) Alg. S4\n(cid:46) Update residual buffer\n(cid:46) Update W, M, [A, b] every lb timesteps\n(cid:46) Equation (3)\n(cid:46) Alg. S5\n\nFigure 2: Application to simulated data. A) Detected and missed components. B) Tukey boxplot\nof spike train correlations with ground truth. Online deconvolution recovers spike trains well and the\naccuracy increases with the allowed lag in spike assignment. C) Processing time is less than 33 ms\nfor all the frames.\n\nAdditionally, the suf\ufb01cient statistics Wt, Mt are only needed for updating the estimates of [A, b] so\nthey can be updated only when required. Motion correction can be sped up by estimating the motion\nonly on a small (active) contiguous part of the FOV. Finally, as shown in [10], spatial decimation can\nbring signi\ufb01cant speed bene\ufb01ts without compromising the quality of the results.\nSoftware: OnACID is implemented in Python and is available at https://github.com/\nsimonsfoundation/caiman as part of the CaImAn package [13].\n\n3 Results\n\nBenchmarking on simulated data: To compare to ground truth spike trains, we simulated a 2000\nframe dataset taken at 30Hz over a 256\u00d7 256 pixel wide FOV containing 400 \"donut\" shaped neurons\nwith Poisson spike trains (see supplement for details). OnACID was initialized on the \ufb01rst 500 frames.\nDuring initialization, 265 active sources were accurately detected (Fig. S2). After the full 2000\nframes, the algorithm had detected and tracked all active sources, plus one false positive (Fig. 2A).\nAfter detecting a neuron, we need to extract its spikes with a short time-lag, to enable interesting\nclosed loop experiments. To quantify performance we measured the correlation of the inferred spike\ntrain with the ground truth (Fig. 2B). We varied the lag in the online estimator, i.e. the number of\nfuture samples observed before assigning a spike at time zero. Lags of 2-5 yield already similar\n\n5\n\nABCtrue posfalse posfalse neglag (frames)202428200015001000500frameProcessing time [ms]o(cid:31)ine\u221e50.2200.40.60.81Pearson\u2019s r\fTable 1: OnACID signi\ufb01cantly outperforms the of\ufb02ine CNMF approach. Benchmarking is against\ntwo independent manual annotations within the precision/recall (and their harmonic mean F1 score)\nframework. For each row-column pair, the column dataset is regarded as the ground truth.\n\nF1 (precision, recall)\n\nOnACID\nCNMF\nLabeler 2\n\nLabeler 1\n\n0.79 (0.87,0.72)\n0.71 (0.74, 0.69)\n0.89 (0.89,0.89)\n\nLabeler 2\n\n0.78 (0.86,0.71)\n0.71 (0.75,0.68)\n\n-\n\nCNMF\n\n0.79 (0.83,0.75)\n\n-\n-\n\nresults as the solution with unrestricted lag. A further requirement for online closed-loop experiments\nis that the computational processing is fast enough. To balance the computational load over frames,\nwe distributed here the shape update over the frames, while still updating each neuron every 30\nframes on average. Because the shape update is the last step of the loop in Algorithm 1, we keep track\nof the time already spent in the iteration and increase or decrease the number of updated neurons\naccordingly. In this way the frame processing rate remained always higher than 30Hz (Fig. 2C).\nApplication to in vivo 2p mouse hippocampal data: Next we considered a larger scale (90K frames,\n480\u00d7 480 pixels) real 2p calcium imaging dataset taken at 30Hz (i.e., 50 minute experiment). Motion\nartifacts were corrected prior to the analysis described below. The online algorithm was initialized on\nthe \ufb01rst 1000 frames of the dataset using a Python implementation of the CNMF algorithm found in\nthe CaImAn package [13]. During initialization 139 active sources were detected; by the end of all\n90K frames, 727 active sources had been detected and tracked (5 of which were discarded due to\ntheir small size).\nBenchmarking against of\ufb02ine processing and manual annotations: We collected manual annota-\ntions from two independent labelers who were instructed to \ufb01nd round or donut shaped neurons of\nsimilar size using the ImageJ Cell Magic Wand tool [31] given i) a movie obtained by removing a\nrunning 20th percentile (as a crude background approximation) and downsampling in time by a factor\nof 10, and ii) the max-correlation image. The goal of this pre-processing was to suppress silent and\npromote active cells. The labelers found respectively 872 and 880 ROIs. We also compared with the\nCNMF algorithm applied to the whole dataset which found 904 sources (805 after \ufb01ltering for size).\nTo quantify performance we used a precision/recall framework similar to [2]. As a distance metric\nbetween two cells we used the Jaccard distance, and the pairing between different annotations was\ncomputed using the Hungarian algorithm, where matches with distance > 0.7 were discarded4.\nTable. 1 summarizes the results within the precision/recall framework. The online algorithm not\nonly matches but outperforms the of\ufb02ine approach of CNMF, reaching high performance values\n(F1 = 0.79 and 0.78 against the two manual annotations, as opposed to 0.71 against both annotations\nfor CNMF). The two annotations matched closely with each other (F1 = 0.89), indicating high\nreliability, whereas OnACID vs CNMF also produced a high score (F1 = 0.79), suggesting signi\ufb01cant\noverlap in the mismatches between the two algorithms against manual annotations.\nFig. 3 offers a more detailed view, where contour plots of the detected components are superimposed\non the max-correlation image for the online (Fig. 3A) and of\ufb02ine (Fig. 3B) algorithms (white) and the\nannotations of Labeler 1 (red) restricted to a 200\u00d7200 pixel part of the FOV. Annotations of matches\nand mismatches between the online algorithm and the two labelers, as well as between the two labelers\nin the entire FOV are shown in Figs. S3-S8. For the automated procedures binary masks and contour\nplots were constructed by thresholding the spatial footprint of each component at a level equal to 0.2\ntimes its maximum value. A close inspection at the matches between the online algorithm and the\nmanual annotation (Fig. 3A-left) indicates that neurons with a strong footprint in the max-correlation\nimage (indicating calcium transients with high amplitude compared to noise and background/neuropil\nactivity) are reliably detected, despite the high neuron density and level of overlap. On the other\nhand, mismatches (Fig. 3B-left) can sometimes be attributed to shape mismatches, manually selected\ncomponents with no signature in the max-correlation image (indicating faint or possibly unclear\nactivity) that are not detected by the online algorithm (false negatives), or small partially visible\nprocesses detected by OnACID but ignored by the labelers (\"false\" positives).\n\n4Note that the Cell Magic Wand Tool by construction, tends to select circular ROI shapes whereas the results\nof the online algorithm do not pose restrictions on the shapes. As a result the computed Jaccard distances tend to\nbe overestimated. This explains our choice of a seemingly high mismatch threshold.\n\n6\n\n\fFigure 3: Application to an in vivo 50min long hippocampal dataset and comparison against\nan of\ufb02ine approach and manual annotation. A-left) Matched inferred locations between the\nonline algorithm (white) and the manual annotation of Labeler 1 (red), superimposed on the max-\ncorrelation image. A-right) False positive (white) and false negative (red) mismatches between the\nonline algorithm and a manual annotation. B) Same for the of\ufb02ine CNMF algorithm (grey) against\nthe same manual annotation (red). The online approach outperforms the CNMF algorithm in the\nprecision/recall framework (F1 score 0.77 vs 0.71). The images are restricted to a 200\u00d7200 pixel\npart of the FOV. Matches and non-matches for the whole FOV are shown in the supplement. C)\nExamples of inferred sources and their traces from the two algorithms and corresponding annotation\nfor three indenti\ufb01ed neurons (also shown with orange arrows in panels A,B left). The algorithm is\ncapable of identifying new neurons once they become active, and track their activity similarly to\nof\ufb02ine approaches. D) Empirical CDF of correlation coef\ufb01cients between the matched traces between\nthe online and the of\ufb02ine approaches over the entire 50 minute traces. The majority of the correlation\ncoef\ufb01cients has very high values suggesting that the online algorithm accurately tracks the neural\nactivity across time (see also correlation coef\ufb01cients for the three examples shown in panel C). E)\nTiming of the online process. Top: Time required per frame when no shapes are updated and no\nneurons are updated (top). The algorithms is always faster than real time in tracking neurons and\nscales mildly with the number of neurons. Time required to update shapes per neuron (middle), and\nadd new neurons (bottom) as a function of the number of neurons. Adding neurons is slower but\noccurs only sporadically affecting only mildly the required processing time (see text for details).\n\n7\n\nHumanO(cid:31)ineHumanOnline12310.60.200.51correlation coe(cid:30)cientCDF100203040508121620242832200500800inferred neuronsCost of adding neuronsCost of tracking neurons\u2019 activityTime [min]Time [ms]onlineo(cid:31)ine123123r=0.89r=0.98r=0.7620 minABCDE0.30.9MatchesMismatches 300real time500700neurons0.450.550.65Time [ms]Cost of updating shapes per neuron20 \u0003m200400600800neurons20406080Time [ms]real time\fFig. 3C shows examples of the traces from three selected neurons. OnACID can detect and track\nneurons with very sparse spiking over the course of the entire 50 minute experiment (Fig. 3C-top),\nand produce traces that are highly correlated with their of\ufb02ine counterparts. To examine the quality\nof the inferred traces (where ground truth collection at such scale is both very strenuous and severely\nimpeded by the presence of background signals and neuropil activity), we compared the traces\nbetween the online algorithm and the CNMF approach on matched pairs of components. Fig. 3D\nshows the empirical cumulative distribution function (CDF) of the correlation coef\ufb01cients from this\ncomparison. The majority of the coef\ufb01cients attain values close to 1, suggesting that the online\nalgorithm can detect new neurons once they become active and then reliably track their activity.\nOnACID is faster than real time on average: In addition to being more accurate, OnACID is also\nconsiderably faster as it required \u223c27 minutes, i.e., \u223c 2\u00d7 faster than real time on average, to analyze\nthe full dataset (2 minutes for initialization and 25 for the online processing) as opposed to \u223c1.5\nhours for the of\ufb02ine approach and \u223c10 hours for each of the annotators (who only select ROIs).\nFig. 3E illustrates the time consumption of the various steps. In the majority of the frames where no\nspatial shapes are being updated and no new neurons are being incorporated, OnACID processing\nspeed exceeds the data rate of 30Hz (Fig. 3E-top), and this processing time scales only mildly with\nthe inclusion of new neurons. The cost of updating shapes and suf\ufb01cient statistics per neuron is also\nvery low (< 1ms), and only scales mildly with the number of existing neurons (Fig. 3E-middle). As\nargued before this cost can be distributed among all the frames while maintaining faster than real time\nprocessing rates. The expensive step appears when detecting and including one or possibly more new\nneurons in the algorithm (Fig. 3E-bottom). Although this occurs only sporadically, several speedups\ncan be potentially employed here to achieve beyond real time at every frame (see also Discussion\nsection), which would facilitate zero-lag closed-loop experiments.\nApplication to in vivo 2p mouse parietal cortex data: As a second application to 2p data we used\na 116,000 frame dataset, taken at 30Hz over a 512\u00d7512 FOV (64min long). The \ufb01rst 3000 frames\nwere used for initialization during which the CNMF algorithm found 442 neurons, before switching\nto OnACID, which by the end of the experiment found a total of 752 neurons (734 after \ufb01ltering for\nsize). Compared to two independent manual annotations of 928 and 875 ROIs respectively, OnACID\nachieved F1 = 0.76, 0.79 signi\ufb01cantly outperforming CNMF (F1 = 0.65, 0.66 respectively). The\nmatches and mismatches between OnACID and Labeler 1 on a 200\u00d7200 pixel part of the FOV are\nshown in Fig. 4A. Full FOV pairings as well as precision/recall metrics are given in Table 2.\n\nTable 2: Comparison of performance of OnACID and the CNMF algorithm using the precision/recall\nframework for the parietal cortex 116000 frames dataset. For each row-column pair, the column\ndataset is regarded as ground truth. The numbers in the parentheses are the precision and recall,\nrespectively, preceded by their harmonic mean (F1 score). OnACID signi\ufb01cantly outperforms the\nof\ufb02ine CNMF approach.\n\nF1 (precision, recall)\n\nLabeler 1\n\nLabeler 2\n\nCNMF\n\nOnACID\nCNMF\nLabeler 2\n\n0.76 (0.86,0.68)\n0.65 (0.70, 0.60)\n0.89 (0.86,0.91)\n\n0.79 (0.86,0.72)\n0.66 (0.74,0.59)\n\n-\n\n0.65 (0.55,0.82)\n\n-\n-\n\nFor this dataset, rigid motion correction was also performed according to the simple method of\naligning each frame to the denoised (and registered) background from the previous frame. Fig. 4B\nshows that this approach produced strikingly similar results to an of\ufb02ine template based, rigid motion\ncorrection method [25]. The difference in the displacements produced by the two methods was less\nthan 1 pixel for all 116,000 frames with standard deviations 0.11 and 0.12 pixel for the x and y\ndirections, respectively. In terms of timing, OnACID processed the dataset in 48 minutes, again faster\nthan real time on average. This also includes the time needed for motion correction, which on average\ntook 5ms per frame (a bit less than 10 minutes in total).\n\n4 Discussion - Future Work\n\nAlthough at \ufb01rst striking, the superior performance of OnACID compared to of\ufb02ine CNMF, for the\ndatasets presented in this work, can be attributed to several factors. Calcium transient events are\nlocalized both in space (spatial footprint of a neuron), and in time (typically 0.3-1s for genetic indica-\n\n8\n\n\fFigure 4: Application to an in vivo 64min long parietal cortex dataset. A-left) Matched inferred\nlocations between the online algorithm (white) and the manual annotation of Labeler 1 (red). A-right)\nFalse positive (white) and false negative (red) mismatches between the online algorithm and a manual\nannotation. B) Displacement vectors estimated by OnACID during motion registration compared to a\ntemplate based algorithm. OnACID estimates the same motion vectors at a sub-pixel resolution (see\ntext for more details).\n\ntors). By looking at a short rolling buffer OnACID is able to more robustly detect activity compared\nto of\ufb02ine approaches that look at all the data simultaneously. Moreover, OnACID searches for new\nactivity in the residuals buffer that excludes the activity of already detected neurons, making it easier\nto detect new overlapping components. Finally, of\ufb02ine CNMF requires the a priori speci\ufb01cation of the\nnumber of components, making it more prone to either false positive or false negative components.\nFor both the datasets presented above, the analysis was done using the same space correlation\nthreshold \u03b8s = 0.9. This strict choice leads to results with high precision and lower recall (see Tables\n1 and 2). Results can be moderately improved by allowing a second pass of the data that can identify\nneurons that were initially not selected. Moreover, by relaxing the threshold the discrepancy between\nthe precision and recall scores can be reduced, with only marginal modi\ufb01cations to the F1 scores\n(data not shown).\nOur current implementation performs all processing serially. In principle, signi\ufb01cant speed gains can\nbe obtained by performing computations not needed at each timestep (updating shapes and suf\ufb01cient\nstatistics) or occur only sporadically (incorporating a new neuron) in a parallel thread with shared\nmemory. Moreover, different online dictionary learning algorithms that do not require the solution of\nan inverse problem at each timestep can potentially further speed up our framework [17].\nFor detecting centroids of new sources OnACID examines a static image obtained by computing the\nvariance across time of the spatially smoother residual buffer. While this approach works very well in\npractice it effectively favors shapes looking similar to a pre-de\ufb01ned Gaussian blob (when spatially\nsmoothed). Different approaches for detecting neurons in static images can be possibly used here,\ne.g., [22], [2], [29], [27].\nApart from facilitating closed-loop behavioral experiments and rapid general calcium imaging data\nanalysis, our online pipeline can be potentially employed to future, optical-based, brain computer\ninterfaces [6, 21] where high quality real-time processing is critical to their performance. These\ndirections will be pursued in future work.\n\nAcknowledgments\n\nWe thank Sue Ann Koay, Jeff Gauthier and David Tank (Princeton University) for sharing their cortex\nand hippocampal data with us. We thank Lindsey Myers, Sonia Villani and Natalia Roumelioti for\nproviding manual annotations. We thank Daniel Barabasi (Cold Spring Harbor Laboratory) for useful\ndiscussions. AG, DC, and EAP were internally funded by the Simons Foundation. Additional support\nwas provided by SNSF P300P2_158428 (JF), and NIH BRAIN Initiative R01EB22913, DARPA\nN66001-15-C-4032, IARPA MICRONS D16PC00003 (LP).\n\n9\n\n20\u0003m0.20.8HumanOnlineMatchesMismatches Ao(cid:30)ineo(cid:30)ineonlineonline40-4-8x shift [pixels]B200406040-4-8-12Time [min]y shift [pixels]Displacements\fReferences\n[1] Misha B Ahrens, Michael B Orger, Drew N Robson, Jennifer M Li, and Philipp J Keller.\nWhole-brain functional imaging at cellular resolution using light-sheet microscopy. Nature\nmethods, 10(5):413\u2013420, 2013.\n\n[2] Noah Apthorpe, Alexander Riordan, Robert Aguilar, Jan Homann, Yi Gu, David Tank, and\nH Sebastian Seung. Automatic neuron detection in calcium imaging data using convolutional\nnetworks. In Advances in Neural Information Processing Systems, pages 3270\u20133278, 2016.\n\n[3] Richard E Barlow, David J Bartholomew, JM Bremner, and H Daniel Brunk. Statistical inference\nunder order restrictions: The theory and application of isotonic regression. Wiley New York,\n1972.\n\n[4] Matthew B Bouchard, Venkatakaushik Voleti, C\u00e9sar S Mendes, Clay Lace\ufb01eld, Wesley B\nGrueber, Richard S Mann, Randy M Bruno, and Elizabeth MC Hillman. Swept confocally-\naligned planar excitation (scape) microscopy for high-speed volumetric imaging of behaving\norganisms. Nature photonics, 9(2):113\u2013119, 2015.\n\n[5] Edward S Boyden, Feng Zhang, Ernst Bamberg, Georg Nagel, and Karl Deisseroth. Millisecond-\ntimescale, genetically targeted optical control of neural activity. Nature neuroscience, 8(9):1263\u2013\n1268, 2005.\n\n[6] Kelly B Clancy, Aaron C Koralek, Rui M Costa, Daniel E Feldman, and Jose M Carmena.\nVolitional modulation of optically recorded calcium signals during neuroprosthetic learning.\nNature neuroscience, 17(6):807\u2013809, 2014.\n\n[7] Daniel A Dombeck, Anton N Khabbaz, Forrest Collman, Thomas L Adelman, and David W\nTank. Imaging large-scale neural activity with cellular resolution in awake, mobile mice. Neuron,\n56(1):43\u201357, 2007.\n\n[8] Valentina Emiliani, Adam E Cohen, Karl Deisseroth, and Michael H\u00e4usser. All-optical interro-\n\ngation of neural circuits. Journal of Neuroscience, 35(41):13917\u201313926, 2015.\n\n[9] Jeremy Freeman, Nikita Vladimirov, Takashi Kawashima, Yu Mu, Nicholas J Sofroniew,\nDavis V Bennett, Joshua Rosen, Chao-Tsung Yang, Loren L Looger, and Misha B Ahrens.\nMapping brain activity at scale with cluster computing. Nature methods, 11(9):941\u2013950, 2014.\n[10] Johannes Friedrich, Weijian Yang, Daniel Soudry, Yu Mu, Misha B Ahrens, Rafael Yuste,\nDarcy S Peterka, and Liam Paninski. Multi-scale approaches for high-speed imaging and\nanalysis of large neural populations. bioRxiv, page 091132, 2016.\n\n[11] Johannes Friedrich, Pengcheng Zhou, and Liam Paninski. Fast online deconvolution of calcium\n\nimaging data. PLOS Computational Biology, 13(3):e1005423, 2017.\n\n[12] Sahil Garg, Irina Rish, Guillermo Cecchi, and Aurelie Lozano. Neurogenesis-inspired dictionary\nlearning: Online model adaption in a changing world. arXiv preprint arXiv:1701.06106, 2017.\n[13] A Giovannucci, J Friedrich, B Deverett, V Staneva, D Chklovskii, and E Pnevmatikakis. Caiman:\nAn open source toolbox for large scale calcium imaging data analysis on standalone machines.\nCosyne Abstracts, 2017.\n\n[14] David S Greenberg and Jason ND Kerr. Automated correction of fast motion artifacts for\ntwo-photon imaging of awake animals. Journal of neuroscience methods, 176(1):1\u201315, 2009.\n[15] Logan Grosenick, James H Marshel, and Karl Deisseroth. Closed-loop and activity-guided\n\noptogenetic control. Neuron, 86(1):106\u2013139, 2015.\n\n[16] Manuel Guizar-Sicairos, Samuel T Thurman, and James R Fienup. Ef\ufb01cient subpixel image\n\nregistration algorithms. Optics letters, 33(2):156\u2013158, 2008.\n\n[17] Tao Hu, Cengiz Pehlevan, and Dmitri B Chklovskii. A hebbian/anti-hebbian network for online\nsparse dictionary learning derived from symmetric matrix factorization. In Signals, Systems and\nComputers, 2014 48th Asilomar Conference on, pages 613\u2013619. IEEE, 2014.\n\n[18] Patrick Kaifosh, Jeffrey D Zaremba, Nathan B Danielson, and Attila Losonczy. Sima: Python\nsoftware for analysis of dynamic \ufb02uorescence imaging data. Frontiers in neuroinformatics,\n8:80, 2014.\n\n[19] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online learning for matrix\nfactorization and sparse coding. Journal of Machine Learning Research, 11(Jan):19\u201360, 2010.\n\n10\n\n\f[20] Eran A Mukamel, Axel Nimmerjahn, and Mark J Schnitzer. Automated analysis of cellular\n\nsignals from large-scale calcium imaging data. Neuron, 63(6):747\u2013760, 2009.\n\n[21] Daniel J O\u2019shea, Eric Trautmann, Chandramouli Chandrasekaran, Sergey Stavisky, Jonathan C\nKao, Maneesh Sahani, Stephen Ryu, Karl Deisseroth, and Krishna V Shenoy. The need for\ncalcium imaging in nonhuman primates: New motor neuroscience and brain-machine interfaces.\nExperimental neurology, 287:437\u2013451, 2017.\n\n[22] Marius Pachitariu, Adam M Packer, Noah Pettit, Henry Dalgleish, Michael Hausser, and\nManeesh Sahani. Extracting regions of interest from biological images with convolutional\nsparse block coding. In Advances in Neural Information Processing Systems, pages 1745\u20131753,\n2013.\n\n[23] Marius Pachitariu, Carsen Stringer, Sylvia Schr\u00f6der, Mario Dipoppa, L Federico Rossi, Matteo\nCarandini, and Kenneth D Harris. Suite2p: beyond 10,000 neurons with standard two-photon\nmicroscopy. bioRxiv, page 061507, 2016.\n\n[24] Adam M Packer, Lloyd E Russell, Henry WP Dalgleish, and Michael H\u00e4usser. Simultaneous\nall-optical manipulation and recording of neural circuit activity with cellular resolution in vivo.\nNature Methods, 12(2):140\u2013146, 2015.\n\n[25] Eftychios A Pnevmatikakis and Andrea Giovannucci. Normcorre: An online algorithm for\npiecewise rigid motion correction of calcium imaging data. Journal of Neuroscience Methods,\n291:83\u201394, 2017.\n\n[26] Eftychios A Pnevmatikakis, Daniel Soudry, Yuanjun Gao, Timothy A Machado, Josh Merel,\nDavid Pfau, Thomas Reardon, Yu Mu, Clay Lace\ufb01eld, Weijian Yang, et al. Simultaneous\ndenoising, deconvolution, and demixing of calcium imaging data. Neuron, 89(2):285\u2013299,\n2016.\n\n[27] Stephanie Reynolds, Therese Abrahamsson, Renaud Schuck, P Jesper Sj\u00f6str\u00f6m, Simon R\nSchultz, and Pier Luigi Dragotti. Able: an activity-based level set segmentation algorithm for\ntwo-photon calcium imaging data. eNeuro, pages ENEURO\u20130012, 2017.\n\n[28] Spencer L Smith and Michael H\u00e4usser. Parallel processing of visual space by neighboring\n\nneurons in mouse visual cortex. Nature neuroscience, 13(9):1144\u20131149, 2010.\n\n[29] Quico Spaen, Dorit S Hochbaum, and Roberto As\u00edn-Ach\u00e1. HNCcorr: A novel combinatorial\napproach for cell identi\ufb01cation in calcium-imaging movies. arXiv preprint arXiv:1703.01999,\n2017.\n\n[30] Joshua T Vogelstein, Adam M Packer, Timothy A Machado, Tanya Sippy, Baktash Babadi,\nRafael Yuste, and Liam Paninski. Fast nonnegative deconvolution for spike train inference from\npopulation calcium imaging. Journal of neurophysiology, 104(6):3691\u20133704, 2010.\n\n[31] Theo Walker. Cell magic wand tool, 2014.\n\n11\n\n\f", "award": [], "sourceid": 1403, "authors": [{"given_name": "Andrea", "family_name": "Giovannucci", "institution": "Flatiron Institute, Simons Foundation"}, {"given_name": "Johannes", "family_name": "Friedrich", "institution": "Columbia University"}, {"given_name": "Matt", "family_name": "Kaufman", "institution": "Cold Spring Harbor Laboratory"}, {"given_name": "Anne", "family_name": "Churchland", "institution": "Cold Spring Harbor Laboratory"}, {"given_name": "Dmitri", "family_name": "Chklovskii", "institution": "Simons Foundation"}, {"given_name": "Liam", "family_name": "Paninski", "institution": "Columbia University"}, {"given_name": "Eftychios", "family_name": "Pnevmatikakis", "institution": "Flatiron Institute"}]}