{"title": "Ultrafast Monte Carlo for Statistical Summations", "book": "Advances in Neural Information Processing Systems", "page_first": 673, "page_last": 680, "abstract": null, "full_text": "Ultrafast Monte Carlo for Kernel Estimators and Generalized Statistical Summations\nMichael P. Holmes, Alexander G. Gray, and Charles Lee Isbell, Jr. College Of Computing Georgia Institute of Technology Atlanta, GA 30327 {mph, agray, isbell}@cc.gatech.edu\n\nAbstract\nMachine learning contains many computational bottlenecks in the form of nested summations over datasets. Kernel estimators and other methods are burdened by these expensive computations. Exact evaluation is typically O(n2 ) or higher, which severely limits application to large datasets. We present a multi-stage stratified Monte Carlo method for approximating such summations with probabilistic relative error control. The essential idea is fast approximation by sampling in trees. This method differs from many previous scalability techniques (such as standard multi-tree methods) in that its error is stochastic, but we derive conditions for error control and demonstrate that they work. Further, we give a theoretical sample complexity for the method that is independent of dataset size, and show that this appears to hold in experiments, where speedups reach as high as 1014 , many orders of magnitude beyond the previous state of the art.\n\n1\n\nIntroduction\n\nMany machine learning methods have computational bottlenecks in the form of nested summations that become intractable for large datasets. We are particularly motivated by the nonparametric kernel estimators (e.g. kernel density estimation), but a variety of other methods require computations of similar form. In this work we formalize the general class of nested summations and present a new multi-stage Monte Carlo method for approximating any problem in the class with rigorous relative error control. Key to the efficiency of this method is the use of tree-based data stratification, i.e. sampling in trees. We derive error guarantees and sample complexity bounds, with the intriguing result that runtime depends not on dataset size but on statistical features such as variance and kurtosis, which can be controlled through stratification. We also present experiments that validate these theoretical results and demonstrate tremendous speedup over the prior state of the art. Previous approaches to algorithmic acceleration of this kind fall into roughly two groups: 1) methods that run non-accelerated algorithms on subsets of the data, typically without error bounds, and 2) multi-tree methods with deterministic error bounds. The former are of less interest due to the lack of error control, while the latter are good when exact error control is required, but have built-in overconservatism that limits speedup, and are difficult to extend to new problems. Our Monte Carlo approach offers much larger speedup and a generality that makes it simple to adapt to new problems, while retaining strong error control. While there are non-summative problems to which the standard multi-tree methodology is applicable and our Monte Carlo method is not, our method appears to give greater speedup by many orders of magnitude on problems where both methods can be used. In summary, this work makes the following contributions: formulation of the class of generalized nested data summations; derivation of recursive Monte Carlo algorithms with rigorous error guarantees for this class of computation; derivation of sample complexity bounds showing no explicit 1\n\n\f\ndependence on dataset size; variance-driven tree-based stratified sampling of datasets, which allows Monte Carlo approximation to be effective with small sample sizes; application to kernel regression and kernel conditional density estimation; empirical demonstration of speedups as high as 10 14 on datasets with points numbering in the millions. It is the combination of all these elements that enables our method to perform so far beyond the previous state of the art.\n\n2\n\nProblem definition and previous work\n\nWe first illustrate the problem class by giving expressions for the least-squares cross-validation scores used to optimize bandwidths in kernel regression (KR), kernel density estimation (KDE), and kernel conditional density estimation (KCDE):\nSK R SK D E SK C D E !2 P 1X j =i Kh (||xi - xj ||)yj yi - P = ni j =i Kh (||xi - xj ||) ,, XXZ X 2 1X 1 Kh (||x - xj ||)Kh (||x - xk ||)dx - = Kh (||xi - xj ||) ni (n - 1)2 (n - 1) j =i k=i j =i R P P ,, Kh1 (y - yj )Kh1 (y - yk )dy 1X j =i k=i Kh2 (||xi - xj ||)Kh2 (||xi - xk ||) P P = ni Kh2 (||xi - xj ||)Kh2 (||xi - xk ||) j =i k=i P Kh2 (||xi - xj ||)Kh1 (yi - yj ) j =i P . -2 j =i Kh2 (||xi - xj ||)\n\nThese nested sums have quadratic and cubic computation times that are intractable for large datasets. We would like a method for quickly approximating these and similar computations in a simple and general way. We begin by formulating an inductive generalization of the problem class: i B (Xc ) f (Xc , Xi ) (1)\nI (Xc )\n\nG (Xc ) B (Xc ) |\n\ni\n\nI (Xc )\n\nf (Xc , G1 (Xc , Xi ) , G2 (Xc , Xi ) , . . . ) .\n\n(2)\n\nB represents the base case, in which a tuple of constant arguments Xc may be specified and a tuple of variable arguments Xi is indexed by a set I , which may be a function of Xc . For instance, in the innermost leave-one-out summations of SK R , Xc is the single point xi while I (Xc ) indexes all single points other than xi . Note that |I | is the number of terms in a summation of type B , and therefore represents the base time complexity. Whenever I consists of all k -tuples or leave-one-out k -tuples, the base complexity is O(nk ), where n is the size of the dataset. The inductive case G is either: 1) the base case B , or 2) a sum where the arguments to the summand function are Xc and a series of nested instances of type G. In SK R the outermost summation is an example of this. The base complexity here is |I | multiplied by the maximum base complexity among the nested instances, e.g. if, as in SK R , I is all single points and the most expensive inner G is O(n), then the overall base complexity is O(n2 ). Previous work. Past efforts at scaling this class of computation have fallen into roughly two groups. First are methods where data is simply subsampled before running a non-accelerated algorithm. Stochastic gradient descent and its variants (e.g. [1]) are prototypical here. While these approaches can have asymptotic convergence, there are no error guarantees for finite sample sizes. This is not show-stopping in practice, but the lack of quality assurance is a critical shortcoming. Our approach also exploits the speedup that comes from sampling, but provides a rigorous relative error guarantee and is able to automatically determine the necessary sample size to provide that guarantee. The other main class of acceleration methods consists of those employing \"higher order divide and conquer\" or multi-tree techniques that give either exact answers or deterministic error bounds (e.g. [2, 3, 4]). These approaches apply to a broad class of \"generalized n-body problems\" (GNPs), and feature the use of multiple spatial partitioning structures such as k d-trees or ball trees to decompose and reuse portions of computational work. While the class of GNPs has yet to be formally defined, the generalized summations we address are clearly related and have at least partial overlap. 2\n\n\f\nThe standard multi-tree methodology has three significant drawbacks. First, although it gives deterministic error bounds, the bounds are usually quite loose, resulting in overconservatism that prevents aggressive approximation that could give greater speed. Second, creating a new multi-tree method to accelerate a given algorithm requires complex custom derivation of error bounds and pruning rules. Third, the standard multi-tree approach is conjectured to reduce O(n p ) computations at best to O(nlog p ). This still leaves an intractable computation for p as small as 4. In [5], the first of these concerns began to be addressed by employing sample-based bounds within a multi-tree error propagation framework. The present work builds on that idea by moving to a fully Monte Carlo scheme where multiple trees are used for variance-reducing stratification. Error is rigorously controlled and driven by sample variance, allowing the Monte Carlo approach to make aggressive approximations and avoid the overconservatism of deterministic multi-tree methods. This yields greater speedups by many orders of magnitude. Further, our Monte Carlo approach handles the class of nested summations in full generality, making it easy to specialize to new problems. Lastly, the computational complexity of our method is not directly dependent on dataset size, which means it can address high degrees of nesting that would make the standard multi-tree approach intractable. The main tradeoff is that Monte Carlo error bounds are probabilistic, though the bound probability is a parameter to the algorithm. Thus, we believe the Monte Carlo approach is superior for all situations that can tolerate minor stochasticity in the approximated output.\n\n3\n\nSingle-stage Monte Carlo\n\nWe first derive a Monte Carlo approximation for the base case of a single-stage, flat summation, i.e. Equation 1. The basic results for this simple case (up to and including Algorithm 1 and Theorem 1) mirror the standard development of Monte Carlo as in [6] or [7], with some modification to accommodate our particular problem setup. We then move beyond to present novel sample complexity bounds and extend the single-stage results to the multi-stage and multi-stage stratified cases. These extensions allow us to efficiently bring Monte Carlo principles to bear on the entire class of generalized summations, while yielding insights into the dependence of computational complexity on sample statistics and how tree-based methods can improve those statistics. To begin, note that the summation B (Xc ) can be written as nE [fi ] = nf , where n = |I | and the 1 expectation is taken over a discrete distribution Pf that puts mass n on each term fi = f (Xc , Xi ). ^ Our goal is to produce an estimate B that has low relative error with high probability. More precisely, ^ - B | |B| with probability at least 1 - . This is equivalent for a specified and , we want |B to estimating f by f such that |f - f | |f |. Let f be the sample mean of m samples taken ^ ^ ^ from Pf . From the Central Limit Theorem, we have asymptotically f ^ N (f , f /m), where f ^2 ^2 is the sample variance, from which we can construct the standard confidence interval: | f - f | ^ z/2 f / m with probability 1 - . When f satisfies this bound, our relative error condition is ^ ^ implied by z/2 f / m |f |, and we also have |f | |f | - z/2 f / m. Combining these, ^ ^ ^ ^ we can ensure our target relative error by requiring that z/2 f / m (|^f | - z/2 f / m), ^ which rearranges to:\n2 m z/2\n\nEquation 3 gives an empirically testable condition that guarantees the target relative error level with probability 1 - , given that f has reached its asymptotic distribution N (f , f /m). This ^ ^2 suggests an iterative sampling procedure in which m starts at a value mmin chosen to make the normal approximation valid, and then is increased until the condition of Equation 3 is met. This procedure is summarized in Algorithm 1, and we state its error guarantee as a theorem. Theorem 1. Given mmin large enough to put f in its asymptotic normal regime, with probability ^ at least 1 - Algorithm 1 approximates the summation S with relative error no greater than . Proof. We have already established that Equation 3 is a sufficient condition for relative error with probability 1 - . Algorithm 1 simply increases the sample size until this condition is met. Sample Complexity. Because we are interested in fast approximations, Algorithm 1 is only useful if it terminates with m significantly smaller than the number of terms in the full summation. Equation 3 3\n\n^2 (1 + )2 f . 2 2 ^f\n\n(3)\n\n\f\nAlgorithm 1 Iterative Monte Carlo approximation for flat summations.\nMC-Approx(S, Xc , , , mmin ) samples , mneeded mmin repeat addS amples(samples, mneeded , S, Xc ) m, f , f calcS tats(samples) ^ ^2 2 mthresh z/2 (1 + )2 f / 2 2 ^2 ^f mneeded mthresh - m until m mthresh return |S.I |f ^ addSamples(samples, mneeded , S, Xc ) for i = 1 to mneeded Xi rand(S.I ) samples samples S.f (Xc , Xi ) end for calcStats(samples) m count(samples) f av g (samples) ^ f v ar(samples) ^2 return m, f , f ^ ^2\n\nProof. The termination condition is driven by f /2 , so we proceed by bounding this ratio. ^ 2 ^f First, with probability 1 we have a lower bound on the absolute value of the sample mean: - |f | |f | - z/2 f / m. Next, because the sample variance is asymptotically distributed as ^ ^ 2 4 N (f , (4f - f )/m), where 4f is the fourth central moment, we can apply the delta method 2 4 to infer that f converges in distribution to N (f , (4f - f )/4f m). Using the normal-based ^ confidence interval, his gives the following 1 - upper bound for the sample standard deviation: t 4 f f + z/2 ^ 4f - f /(2f m). We now combine these bounds, but since we only know that each bound individually covers at least a 1 - fraction of outcomes, we can only guarantee they will jointly hold with probability at least 1 - 2, giving the following 1 - 2 bound: 4f - 4 f + z/2 2f m f f ^ . f |f | ^ |f | - z/2 m Combining this with Equation 3 and solving for m shows that, with probability at least 1 - 2, the algorithm will terminate with m no larger than:\n2 z/2 (1 + 2 )2 f 2 2 |f |\n\ngives an empirical test indicating when m is large enough for sampling to terminate; we now provide an upper bound, in terms of the distributional properties of the full set of fi , for the value of m at which Equation 3 will be satisfied. Theorem 2. Given mmin large enough to put f and f in their asymptotic normal regimes, with ^ ^ . 2 probability at least 1 - 2 Algorithm 1 terminates with m O f + |f | 44f - 1 2 f\nf f\n\n\"\n\n(1 + ) f + |f | (1 + 2 )2\n\nr\n\n4f -1+ 4 f\n\nr\n\nf |f |\n\ns\n\n2 (1 + ) f + |f | (1 + 2 )2\n\nr\n\n4f -1 4 f\n\n#\n\n.\n\n(4)\n\nThree aspects of this bound are salient. First, computation time is liberated from dataset size. This 2 is because the sample complexity depends only on the distributional features ( f , f , and 4f ) of the summation terms, and not on the number of terms. For i.i.d. datasets in particular, these distributional features are convergent, which means the sample or computational complexity converges to a constant while speedup becomes unbounded as the dataset size goes to infinity.\n4 Second, the bound has sensible dependence on f /|f | and 4f /f . The former is a standard dispersion measure known as the coefficient of variation, and the latter is the kurtosis. Algorithm 1 therefore gives greatest speedup for summations whose terms have low dispersion and low kurtosis. The intuition is that sampling is most efficient when values are concentrated tightly in a few clusters, making it easy to get a representative sample set. This motivates the additional speedup we later gain by stratifying the dataset into low-variance regions.\n\nFinally, the sample complexity bound indicates whether Algorithm 1 will actually give speedup for any particular problem. For a given summation, let the speedup be defined as the total number of terms n divided by the number of terms evaluated by the approximation. For a desired speedup , we need n mbound , where mbound is the expression in Equation 4. This is the fundamental characterization of whether speedup will be attained. 4\n\n\f\nAlgorithm 2 Iterative Monte Carlo approximation for nested summations.\nMC-Approx: as in Algorithm 1 calcStats: as in Algorithm 1 addSamples(samples, mneeded , S, Xc ) for i = 1 to mneeded Xi rand(S.I (Xc )) mcArg s map(MC-Approx(, Xc Xi , . . .), S.Gj ) samples samples S.f (Xc , mcArg s) end for\n\n4\n\nMulti-stage Monte Carlo\n\nWe now turn to the inductive case of nested summations, i.e. Equation 2. The approach we take is to apply the single-stage Monte Carlo algorithm over the terms fi as before, but with recursive invocation to obtain approximations for the arguments Gj . Algorithm 2 specifies this procedure. Theorem 3. Given mmin large enough to put f in its asymptotic normal regime, with probability ^ at least 1 - Algorithm 2 approximates the summation S with relative error no greater than . Proof. We begin by noting that the proof of correctness for Aligorithm 1 rests on 1) the ability to 1 fi , and 2) the ability to invoke the sample from a distribution Pf whose expectation is f = n 2 CLT on the sample mean f in terms of the sample variance f . Given these properties, Equation 3 ^ ^ follows as a sufficient condition for relative error no greater than with probability at least 1 - . We therefore need only establish that Algorithm 2 samples from a distribution having these properties. For each sampled fi , let Gj be the recursive approximation for argument Gj . We assume Gj has been drawn from a CLT-type normal distribution. Because the Gj are recursively approximated, this is an inductive hypothesis, with the remainder of the proof showing that if the hypothesis holds for the recursive invocations, it also holds for the outer invocation. The base case, where all recursions must bottom out, is the type-B summation already shown to give CLT-governed answers (see proof of Theorem 1). Let Gm = (G1 , G2 , . . .) be the vector of Gj values after each Gj has been estimated m G from mj samples ( j = m), and let G be the vector of true Gj values. Since each component j 2 Gm satisfies Gm N (G , m ). We leave the detailed converges in distribution to N (Gj , /mj ),\nj\n\n2 entries of the covariance m unspecified, except to note that its j j th element is j /mj , and that its off-diagonal elements may be non-zero if the Gj are generated in a correlated way (this can be used as a variance reduction technique).\n\nThis being the case, uniform sampling of the recursively estimated fi is equivalent to sampling from 1 ~ a distribution Pf that gives weight n to a normal distribution centered on each fi . The expectation ~f is f , and since the algorithm uses a simple sample mean the CLT does apply. These are over P the properties we need for correctness, and the applicability of the CLT combined with the proven base case completes the inductive proof.\n\nGiven the asymptotic normality of Gm , the same arguments used to derive the multivariate delta method can be used, with some modification, to show that fi (Gm ) N (fi (G ), f (G )m T (G )). f Gm ) is normally distributed around its true value with a variance that deThus, asymptotically, fi ( pends on both the gradient of f and the covariance matrix of the approximated arguments in Gm .\n\ni 1 2 2 2 ~ Note that the variance over Pf works out to f = f + n ~2 f I i , where i = f (G )m T (G ). 2 In other words, the variance with recursive approximation is the exact variance f plus the average 2 of the variances i of the approximated fi . Likewise one could write an expression for the kurtosis 4f . Because we are still dealing with a sample mean, Theorem 2 still holds in the nested case. ~ Corollary 2.1. Given mmin large enough to put f and f in their asymptotic normal regimes, ^ ^ ~ . ~ 2 ~ with probability at least 1 - 2 Algorithm 2 terminates with m O f + |f | 44f - 1 2 ~f f f It is important to point out that the 1 - confidences and relative error bounds of the recursively approximated arguments do not pass through to or compound in the overall estimator f : their ^ 2 influence appears in the variance i of each sampled fi , which in turn contributes to the overall variance f , and the error from f is independently controlled by the outermost sampling procedure. ~2 ~2 5\n\n\f\nAlgorithm 3 Iterative Monte Carlo approximation for nested summations with stratification.\nMC-Approx: as in Algorithm 1 calcStats(strata, samples) m count(samples) f s stratAv g (strata, samples) ^ f s stratV ar(strata, samples) ^2 return m, f s , f s ^ ^2 addSamples(strata, samples, mneeded , S, Xc ) needP erS trat = optAlloc(samples, strata, mneeded ) for s = 1 to strata.count ms = needP erS trat[s] for i = 1 to ms Xi rand(S.I (Xc ), strata[s]) mcArg s map(MC-Approx(, Xc Xi , . . .), S.Gj ) samples[s] samples[s] S.f (Xc , mcArg s) end for end for\n\n5\n\nVariance Reduction\n\nWith Algorithm 2 we have coverage of the entire generalized summation problem class, and our focus turns to maximizing efficiency. As noted above, Theorem 2 implies we need fewer samples when the summation terms are tightly concentrated in a few clusters. We formalize this by spatially partitioning the data to enable a stratified sampling scheme. Additionally, by use of correlated sampling we induce covariance between recursively estimated summations whenever the overall variance can be reduced by doing so. Adding these techniques to recursive Monte Carlo makes for an extremely fast, accurate, and general approximation scheme. Stratification. Stratification is a standard Monte Carlo principle whereby the values being sampled are partitioned into subsets (strata) whose contributions are separately estimated and then combined. The idea is that strata with higher variance can be sampled more heavily than those with lower variance, thereby making more efficient use of samples than in uniform sampling. Application of this principle requires the development of an effective partitioning scheme for each new domain of interest. In the case of generalized summations, the values being sampled are the f i , which are not known a priori and cannot be directly stratified. However, since f is generally a function with some degree of continuity, its output is similar for similar values of its arguments. We therefore stratify the argument space, i.e. the input datasets, by use of spatial partitioning structures. Though any spatial partitioning could be used, in this work we use modified k d-trees that recursively split the data along the dimension of highest variance. The approximation procedure runs as it did before, except that the sampling and sample statistics are modified to make use of the trees. Trees are expanded up to a user-specified number of nodes, prioritized by a heuristic of expanding nodes in order of largest size times average per-dimensional standard deviation. This heuristic will later be justified by the variance expression for the stratified sample mean. The approximation procedure is summarized in Algorithm 3, and we now establish its error guarantee. Theorem 4. Given mmin large enough to put f in its asymptotic normal regime, with probability ^ at least 1 - Algorithm 3 approximates the summation S with relative error no greater than . Proof. Identical to Theorem 3, but we need to establish that 1) the sample mean remains unbiased under stratification, and 2) the CLT still holds under stratification. These turn out to be standard properties of the stratified sample mean and its variance estimator (see [7]): j f s = ^ pj j ^ (5) 2 (f s ) = ^^ f s ^2 ; m f s ^2 m j p2 j j p2 j ^2 j2 = , ^ mj qj j (6)\n\nwhere j indexes the strata, j and j are the sample mean and variance of stratum j , pj is the ^ ^2 fraction of summation terms in stratum j , and qj is the fraction of samples drawn from stratum j . Algorithm 3 modifies the addSamples subroutine to sample in stratified fashion, and computes the stratified f s and f s instead of f and f in calcStats. Since these estimators satisfy the two ^ ^2 ^ ^2 conditions necessary for the error guarantee, this establishes the theorem.\n2 The true variance 2 (f s ) is identical to Equation 6 but with the exact j substituted for j . In [7], ^ ^2 2 2 it is shown that f s f , i.e. stratification never increases variance, and that any refinement of a\n\n6\n\n\f\n2 stratification can only reduce f s . Although the sample allocation fractions qj can be chosen arbij 2 2 trarily, f s is minimized when qj pj j . With this optimal allocation, f s reduces to ( pj j ) 2 . This motivates our k d-tree expansion heuristic, as described above, which tries to first split the nodes with highest pj j , i.e. the nodes with highest contribution to the variance under optimal allocation. While we never know the j exactly, Algorithm 3 uses the sample estimates j at each stage to ^ approximate the optimal allocation (this is the optAlloc routine).\n\nFinally, the Theorem 2 sample complexity still holds for the CLT-governed stratified sample mean. Corollary 2.2. Given mmin large enough to put f s and f s in their asymptotic normal regimes, ^ ^ . 2 with probability at least 1 - 2 Algorithm 3 terminates with m O f2s + |ffs| 44f s - 1\nf fs\n\nCorrelated Sampling. The variance of recursively estimated fi , as expressed by f (G )m T (G ), f depends on the full covariance matrix of the estimated arguments. If the gradient of f is such that the variance of fi depends negatively (positively) on a covariance j k , we can reduce the variance by inducing positive (negative) covariance between Gj and Gk . Covariance can be induced by sharing sampled points across the estimates of Gj and Gk , assuming they both use the same datasets. In some cases the expression for fi 's variance is such that the effect of correlated sampling is datadependent; when this happens, it is easy to test and check whether correlation helps. All experiments presented here were benefited by correlated sampling on top of stratification.\n\n6\n\nExperiments\n\nWe present experimental results in two phases. First, we compare stratified multi-stage Monte Carlo approximations to exact evaluations on tractable datasets. We show that the error distributions conform closely to our asymptotic theory. Second, having verified accuracy to the extent possible, we run our method on datasets containing millions of points in order to show 1) validation of the theoretical prediction that runtime is roughly independent of dataset size, and 2) many orders of magnitude speedup (as high as 1014 ) relative to exact computation. These results are presented for two method-dataset pairs: kernel regression on a dataset containing 2 million 4-dimensional redshift measurements used for quasar identification, and kernel conditional density estimation on an n-body galaxy simulation dataset containing 3.5 million 3-dimensional locations. In the KR case, the fourth dimension is regressed against the other three, while in KCDE the distribution of the third dimension is predicted as a function of the first two. In both cases we are evaluating the cross-validated score functions used for bandwidth optimization, i.e. SK R and SK C DE as described in Section 2. Error Control. The objective of this first set of experiments is to validate the guarantee that relative error will be less than or equal to with probability 1 - . We measured the distribution of error on a series of random data subsets up to the highest size for which the exact computation was tractable. For the O(n2 ) SK R , the limit was n = 10K, while for the O(n3 ) SK C DE it was n = 250. For each dataset we randomly chose and evaluated 100 bandwidths with 1 - = 0.95 and = 0.1. Figure 1 shows the full quantile spreads of the relative errors. The most salient feature is the relationship of the 95% quantile line (dashed) to the threshold line at = 0.1 (solid). Full compliance with asymptotic theory would require the dashed line never to be above the solid. This is basically the case for KCDE,1 while the KR line never goes above 0.134. The approximation is therefore quite good, and could be improved if desired by increasing mmin or the number of strata, but in this case we chose to trade a slight increase in error for an increase in speed. Speedup. Given the validation of the error guarantees, we now turn to computational performance. As before, we ran on a series of random subsets of the data, this time with n ranging into the millions. At each value of n, we randomly chose and evaluated 100 bandwidths, measuring the time for each evaluation. Figure 2 presents the average evaluation time versus dataset size for both methods. The most striking feature of these graphs is their flatness as n increases by orders of magnitude. This is in accord with Theorem 2 and its corollaries, which predict sample and computational complexity independent of dataset size. Speedups2 for KR range from 1.8 thousand at n = 50K to 2.8 million at n = 2M. KCDE speedups range from 70 million at n = 50K to 1014 at n = 3.5M. This performance is many orders of magnitude better than that of previous methods.\n1 2\n\nThe spike in the max quantile is due to a single outlier point. All speedups are relative to extrapolated runtimes based on the O() order of the exact computation.\n\n7\n\n\f\n0.25\n\n0.2\n\nrelative error\n\n0.15\n\nrelative error\n\n99%-max 90%-99% 75%-90% 50%-75% 25%-50% 10%-25% 1%-10% min-1% 95% error = 0.1\n\n0.7\n\n0.6 0.5 0.4 0.3\n\n99%-max 90%-99% 75%-90% 50%-75% 25%-50% 10%-25% 1%-10% min-1% 95% error = 0.1\n\n0.1\n\n0.2 0.05 0.1\n\n0 1000\n\n2000\n\n3000\n\n4000\n\nFigure 1: Error distribution vs. dataset size for KR (left), and KCDE (right).\n4000 6000\n\n5000 6000 dataset size\n\n7000\n\n8000\n\n9000\n\n10000\n\n0 50\n\n100\n\n150 dataset size\n\n200\n\n250\n\n5000 avg. computation time (ms) avg. computation time (ms) 3000\n\n4000\n\n3000\n\n2000\n\n2000\n\n1000\n\n1000\n\n0\n\n0\n\n0\n\n500,000\n\nFigure 2: Runtime vs. dataset size for KR (left), and KCDE (right). Error bars are one standard deviation.\n\n1,000,000 dataset size\n\n1,500,000\n\n2,000,000\n\n-1000 0\n\n1,000,000\n\n2,000,000 dataset size\n\n3,000,000\n\n7\n\nConclusion\n\nWe have presented a multi-stage stratified Monte Carlo method for efficiently approximating a broad class of generalized nested summations. Summations of this type lead to computational bottlenecks in kernel estimators and elsewhere in machine learning. The theory derived for this Monte Carlo approach predicts: 1) relative error no greater than with probability at least 1 - , for user-specified and , and 2) sample and computational complexity independent of dataset size. Our experimental results validate these theoretical guarantees on real datasets, where we accelerate kernel crossvalidation scores by as much as 1014 on millions of points. This is many orders of magnitude faster than the previous state of the art. In addition to applications, future work will likely include automatic selection of stratification granularity, additional variance reduction techniques, and further generalization to other computational bottlenecks such as linear algebraic operations.\n\nReferences\n[1] Nicol N. Schraudolph and Thore Graepel. Combining conjugate direction methods with stochastic approximation of gradients. In Workshop on Artificial Intelligence and Statistics (AISTATS), 2003. [2] Alexander G. Gray and Andrew W. Moore. N-body problems in statistical learning. In Advances in Neural Information Processing Systems (NIPS) 13, 2000. [3] Mike Klaas, Mark Briers, Nando de Freitas, and Arnaud Doucet. Fast particle smoothing: If I had a million particles. In International Conference on Machine Learning (ICML), 2006. [4] Ping Wang, Dongryeol Lee, Alexander Gray, and James M. Rehg. Fast mean shift with accurate and stable convergence. In Workshop on Artificial Intelligence and Statistics (AISTATS), 2007. [5] Michael P. Holmes, Alexander G. Gray, and Charles Lee Isbell Jr. Fast nonparametric conditional density estimation. In Uncertainty in Artificial Intelligence (UAI), 2007. [6] Reuven Y. Rubinstein. Simulation and the Monte Carlo Method. John Wiley & Sons, 1981. [7] Paul Glasserman. Monte Carlo methods in financial engineering. Springer-Verlag, 2004.\n\n8\n\n\f\n", "award": [], "sourceid": 3166, "authors": [{"given_name": "Charles", "family_name": "Isbell", "institution": null}, {"given_name": "Michael", "family_name": "Holmes", "institution": null}, {"given_name": "Alexander", "family_name": "Gray", "institution": null}]}