{"title": "Newscast EM", "book": "Advances in Neural Information Processing Systems", "page_first": 713, "page_last": 720, "abstract": "", "full_text": "Newscast EM\n\nWojtek Kowalczyk\n\nDepartment of Computer Science\n\nVrije Universiteit Amsterdam\n\nThe Netherlands\n\nNikos Vlassis\n\nInformatics Institute\n\nUniversity of Amsterdam\n\nThe Netherlands\n\nwojtek@cs.vu.nl\n\nvlassis@science.uva.nl\n\nAbstract\n\nWe propose a gossip-based distributed algorithm for Gaussian mixture\nlearning, Newscast EM. The algorithm operates on network topologies\nwhere each node observes a local quantity and can communicate with\nother nodes in an arbitrary point-to-point fashion. The main difference\nbetween Newscast EM and the standard EM algorithm is that the M-step\nin our case is implemented in a decentralized manner: (random) pairs\nof nodes repeatedly exchange their local parameter estimates and com-\nbine them by (weighted) averaging. We provide theoretical evidence and\ndemonstrate experimentally that, under this protocol, nodes converge ex-\nponentially fast to the correct estimates in each M-step of the EM algo-\nrithm.\n\n1\n\nIntroduction\n\nAdvances in network technology, like peer-to-peer networks on the Internet or sensor net-\nworks, have highlighted the need for ef\ufb01cient ways to deal with large amounts of data that\nare distributed over a set of nodes. Examples are \ufb01nancial data reported on the Internet,\nweather data observed by a set of sensors, etc.\nIn particular, in many data mining ap-\nplications we are interested in learning a global model from such data, like a probability\ndistribution or a clustering of the data, without \ufb01rst transferring all the data to a central\nrepository. Ideally, we would like to have a fully decentralized algorithm that computes\nand disseminates aggregates of the data, with minimal processing and communication re-\nquirements and good fault-tolerant behavior.\n\nA recent development in distributed systems technology is the use of gossip-basedmodels\nof computation [1, 2, 3]. Roughly, in a gossip-based protocol each node repeatedly con-\ntacts some other node at random and the two nodes exchange information. Gossip-based\nprotocols are very simple to implement, while they enjoy strong performance guarantees\nas a result of randomization. Their use in data mining and machine learning applications is\ncurrently \ufb01nding inroads [4, 5].\n\nIn this paper we propose a gossip-based, fully decentralized implementation of the\nExpectation-Maximization (EM) algorithm for Gaussian mixture learning [6]. Our algo-\nrithm, which we call \u2018Newscast EM\u2019, assumes a set of data fxig that are drawn indepen-\ndently from a common Gaussian mixture and are distributed over the nodes of a network\n(one data point per node). Newscast EM utilizes a gossip-based protocol in its M-step to\n\n\flearn a global Gaussian mixture model p(x) from the data. The main idea is to perform the\nM-step in a number of cycles. Each node starts with a local estimate of the model param-\neters. Then, in every cycle, each node contacts some other node that is chosen at random\nfrom a list of known nodes, and the two nodes replace their local model estimates by their\n(weighted) averages. As we show below, under such a protocol the (erroneous) local mod-\nels of the individual nodes converge exponentially fast to the (correct) global model in each\nM-step of the algorithm.\n\nOur approach is fundamentally different from other distributed exact implementations of\nthe EM algorithm that resort on global broadcasting [7] or routing trees [8]. In the latter,\nfor instance, data suf\ufb01cient statistics are propagated through a spanning tree in the network,\ncombined with an incremental learning scheme as in [9]. A disadvantage of that approach\nis that only one node is carrying out computations at any time step, whereas in Newscast\nEM all nodes are running the same protocol in parallel. This results in a batch M-step that\nhas average runtime at most logarithmic in the number of nodes, as we will see next.\n\n2 Gaussian mixtures and the EM algorithm\n\nA k-component Gaussian mixture for a random vector x 2 IRd is de\ufb01ned as the convex\ncombination\n\np(x) =\n\nk\n\nX\n\ns=1\n\n(cid:25)sp(xjs)\n\n(1)\n\ns (x (cid:0) ms)=2],\nof k Gaussian densities p(xjs) = (2(cid:25))(cid:0)d=2jCsj(cid:0)1=2 exp[(cid:0)(x (cid:0) ms)>C(cid:0)1\neach parameterized by its mean ms and covariance matrix Cs. The components of the\nmixture are indexed by the random variable s that takes values from 1 to k, and (cid:25)s = p(s)\nde\ufb01nes a discrete prior distribution over the components. Given a set fx1; : : : ; xng of\nindependent and identically distributed samples from p(x), the learning task is to estimate\nthe parameter vector (cid:18) = f(cid:25)s; ms; Csgk\ns=1 of the k components that maximizes the log-\nlikelihood L = Pn\ni=1 log p(xi; (cid:18)): Throughout we assume that the likelihood function is\nbounded from above (e.g., by placing appropriate bounds on the components covariance\nmatrices).\nMaximization of the data log-likelihood L can be carried out by the EM algorithm [6]\nwhich can be seen as iteratively maximizing a lower bound of L [9]. This bound F is\na function of the current mixture parameters (cid:18) and a set of \u2018responsibility\u2019 distributions\nfqi(s)g, i = 1; : : : ; n, where each qi(s) corresponds to a data point xi and de\ufb01nes an\narbitrary discrete distribution over s. This lower bound is given by:\n\nF =\n\nn\n\nk\n\nX\n\ni=1\n\nX\n\ns=1\n\nqi(s)(cid:2) log p(xi; s; (cid:18)) (cid:0) log qi(s)(cid:3):\n\n(2)\n\nIn the E-step of the EM algorithm, the responsibility qi(s) for each point xi is set to the\nBayes posterior p(sjxi) given the parameters found in the previous step. In the M-step we\nsolve for the unknown parameters of the mixture by maximizing F for \ufb01xed qi(s). This\nyields the following updates:\n(cid:25)s = Pn\n\n(cid:0)msm>s : (3)\nNote that the main operation of the M-step is averaging: (cid:25)s is the average of qi(s), ms\nis the average of products qi(s)xi (divided by (cid:25)s), and the covariance matrix Cs is the\naverage of matrices qi(s)xix>i (divided by (cid:25)s and decreased by msm>s ). This observation\nis essential for the proposed algorithm, as we will shortly see.\n\nms = Pn\n\nCs = Pn\n\ni=1 qi(s)\n\n;\n\nn\n\ni=1 qi(s)xix>i\n\ni=1 qi(s)xi\n\n;\n\nn(cid:25)s\n\nn(cid:25)s\n\n\f3 Newscast computing and averaging\n\nThe proposed distributed EM algorithm for Gaussian mixture learning relies on the use of\nthe Newscast protocol for distributed computing [3]. Newscast is a gossip-based protocol\nthat applies to networks where arbitrary point-to-point communication between nodes is\npossible, and it involves repeated data exchange between nodes using randomization: with\nconstant frequency each node contacts some other node at random, and the two nodes\nexchange application-speci\ufb01c data as well as caches with addresses of other nodes. The\nprotocol is very robust, scalable, and simple to implement\u2014its Java implementation is\nonly a few kBytes of code and can run on small network-enabled computing devices such\nas mobile phones, PDAs, or sensors.\n\nAs with other gossip-based protocols [2], Newscast can be used for computing the mean\nof a set of values that are distributed over a network. Suppose that values v1; : : : ; vn are\nstored in the nodes of a network, one value per node. Moreover suppose that each node\nknows the addresses of all other nodes. To compute (cid:22) = 1\ni=1 vi, each node i initially\nsets (cid:22)i = vi as its local estimate of (cid:22), and then runs the following protocol for a number of\ncycles:\n\nn Pn\n\nUniform Newscast (for node i)\n\n1. Contact a node j = f (i) that is chosen uniformly at random from 1; : : : ; n.\n2. Nodes i and j update their estimates as follows: (cid:22)0i = (cid:22)0j = ((cid:22)i + (cid:22)j)=2.\n\nFor the purpose of analysis we will assume that in each cycle every node initiates a single\ncontact (but in practice the algorithm can be fully asynchronous). Note that the mean of\nthe local estimates f(cid:22)ig is always the correct mean (cid:22), while for their variance holds:\nLemma 1. IneachcycleofuniformNewscastthevarianceofthelocalestimatesdropson\ntheaveragebyfactor(cid:21),with(cid:21) (cid:20) 1\nProof. 1 Let (cid:8)t = Pn\ni=1((cid:22)i (cid:0) (cid:22))2 be the unnormalized variance of the local estimates (cid:22)i\nat cycle t. Suppose, without loss of generality, that within cycle t nodes initiate contacts in\nthe order 1; 2; : : : ; n. The new variance after node\u2019s 1 contact is:\n\n2pe.\n\n(cid:8)1 = (cid:8)t (cid:0) ((cid:22)1 (cid:0) (cid:22))2 (cid:0) ((cid:22)f (1) (cid:0) (cid:22))2 + 2(cid:16) (cid:22)1 + (cid:22)f (1)\n\n2\n\n(cid:0) (cid:22)(cid:17)2\n\n= (cid:8)t (cid:0)\n\n1\n2\n\n((cid:22)1 (cid:0) (cid:22))2 (cid:0)\n\n1\n2\n\n((cid:22)f (1) (cid:0) (cid:22))2 + ((cid:22)1 (cid:0) (cid:22))((cid:22)f (1) (cid:0) (cid:22)):\n\n(4)\n\n(5)\n\nTaking expectation over f, and using the fact that P [f (i) = j] = 1\n\nn for all i and j, gives:\n1\n2n(cid:17)(cid:30) (cid:0)\n\n((cid:22)1 (cid:0) (cid:22))2: (6)\n\n1\n2\n\n1\n\n2n(cid:17)n(cid:0)i\n\n(cid:30)\n2pe\n\n:\n\n((cid:22)i (cid:0) (cid:22))2:\n\n(7)\n\n(8)\n\nn\n\n1\n2n\n\n((cid:22)j (cid:0) (cid:22))2 = (cid:16)1 (cid:0)\n\n1\n2\n\nX\n\nj=1\n\n((cid:22)1 (cid:0) (cid:22))2 (cid:0)\n\nE[(cid:8)1j(cid:8)t = (cid:30)] = (cid:30) (cid:0)\nAfter n such exchanges, the variance (cid:8)t+1 is on the average:\n(cid:16)1 (cid:0)\n(cid:30) (cid:0)\n2n )n \ufb01nally gives:\n2(cid:16)1 (cid:0)\n\n2n(cid:17)n\n2n )n(cid:0)i by (1 (cid:0) 1\nE[(cid:8)t+1j(cid:8)t = (cid:30)] (cid:20)\n\nE[(cid:8)t+1j(cid:8)t = (cid:30)] = (cid:16)1 (cid:0)\n\nBounding the term (1 (cid:0) 1\n\n2n(cid:17)n\n\n(cid:30) (cid:20)\n\n1\n\n1\n2\n\nn\n\nX\n\ni=1\n\n1\n\n1\n\n1See [3] for an alternative proof of the same bound.\n\n\fThus after t cycles of uniform Newscast, the original variance (cid:30)0 of the local estimates is\nreduced on the average to (cid:30)t (cid:20) (cid:30)0=(2pe)t. The fact that the variance drops at an exponen-\ntial rate means that the nodes learn the correct average very fast. Indeed, using Chebyshev\u2019s\ninequality Pt[j(cid:22)i (cid:0) (cid:22)j (cid:21) \"] (cid:20) (cid:30)t=(n\"2) we see that for any \" > 0, the probability that\nsome node makes an estimation error larger than \" is dropping exponentially fast with the\nnumber of cycles t. In particular, we can derive a bound on the number of cycles that are\nneeded in order to guarantee with high probability that all nodes know the correct answer\nwith some speci\ufb01c accuracy:\nTheorem 1. Withprobability1(cid:0) (cid:14),afterd0:581(log n + 2 log (cid:27) + 2 log 1\n\" + log 1\nofuniformNewscastholdsmaxi j(cid:22)i (cid:0) (cid:22)j (cid:20) \",forany\" > 0 anddatavariance(cid:27)2.\nProof. Using Lemma 1 and the fact that (cid:30)0 = n(cid:27)2, we obtain E[(cid:8)t] (cid:20) n(cid:27)2=(2pe)t.\n\"2(cid:14) )= log(2pe) we obtain E[(cid:8)(cid:28) ] (cid:20) \"2(cid:14). Using Markov inequality, with\nSetting (cid:28) = log( n(cid:27)2\nprobability at least 1 (cid:0) (cid:14) holds (cid:8)(cid:28) (cid:20) \"2. Therefore, since (cid:8)(cid:28) is the sum of local terms, for\neach of them must hold j(cid:22)i (cid:0) (cid:22)j (cid:20) \". It is straightforward to show by induction over (cid:28) that\nthe same inequality will hold for any time (cid:28) 0 > (cid:28) .\n\n(cid:14) )e cycles\n\nFor example, for unit-variance data and a network with n = 104 nodes we need 49 cycles\nto guarantee with probability 95% that each node is within 10(cid:0)10 from the correct answer.\nNote that in uniform Newscast, each node in the network is assumed to know the addresses\nof all other nodes, and therefore can choose in each cycle one node uniformly at random\nto exchange data with. In practice, however, each node can only have a limited cache of\naddresses of other nodes. In this case, the averaging algorithm is modi\ufb01ed as follows:\n\nNon-uniform Newscast (for node i)\n\n1. Contact a node j = f (i) that is appropriately chosen from i\u2019s local cache.\n2. Nodes i and j update their estimates as follows: (cid:22)0i = (cid:22)0j = ((cid:22)i + (cid:22)j)=2.\n3. Nodes i and j update their caches appropriately.\n\nStep 3 implements a \u2018membership management\u2019 schedule which effectively de\ufb01nes a dy-\nnamically changing random graph topology over the network.\nIn our experiments we\nadopted the protocol of [10] which roughly operates as follows. Each entry k in node\u2019s i\ncache contains an \u2018age\u2019 attribute that indicates the number of cycles that have been elapsed\nsince node k created that entry. In step 1 above, node i contacts the node j with the largest\nage from i\u2019s cache, and increases by one the age of all other entries in i\u2019s cache. Then\nnode i exchanges estimates with node j as in step 2. In step 3, both nodes i and j select a\nrandom subset of their cache entries and mutually exchange them, \ufb01lling empty slots and\ndiscarding self-pointers and duplicates. Finally node i creates an entry with i\u2019s address in\nit and age zero, which is added in j\u2019s cache. The resulting protocol is particularly effective\nand, as we show in the experiments below, in some cases it even outperforms the uniform\nNewscast. We refer to [10] for more details.\n\n4 The Newscast EM algorithm\n\nNewscast EM (NEM) is a gossip-based distributed implementation of the EM algorithm for\nGaussian mixture learning, that applies to the following setting. We are given a set of data\nfxig that are distributed over the nodes of a network (one data point per node). The data\nare assumed independent samples from a common k-component Gaussian mixture p(x)\nwith (unknown) parameters (cid:18) = f(cid:25)s; ms; Csgk\ns=1. The task is to learn the parameters of\nthe mixture with maximum likelihood in a decentralizedmanner: that is, all learning steps\n\n\fshould be performed locally at the nodes, and they should involve as little communication\nas possible.\n\nThe NEM algorithm is a direct application of the averaging protocol of Section 3 for esti-\nmating the parameters (cid:18) of p(x) using the EM updates (3). The E-step of NEM is identical\nto the E-step of the standard EM algorithm, and it can be performed by all nodes in par-\nallel. The novel characteristic of NEM is the M-step which is implemented as a sequence\nof gossip-based cycles: At the beginning of each M-step, each node i starts with a local\nestimate (cid:18)i of the \u2018correct\u2019 parameter vector (cid:18) (correct according to EM and for the current\nEM iteration). Then, in every cycle, each node contacts some other node at random, and\nthe two nodes replace their local estimates (cid:18)i by their (weighted) averages. At the end of\nthe M-step each node has converged (within machine precision) to the correct parameter (cid:18).\nTo simplify notation, we will denote by (cid:18)i = f(cid:25)si; msi; ~Csig the local estimates of node i\nfor the parameters of component s at any point of the algorithm. The parameter ~Csi is\nde\ufb01ned such that Csi = ~Csi (cid:0) msim>si. The complete algorithm, which runs identically\nand in parallel for each node, is as follows:\n\nNewscast EM (for node i)\n\n1. Initialization. Set qi(s) to some random positive value and then normalize all qi(s) to\nsum to 1 over all s.\n2. M-step. Initialize i\u2019s local estimates for each component s as follows: (cid:25)si = qi(s),\nmsi = xi, ~Csi = xix>i . Then repeat for (cid:28) cycles:\na. Contact a node j = f (i) from i\u2019s local cache.\nb. Nodes i and j update their local estimates for each component s as follows:\n\n(cid:25)0si = (cid:25)0sj =\n\nm0si = m0sj =\n\n~C0si = ~C0sj =\n\n(cid:25)si + (cid:25)sj\n\n;\n\n2\n\n(cid:25)simsi + (cid:25)sjmsj\n\n(cid:25)si + (cid:25)sj\n\n(cid:25)si ~Csi + (cid:25)sj ~Csj\n\n(cid:25)si + (cid:25)sj\n\n:\n\n;\n\n(9)\n\n(10)\n\n(11)\n\nc. Nodes i and j update their caches appropriately.\n\n3. E-step. Compute new responsibilities qi(s) = p(sjxi) for each component s using the\nM-step estimates (cid:25)si, msi, and Csi = ~Csi (cid:0) msim>si.\n4. Loop. Go to step 2, unless a stopping criterion is satis\ufb01ed that involves the parameter\nestimates themselves or the energy F.2\n\nA few observations about the algorithm are in order. First note that both the initialization\nof the algorithm (step 1) as well as the E-step are completely local to each node. Similarly,\na stopping criterion involving the parameter estimates can be implemented locally if each\nnode caches its estimates from the previous EM-iteration. The M-step involves a total of\nk[1 + d + d(d + 1)=2] averages, for each one of the k components and for dimensional-\nity d, which are computed with the Newscast protocol. Given that all nodes agree on the\nnumber (cid:28) of Newscast cycles in the M-step, and assuming that (cid:28) is large enough to guar-\nantee convergence to the correct parameter estimates, the complete NEM algorithm can be\nperformed identically and in parallel by all nodes in the network.\n\nIt is easy to see that at any cycle of an M-step, and for any component s, the weighted\n\n2Note that F is a sum of local terms, and thus it can also be computed using the same protocol.\n\n\faverages over all nodes of the local estimates are always the EM-correct estimates, i.e.,\n\nPn\ni=1 (cid:25)simsi\nPn\n\ni=1 (cid:25)si\n\n= ms\n\n(12)\n\nand similarly for the ~Csi. Moreover, note that the weighted averages of the msi in (10)\nand the ~Csi in (11), with weights given by (9), can be written as unweighted averages of\nthe corresponding products (cid:25)simsi and (cid:25)si ~Csi. In other words, each local estimate can be\nwritten as the ratio of two local estimates that converge to the correct values at the same\nexponential rate (as shown in the previous section). The above observations establish the\nfollowing:\nTheorem 2. IneveryM-stepofNewscastEM,eachnodeconvergesexponentiallyfastto\nthecorrectparameterestimatesforeachcomponentofthemixture.\n\nSimilarly, the number of cycles (cid:28) for each M-step can be chosen according to Theorem 1.\nHowever, note that in every M-step each node has to wait (cid:28) cycles before its local estimates\nhave converged, and only then can it use these estimates in a new next E-step. We describe\nhere a modi\ufb01cation of NEM that allows a node to run a local E-step before its M-step has\nconverged. This \u2018partial\u2019 NEM algorithm is based on the following \u2018self-correction\u2019 idea:\ninstead of waiting until the M-step converges, after a small number of cycles each node runs\na local E-step, adjusts its responsibilities, and propagates appropriate corrections through\nthe network.\n\nSuch a scheme additionally requires that each node caches its responsibilities from the\nprevious E-step, denoted by ~qi(s). The only modi\ufb01cation is in the initialization of the M-\nstep: instead of fully resetting the local estimates as in step 2 above, a node makes the\nfollowing corrections to its current estimates (cid:25)si; msi; ~Csi for each component s:\n\n(cid:25)0si = (cid:25)si + qi(s) (cid:0) ~qi(s);\nm0si = fmsi(cid:25)si + xi[qi(s) (cid:0) ~qi(s)]g=(cid:25)0si;\n~C0si = f ~Csi(cid:25)si + xix>i [qi(s) (cid:0) ~qi(s)]g=(cid:25)0si:\n\n(13)\n(14)\n(15)\n\nAfter these corrections, the Newscast averaging protocol is executed for a number of cy-\ncles (smaller than the number (cid:28) of the \u2018full\u2019 NEM). These corrections may increase the\nvariance of the local estimates, but in most cases the corresponding increase of variance is\nrelatively small. This results in speed-ups (often as large as 10-fold), however guaranteed\nconvergence is hard to establish.3\n\n5 Experiments\n\nTo get an insight into the behavior of the presented algorithms we ran several experiments\nusing a Newscast simulator.4 In Fig. 1 we demonstrate the the performance of uniform\nand non-uniform Newscast in typical averaging tasks involving zero-mean unit-variance\ndata. In Fig. 1(left) we plot the variance reduction rate (cid:21) (mean and one standard deviation\nfor 50 runs) as a function of the number of cycles, for averaging problems involving n =\n105 data. Note that in uniform Newscast the observed rate is always below the derived\n\nbound 1=(2pe) (cid:25) 0:303 from Lemma 1. Moreover note that in non-uniform Newscast the\n\nvariance drops faster than in uniform Newscast. This is due to the fact that the dynamic\ncache exchange scheme of [10] results in in-degree network distributions that are very\npeaked around the cache size.\nIn practice this means that on the average each node is\n\n3This would require, for instance, that individual nodes have estimates of the total variance over\n\nthe network, which is not obvious how it can be done.\n\n4Available from http://www.cs.vu.nl/\u02dcsteen/globesoul/sim.tgz\n\n\fUniform Newscast\nNon\u2212uniform (cache 20)\n\n0.305\n\n0.3\n\n0.295\n\n0.29\n\n0.285\n\ne\n\nt\n\na\nr\n \n\nn\no\n\ni\nt\nc\nu\nd\ne\nr\n \n\ne\nc\nn\na\ni\nr\na\nV\n\n0\n\n5\n\n10\n\n15\n25\nNumber of cycles\n\n20\n\n30\n\n35\n\n40\n\n42.5\n\n42\n\n41.5\n\n41\n\n40.5\n\n40\n\n39.5\n\ne\nc\nn\ne\ng\nr\ne\nv\nn\no\nc\n \nr\no\n\nl\n\nf\n \ns\ne\nc\ny\nC\n\n39\n103\n\nUniform Newscast\nNon\u2212uniform (cache 20)\n\n104\n\nNumber of nodes\n\n105\n\n106\n\nFigure 1: (Left) Variance reduction rate of uniform and non-uniform Newscast, in aver-\naging tasks involving n = 105 nodes. (Right) Number of cycles to achieve convergence\nwithin \" = 10(cid:0)10 for unit-variance datasets of various sizes.\n\nequally often contacted to by other nodes in each cycle of the protocol. We also observed\nthat the variance reduction rate is on the average unaffected by the network size, while\nlarger networks result in smaller deviations. For n = 8 (cid:3) 105, for instance, the standard\ndeviation is half the one plotted above.\n\nIn Fig. 1(right) we plot the number of cycles that are required to achieve model accuracy at\nall nodes within \" = 10(cid:0)10 as a function of the network size. Note that all observed quan-\ntities are below the derived bound of Theorem 1, while non-uniform Newscast performs\nslightly better than uniform Newscast.\n\nWe also ran experiments involving synthetic data drawn from Gaussian mixtures of differ-\nent number of data points, where we observed results essentially identical to those obtained\nby the standard (centralized) EM. We also performed some experiments with the \u2018partial\u2019\nNEM, where it turned out that in most cases we could obtain the same model accuracy with\na much smaller number of cycles (5\u201310 times than the \u2018full\u2019 NEM), but in some cases the\nalgorithm did not converge.\n\n6 Summary and extensions\n\nWe presented Newscast EM, a distributed gossip-based implementation of the EM algo-\nrithm for learning Gaussian mixture models. Newscast EM applies on networks where\neach one of a (large) number of nodes observes a local quantity, and can communicate with\nother nodes in a point-to-point fashion. The algorithm utilizes a gossip-based protocol in\nits M-step to learn a global Gaussian mixture model from the data: each node starts with\na local estimate of the parameters of the mixture and then, for a number of cycles till con-\nvergence, pairs of nodes repeatedly exchange their local parameter estimates and combine\nthem by (weighted) averaging. Newscast EM implements a batch M-step that has aver-\nage runtime logarithmic in the network size. We believe that gossip-based protocols like\nNewscast can be used in several other algorithms that learn models from distributed data.\n\nSeveral extensions of the algorithm are possible. Here we have assumed that each node\nin the network observes one data point. We can easily generalize this to situations where\neach node observes (and stores) a collection of points, like in [8]. On the other hand, if the\nlocally observed data are too many, one may consider storing only some suf\ufb01cient statis-\ntics of these data locally, and appropriately bound the energy F in each iteration to get a\nconvergent EM algorithm [11]. Another interesting extension is to replace the averaging\n\n\fstep 2 of uniform and non-uniform Newscast with weighted averaging (for some choice of\nweights), and study the variance reduction rate in this case. Another interesting problem is\nwhen the E-step cannot be performed locally at a node but it requires distributing some in-\nformation over the network. This could be the case, for instance, when each node observes\nonly a few elements of a vector-valued quantity while, for instance, all nodes together ob-\nserve the complete sample. We note that if the component models factorize, several useful\nquantities can be computed by averaging in the log domain. Finally, it would be interesting\nto investigate the applicability of the Newscast protocol in problems involving distributed\ninference/learning in more general graphical models [12].\n\nAcknowledgments\n\nWe want to thank Y. Sfakianakis for helping in the experiments, T. Pylak for making his\nNewscast simulator publicly available, and D. Barber, Z. Ghahramani, and J.J. Verbeek for\ntheir comments. N. Vlassis is supported by PROGRESS, the embedded systems research\nprogram of the Dutch organization for Scienti\ufb01c Research NWO, the Dutch Ministry of\nEconomic Affairs and the Technology Foundation STW, project AES 5414.\n\nReferences\n\n[1] R. Karp, C. Schindelhauer, S. Shenker, and B. V\u00a8ocking. Randomized rumour spreading. In\nProc.41thIEEESymp.onFoundationsofComputerScience, Redondo Beach, CA, November\n2000.\n\n[2] D. Kempe, A. Dobra, and J. Gehrke. Gossip-based computation of aggregate information. In\nProc.44thIEEESymp.onFoundationsofComputerScience, Cambridge, MA, October 2003.\n[3] M. Jelasity, W. Kowalczyk, and M. van Steen. Newscast computing. Technical report, Dept. of\n\nComputer Science, Vrije Universiteit Amsterdam, 2003. IR-CS-006.\n\n[4] C. C. Moallemi and B. Van Roy. Distributed optimization in adaptive networks. In S. Thrun,\nL. Saul, and B. Sch\u00a8olkopf, editors, Advances in Neural Information Processing Systems 16.\nMIT Press, Cambridge, MA, 2004.\n\n[5] D. Kempe and F. McSherry. A decentralized algorithm for spectral analysis. In Proc.36thACM\n\nSymp.onTheoryofComputing, Chicago, IL, June 2004.\n\n[6] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via\n\nthe EM algorithm. J.Roy.Statist.Soc.B, 39:1\u201338, 1977.\n\n[7] G. Forman and B. Zhang. Distributed data clustering can be ef\ufb01cient and exact. ACMSIGKDD\n\nExplorations, 2(2):34\u201338, 2000.\n\n[8] R. D. Nowak. Distributed EM algorithms for density estimation and clustering in sensor net-\n\nworks. IEEETrans.onSignalProcessing, 51(8):2245\u20132253, August 2003.\n\n[9] R. M. Neal and G. E. Hinton. A view of the EM algorithm that justi\ufb01es incremental, sparse, and\nother variants. In M. I. Jordan, editor, Learning in graphical models, pages 355\u2013368. Kluwer\nAcademic Publishers, 1998.\n\n[10] S. Voulgaris, D. Gavidia, and M. van Steen. Inexpensive membership management for unstruc-\n\ntured P2P overlays. JournalofNetworkandSystemsManagement, 2005. To appear.\n\n[11] J. R. J. Nunnink, J. J. Verbeek, and N. Vlassis. Accelerated greedy mixture learning. In Proc.\n\nBelgian-DutchConferenceonMachineLearning, Brussels, Belgium, January 2004.\n\n[12] M. A. Paskin and C. E. Guestrin. Robust probabilistic inference in distributed systems. In Proc.\n\n20thInt.Conf.onUncertaintyinArti\ufb01cialIntelligence, Banff, Canada, July 2004.\n\n\f", "award": [], "sourceid": 2650, "authors": [{"given_name": "Wojtek", "family_name": "Kowalczyk", "institution": null}, {"given_name": "Nikos", "family_name": "Vlassis", "institution": "Adobe Research"}]}