{"title": "Bayesian Nonparametric Models on Decomposable Graphs", "book": "Advances in Neural Information Processing Systems", "page_first": 225, "page_last": 233, "abstract": "Over recent years Dirichlet processes and the associated Chinese restaurant process (CRP) have found many applications in clustering while the Indian buffet process (IBP) is increasingly used to describe latent feature models. In the clustering case, we associate to each data point a latent allocation variable. These latent variables can share the same value and this induces a partition of the data set. The CRP is a prior distribution on such partitions. In latent feature models, we associate to each data point a potentially infinite number of binary latent variables indicating the possession of some features and the IBP is a prior distribution on the associated infinite binary matrix. These prior distributions are attractive because they ensure exchangeability (over samples). We propose here extensions of these models to decomposable graphs. These models have appealing properties and can be easily learned using Monte Carlo techniques.", "full_text": "Bayesian Nonparametric Models on Decomposable\n\nGraphs\n\nFranc\u00b8ois Caron\n\nINRIA Bordeaux Sud\u2013Ouest\n\nInstitut de Math\u00b4ematiques de Bordeaux\n\nUniversity of Bordeaux, France\nfrancois.caron@inria.fr\n\nArnaud Doucet\n\nDepartments of Computer Science & Statistics\n\nUniversity of British Columbia, Vancouver, Canada\n\nand The Institute of Statistical Mathematics\n\nTokyo, Japan\n\narnaud@cs.ubc.ca\n\nAbstract\n\nOver recent years Dirichlet processes and the associated Chinese restaurant pro-\ncess (CRP) have found many applications in clustering while the Indian buffet\nprocess (IBP) is increasingly used to describe latent feature models. These mod-\nels are attractive because they ensure exchangeability (over samples). We propose\nhere extensions of these models where the dependency between samples is given\nby a known decomposable graph. These models have appealing properties and\ncan be easily learned using Monte Carlo techniques.\n\n1 Motivation\n\nThe CRP and IBP have found numerous applications in machine learning over recent years [5,\n10]. We consider here the case where the data we are interested in are \u2018locally\u2019 dependent; these\ndependencies being represented by a known graph G where each data point/object is associated\nto a vertex. These local dependencies can correspond to any conceptual or real (e.g. space, time)\nmetric. For example, in the context of clustering, we might want to propose a prior distribution on\npartitions enforcing that data which are \u2018close\u2019 in the graph are more likely to be in the same cluster.\nSimilarly, in the context of latent feature models, we might be interested in a prior distribution on\nfeatures enforcing that data which are \u2018close\u2019 in the graph are more likely to possess similar features.\nThe \u2018standard\u2019 CRP and IBP correspond to the case where the graph G is complete; that is it is fully\nconnected. In this paper, we generalize the CRP and IBP to decomposable graphs. The resulting\ngeneralized versions of the CRP and IBP enjoy attractive properties. Each clique of the graph follows\nmarginally a CRP or an IBP process and explicit expressions for the joint prior distribution on the\ngraph is available. It makes it easy to learn those models using straightforward generalizations of\nMarkov chain Monte Carlo (MCMC) or Sequential Monte Carlo (SMC) algorithms proposed to\nperform inference for the CRP and IBP [5, 10, 14].\nThe rest of the paper is organized as follows. In Section 2, we review the popular Dirichlet multi-\nnomial allocation model and the Dirichlet Process (DP) partition distribution. We propose an exten-\nsion of these two models to decomposable graphical models. In Section 3 we discuss nonparametric\nlatent feature models, reviewing brie\ufb02y the construction in [5] and extending it to decomposable\ngraphs. We demonstrate these models in Section 4 on two applications: an alternative to the hierar-\nchical DP model [12] and a time-varying matrix factorization problem.\n\n2 Prior distributions for partitions on decomposable graphs\n\nAssume we have n observations. When performing clustering, we associate to each of this observa-\ntion an allocation variable zi \u2208 [K] = {1, . . . , K}. Let \u03a0n be the partition of [n] = {1, . . . , n} de-\n\ufb01ned by the equivalence relation i \u2194 j \u21d4 zi = zj. The resulting partition \u03a0n = {A1, . . . , An(\u03a0n)}\n\n1\n\n\fis an unordered collection of disjoint non-empty subsets Aj of [n], j = 1, . . . , n(\u03a0n), where\n\u222ajAj = [n] and n(\u03a0n) is the number of subsets for partition \u03a0n. We also denote by Pn be the\nset of all partitions of [n] and let nj, j = 1, . . . , n(\u03a0n), be the size of the subset Aj.\nEach allocation variable zi is associated to a vertex/site of an undirected graph G, which is assumed\nto be known. In the standard case where the graph G is complete, we \ufb01rst review brie\ufb02y here two\npopular prior distributions on z1:n, equivalently on \u03a0n. We then extend these models to undirected\ndecomposable graphs; see [2, 8] for an introduction to decomposable graphs. Finally we brie\ufb02y\ndiscuss the directed case. Note that the models proposed here are completely different from the\nhyper multinomial-Dirichlet in [2] and its recent DP extension [6].\n\n2.1 Dirichlet multinomial allocation model and DP partition distribution\n\nAssume for the time being that K is \ufb01nite. When the graph is complete, a popular choice for the\nallocation variables is to consider a Dirichlet multinomial allocation model [11]\n\n(1)\nwhere D is the standard Dirichlet distribution and \u03b8 > 0. Integrating out \u03c0, we obtain the following\nDirichlet multinomial prior distribution\n\n, . . . ,\n\n\u03c0 \u223c D( \u03b8\nK\n\n), zi|\u03c0 \u223c \u03c0\n\n\u03b8\nK\n\n(cid:81)K\n\nand then, using the straightforward equality Pr(\u03a0n) =\nPK where PK = {\u03a0n \u2208 Pn|n(\u03a0n) \u2264 K}, we obtain\n\u0393(\u03b8)\n\nK!\n\nPr(\u03a0n) =\n\n(K \u2212 n(\u03a0n))!\n\nPr(z1:n) =\n\n\u0393(\u03b8)\n\nK )\nj=1 \u0393(nj + \u03b8\nK )K\n\n\u0393(\u03b8 + n)\u0393( \u03b8\n\n(2)\n(K\u2212n(\u03a0n))! Pr(z1:n) valid for for all \u03a0n \u2208\n(cid:81)n(\u03a0n)\n\nK!\n\n.\n\n(3)\n\nj=1 \u0393(nj + \u03b8\nK )\nK )n(\u03a0n)\n\n\u0393(\u03b8 + n)\u0393( \u03b8\n\n\u03b8n(\u03a0n)(cid:81)n(\u03a0n)\n(cid:81)n\nj=1 \u0393(nj)\ni=1(\u03b8 + i \u2212 1)\n\nDP may be seen as a generalization of the Dirichlet multinomial model when the number of com-\nponents K \u2192 \u221e; see for example [10]. In this case the distribution over the partition \u03a0n of [n] is\ngiven by [11]\n\nPr(\u03a0n) =\n\n(4)\nLet \u03a0\u2212k = {A1,\u2212k, . . . , An(\u03a0\u2212k),\u2212k} be the partition induced by removing item k to \u03a0n and nj,\u2212k\nbe the size of cluster j for j = 1, . . . , n(\u03a0\u2212k).\nIt follows from (4) that an item k is assigned\nto an existing cluster j, j = 1, . . . , n(\u03a0\u2212k), with probability proportional to nj,\u2212k/ (n \u2212 1 + \u03b8)\nand forms a new cluster with probability \u03b8/ (n \u2212 1 + \u03b8). This property is the basis of the CRP.\nWe now extend the Dirichlet multinomial allocation and the DP partition distribution models to\ndecomposable graphs.\n\n.\n\n2.2 Markov combination of Dirichlet multinomial and DP partition distributions\nLet G be a decomposable undirected graph, C = {C1, . . . , Cp} a perfect ordering of the cliques\nand S = {S2, . . . , Cp} the associated separators.\nIt can be easily checked that if the marginal\ndistribution of zC for each clique C \u2208 C is de\ufb01ned by (2) then these distributions are consistent as\nthey yield the same distribution (2) over the separators. Therefore, the unique Markov distribution\nover G with Dirichlet multinomial distribution over the cliques is de\ufb01ned by [8]\n\n(5)\nwhere for each complete set B \u2286 G, we have Pr(zB) given by (2). It follows that we have for any\n\u03a0n \u2208 PK\n\nPr(z1:n) =\n\nPr(\u03a0n) =\n\nK!\n\n(K \u2212 n(\u03a0n))!\n\n(6)\n\n(cid:81)\n(cid:81)\n(cid:81)\n(cid:81)\n\n2\n\nC\u2208C Pr(zC)\nS\u2208S Pr(zS)\n(cid:81)K\n(cid:81)K\n\nC\u2208C\n\n\u0393(\u03b8)\n\n\u0393(\u03b8)\n\nS\u2208S\n\nj=1 \u0393(nj,C + \u03b8\n\nK )\n\n\u0393(\u03b8+nC )\u0393( \u03b8\n\nK )K\n\nj=1 \u0393(nj,S + \u03b8\n\nK )\n\n\u0393(\u03b8+nS )\u0393( \u03b8\n\nK )K\n\n\fwhere for each complete set B \u2286 G, nj,B is the number of items associated to cluster j, j =\n1, . . . , K in B and nB is the total number of items in B. Within each complete set B, the allocation\nvariables de\ufb01ne a partition distributed according to the Dirichlet-multinomial distribution.\nWe now extend this approach to DP partition distributions; that is we derive a joint distribution over\n\u03a0n such that the distribution of \u03a0B over each complete set B of the graph is given by (4) with\n\u03b8 > 0. Such a distribution satis\ufb01es the consistency condition over the separators as the restriction of\nany partition distributed according to (4) still follows (4) [7].\nProposition. Let PG\nn be the set of partitions \u03a0n \u2208 Pn such that for each decomposition A, B, and\nany (i, j) \u2208 A \u00d7 B, i \u2194 j \u21d2 \u2203k \u2208 A \u2229 B such that k \u2194 i \u2194 j. As K \u2192 \u221e, the prior distribution\nover partitions (6) is given for each \u03a0n \u2208 PG\n\nn by\n\nwhere n(\u03a0B) is the number of clusters in the complete set B.\nProof. From (6), we have\n\nC\u2208C\n\nS\u2208S\n\nPr(\u03a0n) = \u03b8n(\u03a0n)\n\n\u0393(nj,C )\ni=1(\u03b8+i\u22121)\nj=1 \u0393(nj,S )\ni=1(\u03b8+i\u22121)\n\n(cid:81)n(\u03a0C )\n(cid:81)nC\n(cid:81)n(\u03a0S )\n(cid:81)nS\n(cid:81)\n\u03b8n(\u03a0C )(cid:81)n(\u03a0C )\n(cid:81)nC\n(cid:81)\n(cid:80)\nC\u2208C n(\u03a0C )\u2212(cid:80)\n\u03b8n(\u03a0S )(cid:81)n(\u03a0S )\nPr(\u03a0n) = K(K \u2212 1) . . . (K \u2212 n(\u03a0n) + 1)\ni=1(\u03b8+i\u22121)\n(cid:81)nS\n(cid:80)\nC\u2208C n(\u03a0C) \u2212(cid:80)\nj=1 \u0393(nj,S + \u03b8\nC\u2208C n(\u03a0C) \u2212(cid:80)\ni=1(\u03b8+i\u22121)\n(cid:80)\nC\u2208C n(\u03a0C) \u2212(cid:80)\nS\u2208S n(\u03a0S) and 0 otherwise.\nS\u2208S n(\u03a0S) for any \u03a0n \u2208 Pn and the subset of Pn verifying\n\nS\u2208S n(\u03a0S) corresponds to the set PG\n\n\u0393(nj,C + \u03b8\n\nK )\n\nS\u2208S n(\u03a0S )\n\nC\u2208C\n\nS\u2208S\n\nn .(cid:165)\n\n(7)\n\nj=1\n\nK )\n\nThus when K \u2192 \u221e, we obtain (7) if n(\u03a0n) =\n\nK\n\nWe have n(\u03a0n) \u2264(cid:80)\n\nn(\u03a0n) =\n\n(cid:81)\n(cid:81)\n\nj=1\n\nExample. Let the notation i \u223c j (resp. i (cid:28) j) indicates an edge (resp. no edge) between two sites.\nLet n = 3 and G be the decomposable graph de\ufb01ned by the relations 1 \u223c 2, 2 \u223c 3 and 1 (cid:28) 3.\n3 is then equal to {{{1, 2, 3}};{{1, 2},{3}};{{1},{2, 3}};{{1},{2},{3}}}. Note that\nThe set PG\nthe partition {{1, 3},{2}} does not belong to PG\n3 . Indeed, as there is no edge between 1 and 3, they\ncannot be in the same cluster if 2 is in another cluster. The cliques are C1 = {1, 2} and C2 = {2, 3}\nand the separator is S2 = {2}. The distribution is given by Pr(\u03a03) = Pr(\u03a0C1 ) Pr(\u03a0C2 )\nhence we can\ncheck that we obtain Pr({1, 2, 3}) = (\u03b8 + 1)\u22122, Pr({1, 2},{3}) = Pr({1, 2},{3}) = \u03b8(\u03b8 + 1)\u22122\nand Pr({1},{2},{3}) = \u03b82(\u03b8 + 1)\u22122.(cid:165)\nLet now de\ufb01ne the full conditional distributions. Based on (7) the conditional assignment of an item\nk is proportional to the conditional over the cliques divided by the conditional over the separators.\nLet denote G\u2212k the undirected graph obtained by removing vertex k from G. Suppose that \u03a0n \u2208 PG\nn .\nIf \u03a0\u2212k /\u2208 PG\u2212k\nn\u22121, then do not change the value of item k. Otherwise, item k is assigned to cluster j\nwhere j = 1, . . . , n(\u03a0\u2212k) with probability proportional to\n\nPr(\u03a0S2 )\n\n(cid:81)\n(cid:81)\n\n{C\u2208C|n\u2212k,j,C >0} n\u2212k,j,C\n{S\u2208S|n\u2212k,j,S >0} n\u2212k,j,S\n\n(8)\n\nand to a new cluster with probability proportional to \u03b8, where n\u2212k,j,C is the number of items in the\nset C \\{k} belonging to cluster j. The updating process is illustrated by the Chinese wedding party\nprocess1 in Fig. 1. The results of this section can be extended to the Pitman-Yor process, and more\ngenerally to species sampling models.\nGiven \u03a0\u22122 = {A1 = {1}, A2 = {3}}, we have\nExample (continuing).\nPr(item 2 assigned to A1 = {1}| \u03a0\u22122) = Pr(item 2 assigned to A2 = {3}| \u03a0\u22122) = (\u03b8 + 2)\u22121\nand Pr(item 2 assigned to new cluster A3| \u03a0\u22122) = \u03b8 (\u03b8 + 2)\u22121. Given \u03a0\u22122 = {A1 = {1, 3}},\nitem 2 is assigned to A1 with probability 1.(cid:165)\n\n1Note that this representation describes the full conditionals while the CRP represents the sequential updat-\n\ning.\n\n3\n\n\f(a)\n\n(b)\n\n(c)\n\n(d)\n\n(e)\n\nFigure 1: Chinese wedding party. Consider a group of n guests attending a wedding party. Each\nof the n guests may belong to one or several cliques, i.e. maximal groups of people such that\neverybody knows everybody. The belonging of each guest to the different cliques is represented by\ncolor patches on the \ufb01gures, and the graphical representation of the relationship between the guests\nis represented by the graphical model (e). (a) Suppose that the guests are already seated such that\ntwo guests cannot be together at the same table is they are not part of the same clique, or if there\ndoes not exist a group of other guests such that they are related (\u201cAny friend of yours is a friend of\nmine\u201d). (b) The guest number k leaves his table and either (c) joins a table where there are guests\nfrom the same clique as him, with probability proportional to the product of the number of guests\nfrom each clique over the product of the number of guests belonging to several cliques on that table\nor (d) he joins a new table with probability proportional to \u03b8.\n\n2.3 Monte Carlo inference\n\n2.3.1 MCMC algorithm\n\n(cid:81)\n(cid:81)\n\nUsing the full conditionals, a single site Gibbs sampler can easily be designed to approximate the\nposterior distribution Pr(\u03a0n|z1:n). Given a partition \u03a0n, an item k is taken out of the partition. If\n\u03a0\u2212k /\u2208 PG\u2212k\nn\u22121, item k keeps the same value. Otherwise, the item will be assigned to a cluster j,\nj = 1, . . . , n(\u03a0\u2212k), with probability proportional to\n\n\u00d7\n\np(z{k}\u222aAj,\u2212k)\n\n{C\u2208C|n\u2212k,j,C >0} n\u2212k,j,C\n{S\u2208S|n\u2212k,j,S >0} n\u2212k,j,S\n\np(zAj,\u2212k)\n\n(9)\nand the item will be assigned to a new cluster with probability proportional to p(z{k})\u00d7 \u03b8. Similarly\nto [3], we can also de\ufb01ne a procedure to sample from p(\u03b8|n(\u03a0n) = k)). We assume that \u03b8 \u223c G(a, b)\nand use p auxiliary variables x1, . . . , xp. The procedure is as follows.\n\u2022 For j = 1, . . . , p, sample xj|k, \u03b8 \u223c Beta(\u03b8 + nSj , nCj \u2212 nSj )\n\n\u2022 Sample \u03b8|k, x1:p \u223c G(a + k, b \u2212(cid:80)\n\nj log xj)\n\n2.3.2 Sequential Monte Carlo\nWe have so far only treated the case of an undirected decomposable graph G. We can formu-\nlate a sequential updating rule for the corresponding perfect directed version D of G. Indeed, let\n(a1, . . . a|V |) be a perfect ordering and pa(ak) be the set of parents of ak which is by de\ufb01nition com-\nplete. Let \u03a0k\u22121 = {A1,k\u22121, . . . , An(\u03a0k\u22121),k\u22121} denote the partition of the \ufb01rst k\u22121 vertices a1:k\u22121\nand let nj,pa(ak) be the number of elements with value j in the set pa(ak), j = 1, . . . , n(\u03a0k\u22121).\nThen the vertex ak joins the set j with probability nj,pa(ak)/\nand creates a new\ncluster with probability \u03b8/\n\nq nq,pa(ak)\n\n(cid:80)\n\n(cid:179)\n\n(cid:180)\n\n(cid:180)\n\n(cid:179)\n\n\u03b8 +\n\nq nq,pa(ak)\n\n.\n\nOne can then design a particle \ufb01lter/SMC method in a similar fashion as [4]. Consider a set of\nN particles \u03a0(i)\nk\u22121 = 1) that approximate\nthe posterior distribution Pr(\u03a0k\u22121|z1:k\u22121). For each particle i, there are n(\u03a0(i)\nk\u22121) + 1 possible\n\nk\u22121 with weights w(i)\n\nk\u22121 \u221d Pr(\u03a0(i)\n\nk\u22121, z1:k\u22121) (\n\ni=1 w(i)\n\n\u03b8 +\n\n(cid:80)\n(cid:80)N\n\n4\n\n\fallocations for component ak. We denote(cid:101)\u03a0(i,j)\nto cluster j. The weight associated to(cid:101)\u03a0(i,j)\n\uf8f1\uf8f2\uf8f3\n\np(z{ak}\u222aAj,k\u22121)\n\nk\u22121 = w(i)\nk\u22121\n\n(cid:101)w(i,j)\nThen we can perform a deterministic resampling step by keeping the N particles(cid:101)\u03a0(i,j)\nweights (cid:101)w(i,j)\n\nif j = 1, . . . , n(\u03a0(i)\nif j = n(\u03a0(i)\nk\u22121) + 1\n\nk be the resampled particles and w(i)\n\nk\u22121 . Let \u03a0(i)\n\nk\nis given by\n\np(zAj,k\u22121)\n\n(cid:80)\n(cid:80)\n\nq nq,pa(ak)\n\nq nq,pa(ak)\n\nnj,pa(ak)\n\n\u00d7\n\n\u03b8+\n\n\u03b8+\n\nk\n\n\u03b8\n\nk\u22121)\n\nk the associated normalized weights.\n\nthe partition obtained by associating component ak\n\nk with highest\n\n(10)\n\n3 Prior distributions for in\ufb01nite binary matrices on decomposable graphs\nAssume we have n objects; each of these objects being associated to the vertex of a graph G. To\neach object is associated a K-dimensional binary vector zn = (zn,1, . . . , zn,K) \u2208 {0, 1}K where\nzn,i = 1 if object n possesses feature i and zn,i = 0 otherwise. These vectors zt form a binary\nn \u00d7 K matrix denoted Z1:n. We denote by \u03be1:n the associated equivalence class of left-ordered\nmatrices and let EK be the set of left-ordered matrices with at most K features.\nIn the standard case where the graph G is complete, we review brie\ufb02y here two popular prior distribu-\ntions on Z1:n, equivalently on \u03be1:n: the Beta-Bernoulli model and the IBP [5]. We then extend these\nmodels to undirected decomposable graphs. This can be used for example to de\ufb01ne a time-varying\nIBP as illustrated in Section 4.\n\n3.1 Beta-Bernoulli and IBP distributions\n\nThe Beta-Bernoulli distribution over the allocation Z1:n is\n\nK )\n\u0393(n + 1 + \u03b1\nwhere nj is the number of objects having feature j. It follows that\n\nj=1\n\nPr(Z1:n) =\n\n\u03b1\n\nK \u0393(nj + \u03b1\n\nK )\u0393(n \u2212 nj + 1)\n\nK(cid:89)\n\nK(cid:89)\n\nK!(cid:81)2n\u22121\n\nh=0 Kh!\n\nj=1\n\nPr(\u03be1:n) =\n\n\u03b1\n\nK \u0393(nj + \u03b1\n\nK )\u0393(n \u2212 nj + 1)\n\nK )\n\u0393(n + 1 + \u03b1\n\n(11)\n\n(12)\n\n(13)\n\nwhere Kh is the number of features possessing the history h (see [5] for details). The nonparametric\nmodel is obtained by taking the limit when K \u2192 \u221e\n\nPr(\u03be1:n) =\n\n\u03b1K+(cid:81)2n\u22121\n\nh=1 Kh!\n\nK+(cid:89)\n\n(n \u2212 nj)!(nj \u2212 1)!\n\nj=1\n\nn!\n\nexp(\u2212\u03b1Hn)\n\n(cid:80)n\n\nwhere K + is the total number of features and Hn =\n\n1\n\nk . The IBP follows from (13).\n\nk=1\n\n3.2 Markov combination of Beta-Bernoulli and IBP distributions\nLet G be a decomposable undirected graph, C = {C1, . . . , Cp} a perfect ordering of the cliques and\nS = {S2, . . . , Cp} the associated separators. As in the Dirichlet-multinomial case, it is easily seen\nthat if for each clique C \u2208 C, the marginal distribution is de\ufb01ned by (11), then these distributions\nare consistent as they yield the same distribution (11) over the separators. Therefore, the unique\nMarkov distribution over G with Beta-Bernoulli distribution over the cliques is de\ufb01ned by [8]\n\n(14)\nwhere Pr(ZB) given by (11) for each complete set B \u2286 G. The prior over \u03be1:n is thus given, for\n\u03be1:n \u2208 EK, by\n\nPr(Z1:n) =\n\nC\u2208C Pr(ZC)\nS\u2208S Pr(ZS)\n\n(cid:81)\n(cid:81)\n\nC\u2208C\n\nS\u2208S\n\nK!(cid:81)2n\u22121\n\nh=0 Kh!\n\nPr(\u03be1:n) =\n\n\u03b1\n\nK \u0393(nj,C + \u03b1\n\n\u03b1\n\nK \u0393(nj,S + \u03b1\n\n\u0393(nC +1+ \u03b1\n\nK )\u0393(nC\u2212nj,C +1)\nK )\u0393(nS\u2212nj,S +1)\n\nK )\n\n\u0393(nS +1+ \u03b1\n\nK )\n\n(15)\n\n(cid:81)\n(cid:81)\n(cid:81)K\n(cid:81)K\n\nj=1\n\nj=1\n\n5\n\n\fwhere for each complete set B \u2286 G, nj,B is the number of items having feature j, j = 1, . . . , K in\nthe set B and nB is the whole set of objects in set B. Taking the limit when K \u2192 \u221e, we obtain\nafter a few calculations\nPr(\u03be1:n) = \u03b1K+\n\nC HnC \u2212(cid:80)\n(cid:80)\n(cid:81)2n\u22121\n\n(nC\u2212nj,C )!(nj,C\u22121)!\n\n[n] exp [\u2212\u03b1 (\n\n(cid:81)K+\n(cid:81)K+\n\n(cid:81)\n(cid:81)\n\nS HnS )]\n\n(nS\u2212nj,S )!(nj,S\u22121)!\n\nC\u2208C\n\nC\nj=1\n\n\u00d7\n\nh=1 Kh!\n\nS\u2208S\n\nS\nj=1\n\nnC !\n\nnS !\n\n[n] =\n\nS K +\n\nC K +\n\nS and 0 otherwise, where K +\n\nif K +\npossessed by objects in B.\nLet EG\nn be the subset of En such that for each decomposition A, B and any (u, v) \u2208 A \u00d7 B: {u and\nv possess feature j} \u21d2 \u2203k \u2208 A \u2229 B such that {k possesses feature j}. Let \u03be\u2212k be the left-ordered\nmatrix obtained by removing object k from \u03ben and K +\u2212k be the total number of different features in\n\u03be\u2212k. For each feature j = 1, . . . , K +\u2212k, if \u03be\u2212k \u2208 EG\u2212k\n\nB is the number of different features\n\nn\u22121 then we have\n\n(cid:80)\n\nC \u2212(cid:80)\n\n\uf8f1\uf8f2\uf8f3 b\n\nb\n\n(cid:81)\n(cid:81)\n(cid:81)\nC\u2208C nj,C\n(cid:81)\nS\u2208C nj,S\nC\u2208C(nC\u2212nj,C )\nS\u2208C(nS\u2212nj,S )\n\nif i = 1\nif i = 0\n\nPr(\u03bek,j = i) =\n\n(cid:179)\n\n(cid:81)\n(cid:81)\n\n(16)\n\n(cid:180)\n\n{S\u2208S|k\u2208S} nS\nwhere b is the appropriate normalizing constant then the customer k tries Poisson\n{C\u2208C|k\u2208C} nC\nnew dishes. We can easily generalize this construction to a directed version D of G using arguments\nsimilar to those presented in Section 2; see Section 4 for an application to time-varying matrix\nfactorization.\n\n\u03b1\n\n4 Applications\n\n4.1 Sharing clusters among relative groups: An alternative to HDP\n\nConsider that we are given d groups with nj data yi,j in each group, i = 1, . . . , nj, j = 1, . . . , d. We\nconsider latent cluster variables zi,j that de\ufb01ne the partition of the data. We will use alternatively the\nnotation \u03b8i,j = Uzi,j in the following. Hierarchical Dirichlet Process [12] (HDP) is a very popular\nmodel for sharing clusters among related groups. It is based on a hierarchy of DPs\n\nG0 \u223c DP (\u03b3, H),\nGj|G0 \u223c DP (\u03b1, G0) j = 1, . . . d\n\u03b8i,j|Gj \u223c Gj, yi,j|\u03b8i,j \u223c f (\u03b8i,j) i = 1, . . . , nj.\n\nUnder conjugacy assumptions, G0, Gj and U can be integrated out and we can approximate the\nmarginal posterior of (zi,j) given y = (yi,j) with Gibbs sampling using the Chinese restaurant\nfranchise to sample from the full conditional p(zi,j|z\u2212{i,j}, y).\nUsing the graph formulation de\ufb01ned in Section 2, we propose an alternative to HDP. Let\n\u03b80,1, . . . , \u03b80,N be N auxiliary variables belonging to what we call group 0. We de\ufb01ne each clique Cj\n(j = 1, . . . , d) to be composed of elements from group j and elements from group 0. This de\ufb01nes a\ndecomposable graphical model whose separator is given by the elements of group 0. We can rewrite\nthe model in a way quite similar to HDP\n\nG0 \u223c DP (\u03b1, H),\n\u03b80,i|G0 \u223c G0\nGj|\u03b80,1, . . . , \u03b80,N \u223c DP (\u03b1 + N, \u03b1\n\u03b1+N H + \u03b1\n\u03b8i,j|Gj \u223c Gj, yi,j|\u03b8i,j \u223c f(\u03b8i,j) i = 1, . . . , nj\n\ni = 1, ..., N\n\n\u03b1+N\n\n(cid:80)N\n\ni=1 \u03b4\u03b80,i)\n\nj = 1, . . . d,\n\nFor any subset A and j (cid:54)= k \u2208 {1, . . . , p} we have corr(Gj(A), Gk(A)) = N\n\u03b1+N . Again, under\nconjugacy conditions, we can integrate out G0, Gj and U and approximate the marginal posterior\ndistribution over the partition using the Chinese wedding party process de\ufb01ned in Section 2. Note\nthat for latent variables zi,j, j = 1, . . . , d, associated to data, this is the usual CRP update. As in\nHDP, multiple layers can be added to the model. Figures 2 (a) and (b) resp. give the graphical DP\nalternative to HDP and 2-layer HDP.\n\n6\n\n\froot\n\nz0\n\ndocs\n\nz1\n\nz2\n\nz3\n\nroot\n\nz0\n\ncorpora\n\nz1\n\nz2\n\ndocs\n\nz1,1 z1,2 z2,1 z2,2\n\nz2,3\n\n(a) Graphical DP alter-\nnative to HDP\n\n(b) Graphical DP alternative to 2-layer\nHDP\n\nFigure 2: Hierarchical Graphs of dependency with (a) one layer and (b) two layers of hierarchy.\n\nIf N = 0, then Gj \u223c DP (\u03b1, H) for all j and this is equivalent to setting \u03b3 \u2192 \u221e in HDP. If N \u2192 \u221e\nthen Gj = G0 for all j, G0 \u223c DP (\u03b1, H). This is equivalent to setting \u03b1 \u2192 \u221e in the HDP. One\ninteresting feature of the model is that, contrary to HDP, the marginal distribution of Gj at any layer\nof the tree is DP (\u03b1, H). As a consequence, the total number of clusters scales logarithmically (as in\nthe usual DP) with the size of each group, whereas it scales doubly logarithmically in HDP. Contrary\nto HDP, there are at most N clusters shared between different groups. Our model is in that sense\nreminiscent of [9] where only a limited number of clusters can be shared. Note however that contrary\nto [9] we have a simple CRP-like process. The proposed methodology can be straightforwardly\nextended to the in\ufb01nite HMM [12].\nThe main issue of the proposed model is the setting of the number N of auxiliary parameters.\nAnother issue is that to achieve high correlation, we need a large number of auxiliary variables.\nNonetheless, the computational time used to sample from auxiliary variables is negligible compared\nto the time used for latent variables associated to data. Moreover, it can be easily parallelized. The\nmodel proposed offers a far richer framework and ensures that at each level of the tree, the marginal\ndistribution of the partition is given by a DP partition model.\n\nis a binary matrix whereas Y is a matrix of latent features. By assuming that Y \u223c N(cid:161)\n\n4.2 Time-varying matrix factorization\nLet X1:n be an observed matrix of dimension n\u00d7 D. We want to \ufb01nd a representation of this matrix\nin terms of two latent matrices Z1:n of dimension n \u00d7 K and Y of dimension K \u00d7 D. Here Z1:n\nY IK\u00d7D\nand\n\n0, \u03c32\n\n(cid:162)\n\nX1:n = Z1:nY + \u03c3X \u03b5n where \u03b5n \u223c N(cid:161)\n(cid:175)(cid:175)(cid:175)Z+T\n(cid:189)\n(cid:179)\n\n(cid:175)(cid:175)(cid:175)\u2212D/2\n(cid:180)\u22121\n\n1:n + \u03c32\n(n\u2212K+\nX\n\nY IK+\n\u03c3K+\nn D\n\nX /\u03c32\nn )D\n\n1:nZ+\n\nexp\n\n\u03c3\n\nY\n\nn\n\n(cid:162)\n\n(cid:161)\n\n0, \u03c32\n\nX In\u00d7D\n\n,\n\n\u2212 1\n2\u03c32\nX\n\ntr\n\n1:n\u03a3\u22121\nXT\n\nn X1:n\n\n(cid:162)(cid:190)\n\n(17)\n\nwe obtain\n\np(X1:n|Z1:n) \u221d\n\nn\n\n1:n\n\nY IK+\n\nX /\u03c32\n\n1:n + \u03c32\n\nZ+T\n1:n, K +\n\nn = I \u2212Z+\n\nZ+T\n1:n is the \ufb01rst K +\n\nwhere \u03a3\u22121\nn the number of non-zero columns of\n1:nZ+\nZ1:n and Z+\nn columns of Z1:n. To avoid having to set K, [5, 14] assume that Z1:n\nfollows an IBP. The resulting posterior distribution p(Z1:n|X1:n) can be estimated through MCMC\n[5] or SMC [14].\nWe consider here a different model where the object Xt is assumed to arrive at time index t and we\nwant a prior distribution on Z1:n ensuring that objects close in time are more likely to possess similar\nfeatures. To achieve this, we consider the simple directed graphical model D of Fig. 3 where the site\nnumbering corresponds to a time index in that case and a perfect numbering of D is (1, 2, . . .). The\nset of parents pa(t) is composed of the r preceding sites {{t \u2212 r}, . . . ,{t \u2212 1}}. The time-varying\nIBP to sample from p(Z1:n) associated to this directed graph follows from (16) and proceeds as\nfollows.\nAt time t = 1\n\u2022 Sample K new\nAt times t = 2, . . . , r\n\u2022 For k = 1, . . . K +\n\n1 \u223cPoisson(\u03b1), set z1,i = 1 for i = 1, ..., K new\n\nt , sample zt,k \u223c Ber( n1:t\u22121,k\n\n\u223cPoisson( \u03b1\nt ).\n\n1 = Knew.\n\n) and K new\n\nand set K +\n\n1\n\nt\n\nt\n\n7\n\n\f\u0017\u0014\n\u0016\u0015\n\n\u0017\u0014\n\u0016\u0015\n\nt\u2212r+1\n\nt\u2212r\n\n-\n\n-\n\n. . .\n\n-\n\n-\n\n\u0017\u0014\n\u0016\u0015\n\n\u0017\u0014\n\u0016\u0015\n\n\u0017\u0014\n\u0016\u0015\n\nt\u22121\n\n-\n\n?\n\n?\n\n-\n\nt+1\n\nt\n\n\u0017\n\n6\n\n\u0017\n\n6\n\nFigure 3: Directed graph.\n\nt\n\nr+1\n\nr+1).\n\n) and K new\n\n\u223cPoisson( \u03b1\n\nt , sample zt,k \u223c Ber( nt\u2212r:t\u22121,k\n\nAt times t = r + 1, . . . , n\n\u2022 For k = 1, . . . K +\nis the total number of features appearing from time max(1, t \u2212 r) to t \u2212 1 and nt\u2212r:t\u22121,k\nHere K +\nt\nthe restriction of n1:t\u22121 to the r last customers. Using (17) and the prior distribution of Z1:n which\ncan be sampled using the time-varying IBP described above, we can easily design an SMC method\nto sample from p(Z1:n|X1:n). We do not detail it here. Note that contrary to [14], our algorithm\ndoes not require inverting a matrix whose dimension grows linearly with the size of the data but only\na matrix of dimension r\u00d7 r. In order to illustrate the model and SMC algorithm, we create 200 6\u00d76\nimages using a ground truth Y consisting of 4 different 6 \u00d7 6 latent images. The 200 \u00d7 4 binary\n.5 0\nmatrix was generated from Pr(zt,k = 1) = \u03c0t,k, where \u03c0t = ( .6\n0 ) if t = 1, . . . , 30,\n\u03c0t = ( .4\n.6\n.6 ) if t = 51, . . . , 200. The\norder of the model is set to r = 50. The feature occurences Z1:n and true features Y and their\nestimates are represented in Figure 4. Two spurious features are detected by the model (features 2\nand 5 on Fig. 3(c)) but quickly discarded (Fig. 4(d)). The algorithm is able to correctly estimate the\nvarying prior occurences of the features over time.\n\n.4 0 ) if t = 31, . . . , 50 and \u03c0t = ( 0\n\n.8\n\n.3\n\n(a)\nFigure 4: (a) True features, (b) True features occurences, (c) MAP estimate ZM AP and (d) associated\nE[Y|ZM AP ]\n\n(b)\n\n(d)\n\n(c)\n\nFigure 5: (a) E[Xt|\u03c0t, Y] and (b) E[Xt|X1:t\u22121] at t = 20, 50, 100, 200.\n\n(a)\n\n(b)\n\n5 Related work and Discussion\nThe \ufb01xed-lag version of the time-varying DP of Caron et al. [1] is a special case of the proposed\nmodel when G is given by Fig. 3. The bivariate DP of Walker and Muliere [13] is also a special\ncase when G has only two cliques. In this paper, we have assumed that the structure of the graph\nwas known beforehand and we have shown that many \ufb02exible models arise from this framework. It\nwould be interesting in the future to investigate the case where the graphical structure is unknown\nand must be estimated from the data.\n\nAcknowledgment\n\nThe authors thank the reviewers for their comments that helped to improve the writing of the paper.\n\n8\n\nFeature1Feature2Feature3Feature4FeatureTime123420406080100120140160180200Feature1Feature2Feature3Feature4Feature5Feature6FeatureTime12345620406080100120140160180200t=20t=50t=100t=200t=20t=50t=100t=200\fReferences\n[1] F. Caron, M. Davy, and A. Doucet. Generalized Polya urn for time-varying Dirichlet process\n\nmixtures. In Uncertainty in Arti\ufb01cial Intelligence, 2007.\n\n[2] A.P. Dawid and S.L. Lauritzen. Hyper Markov laws in the statistical analysis of decomposable\n\ngraphical models. The Annals of Statistics, 21:1272\u20131317, 1993.\n\n[3] M.D. Escobar and M. West. Bayesian density estimation and inference using mixtures. Journal\n\nof the American Statistical Association, 90:577\u2013588, 1995.\n\n[4] P. Fearnhead. Particle \ufb01lters for mixture models with an unknown number of components.\n\nStatistics and Computing, 14:11\u201321, 2004.\n\n[5] T.L. Grif\ufb01ths and Z. Ghahramani. In\ufb01nite latent feature models and the Indian buffet process.\n\nIn Advances in Neural Information Processing Systems, 2006.\n\n[6] D. Heinz. Building hyper dirichlet processes for graphical models. Electonic Journal of Statis-\n\ntics, 3:290\u2013315, 2009.\n\n[7] J.F.C. Kingman. Random partitions in population genetics. Proceedings of the Royal Society\n\nof London, 361:1\u201320, 1978.\n\n[8] S.L. Lauritzen. Graphical Models. Oxford University Press, 1996.\n[9] P. M\u00a8uller, F. Quintana, and G. Rosner. A method for combining inference across related non-\n\nparametric Bayesian models. Journal of the Royal Statistical Society B, 66:735\u2013749, 2004.\n\n[10] R.M. Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of\n\nComputational and Graphical Statistics, 9:249\u2013265, 2000.\n\n[11] J. Pitman. Exchangeable and partially exchangeable random partitions. Probability theory and\n\nrelated \ufb01elds, 102:145\u2013158, 1995.\n\n[12] Y.W. Teh, M.I. Jordan, M.J. Beal, and D.M. Blei. Hierarchical Dirichlet processes. Journal of\n\nthe American Statistical Association, 101:1566\u20131581, 2006.\n\n[13] S. Walker and P. Muliere. A bivariate Dirichlet process. Statistics and Probability Letters,\n\n64:1\u20137, 2003.\n\n[14] F. Wood and T.L. Grif\ufb01ths. Particle \ufb01ltering for nonparametric Bayesian matrix factorization.\n\nIn Advances in Neural Information Processing Systems, 2007.\n\n9\n\n\f", "award": [], "sourceid": 562, "authors": [{"given_name": "Francois", "family_name": "Caron", "institution": null}, {"given_name": "Arnaud", "family_name": "Doucet", "institution": null}]}