{"title": "Structure Learning with Side Information: Sample Complexity", "book": "Advances in Neural Information Processing Systems", "page_first": 14380, "page_last": 14390, "abstract": "Graphical models encode the stochastic dependencies among random variables (RVs). The vertices represent the  RVs, and the edges signify the conditional dependencies among the RVs. Structure learning is the process of inferring the edges by observing realizations of the RVs, and it has applications in a wide range of technological, social, and biological networks. Learning the structure of graphs when the vertices are treated in isolation from inferential information known about them is well-investigated. In a wide range of domains, however, often there exist additional inferred knowledge about the structure, which can serve as valuable side information. For instance, the gene networks that represent different subtypes of the same cancer share similar edges across all subtypes and also have exclusive edges corresponding to each subtype, rendering partially similar graphical models for gene expression in different cancer subtypes. Hence, an inferential decision regarding a gene network can serve as side information for inferring other related gene networks.  When such side information is leveraged judiciously, it can translate to significant improvement in structure learning. Leveraging such side information can be abstracted as inferring structures of distinct graphical models that are {\\sl partially} similar. This paper focuses on Ising graphical models, and considers the problem of simultaneously learning the structures of two {\\sl partially} similar graphs, where any inference about the structure of one graph offers side information for the other graph. The bounded edge subclass of Ising models is considered, and necessary conditions (information-theoretic ), as well as sufficient conditions (algorithmic) for the sample complexity for achieving a bounded probability of error, are established. Furthermore, specific regimes are identified in which the necessary and sufficient conditions coincide, rendering the optimal sample complexity.", "full_text": "Structure Learning with Side Information:\n\nSample Complexity\n\nSaurabh Sihag\n\nAli Tajer\n\nElectrical, Computer, and Systems Engineering Department\n\nRensselaer Polytechnic Institute\n\nAbstract\n\nGraphical models encode the stochastic dependencies among random vari-\nables (RVs). The vertices represent the RVs, and the edges signify the conditional\ndependencies among the RVs. Structure learning is the process of inferring the\nedges by observing realizations of the RVs, and it has applications in a wide range\nof technological, social, and biological networks. Learning the structure of graphs\nwhen the vertices are treated in isolation from inferential information known about\nthem is well-investigated. In a wide range of domains, however, often there exist\nadditional inferred knowledge about the structure, which can serve as valuable side\ninformation. For instance, the gene networks that represent different subtypes of\nthe same cancer share similar edges across all subtypes and also have exclusive\nedges corresponding to each subtype, rendering partially similar graphical models\nfor gene expression in different cancer subtypes. Hence, an inferential decision\nregarding a gene network can serve as side information for inferring other related\ngene networks. When such side information is leveraged judiciously, it can translate\nto signi\ufb01cant improvement in structure learning. Leveraging such side informa-\ntion can be abstracted as inferring structures of distinct graphical models that are\npartially similar. This paper focuses on Ising graphical models, and considers the\nproblem of simultaneously learning the structures of two partially similar graphs,\nwhere any inference about the structure of one graph offers side information for\nthe other graph. The bounded edge subclass of Ising models is considered, and\nnecessary conditions (information-theoretic ), as well as suf\ufb01cient conditions (algo-\nrithmic) for the sample complexity for achieving a bounded probability of error,\nare established. Furthermore, speci\ufb01c regimes are identi\ufb01ed in which the necessary\nand suf\ufb01cient conditions coincide, rendering the optimal sample complexity.\n\n1\n\nIntroduction\n\nGraphical models are widely used to compactly model the conditional interdependence among\nmultiple random variables Lauritzen [1996] and Pearl [2009]. The vertices of the graph represent\nthe random variables (RVs), while the edges encode the inter-dependence among the RVs. The\ncomplete structure of the graph is analytically captured by the joint probability distribution of the\nrandom variables. Graphical models offer effective and tractable solutions to various inferential\nand decision-making solutions in different domains, e.g., computer vision Won and Derin [1992],\ngenetics Chen et al. [2013], Fang et al. [2016], Dobra et al. [2004], social networks Jacob et al.\n[2014], and power systems Dvijotham et al. [2017]. In this paper, we focus on Ising models and\nconsider the problem of joint model selection of a pair of graphs with partially identical structures\nusing the samples from their joint distributions.\nGraphical models with partially similar structures arise in the domains that consist of multiple layered\nnetworks of information sources. In such an application, each layer shares some of its vertices\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fand the data it generates with other layers that contain the same vertices. For example, the gene\nnetworks that represent the subtypes of the same cancer share similar edges across all subtypes\nand also have unique edges exclusive to each subtype of cancer Chen et al. [2013]. In a different\ncontext, a similar problem emerges in analyzing the voting patterns of the members of the US Senate\nGuo et al. [2015] for different categories of bills, where the statistical models reveal the common\ndependency structure across the members af\ufb01liated to the same political party and other structures\nunique to each class. Similarly, in the context of social networks, the relationships among a group\nof individuals on different platforms (e.g., Twitter and Facebook) form two distinct, but potentially\npartially similar graphical models. In such applications, learning one graph provides a signi\ufb01cant\namount of information that can be used for learning other related graphs.\nDue to the costs associated with collecting and processing data samples in large-scale graphical\nmodels, it is of interest to study the sample complexity of learning multiple structures simultaneously,\nwhere inference about each structure serves as side information for other structures.\n\n1.1 Related Work\n\nWhile the problem of graph structure learning is NP-hard Chickering [1996], it becomes feasible\nunder certain restrictions on the structure of the graph. For instance, the studies in Yuan and Lin\n[2007], Rothman et al. [2008], Ravikumar et al. [2010], Banerjee et al. [2008] investigate recovering\nthe structure of the graphical model under sparsity. Such conditions on the structures of the graphical\nmodels can be analyzed theoretically by considering certain restricted sub-classes of graphical models,\nfor e.g., graphs with a bounded degree or bounded number of edges.\nInformation-theoretic analysis of structure learning establishes the algorithm-independent dif\ufb01culty\nof recovering the structure of different classes of graphs. The studies in Santhanam and Wainwright\n[2012], Tandon et al. [2014], Scarlett and Cevher [2016] characterize the necessary conditions\non the sample complexity of selecting the model of a given graph in various sub-classes of Ising\nmodels. In Santhanam and Wainwright [2012], the necessary and suf\ufb01cient conditions on the sample\ncomplexity for the exact recovery of the graph are established for the class of Ising models under\nrestrictions on the maximum degree and the maximum number of edges in the graph. The results\nin Santhanam and Wainwright [2012] are extended to a set based graphical model selection in Vats and\nMoura [2011], where the graph estimator outputs a set of potentially true graphs instead of a unique\ngraph. Similarly, necessary conditions on the sample complexity are established for girth-bounded\ngraphs and path restricted graphs in Tandon et al. [2014]. In Scarlett and Cevher [2016], the problem\nof graphical model selection is studied for various sub-classes of Ising models under the criterion of\napproximate recovery. In Das et al. [2012], approximate recovery bounds on the sample complexity\nare characterized for Ising and Gaussian models without considering the effect of edge weights. The\ninformation-theoretic bounds on the sample complexity for structure recovery in Gaussian models\nare established in Wang et al. [2010], and the information-theoretic bounds for structure learning in\npower-law graph class are characterized in Tandon and Ravikumar [2013].\nAlgorithm-independent bounds on the sample complexity have also been investigated for other\ninference tasks besides model selection from the samples. In Gangrade et al. [2017], the problem of\ndetecting whether two Markov network structures are identical or different is studied, and sample\ncomplexity is characterized. The problem of property testing for Ising models is investigated\nin Neykov and Liu [2017], and information-theoretic limits for testing graph properties such as\nconnectivity, cycle presence, and maximum clique size are established. In Devroye et al. [2018],\nthe problem of density estimation using the samples from the Ising model, is investigated, and the\nminimax rate of estimation is analyzed.\nJoint inference of multiple graphical models, even though recognized as an inference problem that\narises in various domains, is primarily studied only algorithmically in Chen et al. [2013], Fang et al.\n[2016], Guo et al. [2011], Danaher et al. [2014], Mohan et al. [2014], Yang et al. [2015], Peterson\net al. [2015], Guo et al. [2015], Qiu et al. [2016]. In Chen et al. [2013], an empirical Bayes method,\nis deployed to identify interactions that are unique to each class and that are shared across all classes.\nIn Fang et al. [2016], Guo et al. [2011], Danaher et al. [2014], Mohan et al. [2014], Yang et al. [2015]\ngraphical Lasso-based algorithms are designed for joint inference of Gaussian graphical models. An\noptimization framework is used in Guo et al. [2015] for joint estimation of the graph structures based\non discrete data. Similarly, Peterson et al. [2015] investigates the problem of joint estimation of\n\n2\n\n\fFigure 1: Partially similar structures. Yellow nodes in both graphs have identical structures (p = 8, q = 3).\n\nGaussian graphical models using a Bayesian approach, where the data groups are used to identify\npartially similar models, and their similarity is leveraged.\n\n1.2 Contributions\n\nAll the studies above on joint graphical model inference propose empirical or algorithmic model-\nbased frameworks. In this paper, in sharp contrast, we provide an information-theoretic perspective\nfor jointly learning the structures of a pair of similar Ising models. Such analysis offers algorithm-\nindependent necessary conditions on the sample complexity for achieving any arbitrary level of\nreliability in the inference decision. In our previous work in Sihag and Tajer [2019] we considered\na sparsely connected, path-restricted sub-class in the context of Ising models and established the\nalgorithm-independent necessary conditions on the sample complexity. In this paper, we consider\na more general sub-class of edge bounded Ising models and provide the necessary conditions for\nall feasible values of the number of edges in the graphs. Furthermore, we also analyze a maximum\nlikelihood (ML)-based graph decoder to establish suf\ufb01cient conditions on the sample complexity.\nBased on these bounds, we also provide the asymptotic scaling behavior of these conditions in\ndifferent regimes. These analyses, as a by-product, also recover the existing relevant results on the\nrecovery of single graphs in Scarlett and Cevher [2016]. Finally, we provide numerical evaluations of\nML-based decoder to study the effect of structural similarity on its performance.\nA structurally similar pair of graphs are assumed to have identical connectivities in a subgraph formed\nby a known cluster of nodes. Such settings have been analyzed extensively in the context of seeded\ngraph matching and alignment problems Fishkind et al. [2019], Lyzinski et al. [2014], where in\ncontrast to this paper, the focus is on aligning the vertices of a partially aligned pair of graphs.\n\n2 Graph Model\nConsider two1 undirected graphs G1 (cid:44) (V, E1) and G2 (cid:44) (V, E2), such that the graphs are formed\nby the same set of vertices V (cid:44) {1, . . . , p} but have distinct sets of edges, denoted by E1 \u2286 V \u00d7 V\nand E2 \u2286 V \u00d7 V . When there exists an edge between nodes u, v \u2208 V in graph Gi, we denote it\nby (u, v) \u2208 Ei. Since the graphs are undirected, we have (u, v) = (v, u). We also de\ufb01ne the set\nNi(u) \u2208 V as the neighborhood of node u in graph Gi, i.e.,\n\nNi(u) (cid:44) {w \u2208 V : (u, w) \u2208 Ei} .\n\n(1)\nIt is assumed that a pre-speci\ufb01ed cluster of q nodes denoted by Vc \u2286 V have identical internal graph\nstructures in both G1 and G2. An example of two such graphical models is illustrated in Fig. 1.\nWe assume Ising models for both graphs G1 and G2, where we de\ufb01ne X u\ni \u2208 X (cid:44) {\u22121, 1} as the\nrandom variable associated with the node j \u2208 V in graph Gi, for i \u2208 {1, 2}. Accordingly, one sample\nfrom graph Gi is given by the random vector Xi (cid:44) [X 1\ni ]. The joint probability density\nfunction (pdf) of Xi associated with the graph Gi is given by\n\ni , . . . , X p\n\n\uf8eb\uf8ed (cid:88)\n\nu,v\u2208V\n\n\uf8f6\uf8f8 ,\n\nfi(Xi) =\n\n1\nZi\n\nexp\n\n\u03bbuv\ni X u\n\ni X v\ni\n\n(2)\n\n1The results in this paper can be generalized to settings with more than two graphs. For clarity, we analyze\n\nthe setting with two graphs.\n\n3\n\n\fwhere\n\nand Zi is the partition function, given by\n\n\u03bbuv\ni =\n\n(cid:26) \u03bb,\n(cid:88)\n\n0,\n\nif (u, v) \u2208 Ei\notherwise\n\n,\n\n\uf8eb\uf8ed (cid:88)\n\nu,v\u2208V\n\n\uf8f6\uf8f8 .\n\n(3)\n\n(4)\n\nZi =\n\nexp\n\n\u03bbuv\ni X u\n\ni X v\ni\n\nXi\u2208{\u22121,1}p\n\nThroughout the rest of the paper, we refer to Xi as one graph sample. The parameter \u03bb \u2208 R+ de\ufb01ned\nin (3) captures the interdependency among the random variables associated with the vertices. We\nremark that as \u03bb grows or diminishes, i.e., in the asymptote of large or small values of \u03bb, it becomes\nincreasingly dif\ufb01cult to distinguish the two distinct Ising models Santhanam and Wainwright [2012].\nFinally, corresponding to graph Gi, we also de\ufb01ne the maximum neighborhood weight according to\n(5)\n\n(cid:88)\n\n.\n\n\u03bbwu\ni\n\n\u03b6i (cid:44) max\nw\u2208V\n\nu\u2208Ni(w)\n\n3 Joint Structure Learning with Side Information\n\nIn this section, we formalize the notation of similar graphical models with partially identical structures,\nthe recovery criterion, and the associated performance measures.\n\n3.1 Graph Similarity Model\nDe\ufb01nition 1. Two graphs G1 and G2 with identical subgraphs with q nodes are said to be \u03b7\u2212similar,\nwhere \u03b7 = q\np .\nFor given G1 and G2, the edges between a pair of nodes with at least one node not in Vc are assumed\nto be structurally independent of each other. We denote the class of Ising models by I, and the class\nof \u03b7\u2212similar pairs of Ising models by I\u03b7. In this paper, we focus on an edge-bounded sub-class of\nIsing models de\ufb01ned next.\nDe\ufb01nition 2. This edge-bounded class of all the \u03b7\u2212similar pair of graphs G1 and G2 is speci\ufb01ed by\nparameters k \u2208 N and \u03b3 \u2208 (0, 1). The maximum number of edges in each graph is k and the number\nof edges in the identical subgraphs is \u03b3k.\n\nNote that in the de\ufb01nitions above the choices of \u03b3 and \u03b7 are not independent. Clearly, for any\n\ncombination of k and p, \u03b3 should satisfy \u03b3k \u2264(cid:0)q\n\n(cid:1).\n\n2\n\nFor convenience in notations, we also de\ufb01ne \u00afq (cid:44) p \u2212 q and \u00af\u03b3 (cid:44) 1 \u2212 \u03b3. It is also assumed that\nthe maximum neighborhood weight, de\ufb01ned in (5), is upper bounded by log \u03b6, i.e., \u03b6i \u2264 log \u03b6,\nfor i \u2208 {1, 2}. Finally, we remark that all the results provided for the edge-bounded class have\ncounterparts for the degree-bounded class as well, which due to space limitations are omitted.\n\n3.2 Recovery Criterion and Figure of Merit\nThe objective is to jointly estimate the structures of graphs G1 and G2 based on a collection of n\nindependent samples generated by each graph. The collection of n graph samples from the graph Gi\nis denoted by Xn\n\ni \u2208 X n\u00d7p. We de\ufb01ne the graph decoder\n\n\u03c8 : X n\u00d7p \u00d7 X n\u00d7p \u2192 I\u03b7 ,\n\n(6)\nas a function that maps the collection of samples to the graphs in class I\u03b7. We assume that in\neach recovered graph we can tolerate erroneous decisions about at most d number of edges, where\nd is pre-speci\ufb01ed. To capture the accuracy of such decisions, we de\ufb01ne P(I\u03b7, d) as the maximal\nprobability of error over the class I\u03b7, i.e.,\nP(I\u03b7, d) (cid:44) max\nG1,G2\u2208I\u03b7\n\n{|Ei\u2206 \u02c6Ei|} \u2265 d\n\nmax\ni\u2208{1,2}\n\n(cid:20)\n\n(cid:21)\n\n(7)\n\nP\n\n,\n\n4\n\n\fwhere |Ei\u2206 \u02c6Ei| is the edit distance between Ei and the estimated edge structure \u02c6Ei given by\n\n(8)\nTherefore, |Ei\u2206 \u02c6Ei| represents the number of edges to be inserted or deleted to transform Ei to \u02c6Ei.\nAlso, d represents the distortion level of the estimated graphs with respect to the true graphs.\n\n|Ei\u2206 \u02c6Ei| (cid:44) |(Ei\\ \u02c6Ei) \u222a ( \u02c6Ei\\Ei)| .\n\n4 Sample Complexity: Main Results\n\nIn this section, we provide the suf\ufb01cient and necessary conditions on the sample size n for any graph\ndecoder to recover a pair of graphs with bounded probabilities of error. The necessary conditions\nestablished are algorithm-independent and characterize the performance benchmarks on the sample\ncomplexity for any designed algorithm. The suf\ufb01cient conditions determine the feasibility of graph\nrecovery under the proposed recover algorithm (ML decoding) under given decision reliability\nconstraints.\nA summary of some of the main observations is provided in Table 1.\n\nTable 1: Summary of the main results for recovering Ising models of class I\u03b7.\n\nParameters\n\n\u03bb = O\n\nk\nk = O(p)\n\n\u03bb = O\n\n(cid:17)\n(cid:17)\n(cid:17)\n(cid:17)\n\nk\n\nk\n\n(cid:16) 1\u221a\n(cid:16) 1\u221a\n(cid:16) 1\u221a\n(cid:16) 1\n\np\n\nk = \u2126(p) and k = O(p\n\n4\n3 )\n\n\u03bb = O\n\nk = \u2126(p\n\n4\n3 ) and k = O(p2)\n\n\u03bb = O\n\nk \ufb01xed and k \u2264 p/4\n\nApprox. recovery (d > 0)\n\nApprox. recovery (d > 0)\n\n(Necessary conditions)\n\n(Suf\ufb01cient conditions)\n\nExact recovery (d = 0)\n(Necessary conditions)\n\n\u2126(k log p)\n\n\u2126(k2 log p)\n\n\u2126(k log p)\n\n\u2126(k)\n\n\u2126( p2\u221a\n\nk\n\n)\n\n\u2126(k2 log p)\n\n\u2126(k log p)\n\n\u2126(k2 log p)\n\n\u2126(k log p)\n\n\u2126(p2 log p)\n\n\u2126(p2 log p)\n\n\u2126(p2 log p)\n\n4.1 Suf\ufb01cient Conditions\n\nIn order to establish suf\ufb01cient conditions, we adopt the ML graph decoder de\ufb01ned as\n\n( \u02c6G1, \u02c6G2) (cid:44) arg max\n\n(G1,G2)\u2208I\u03b7\n\nfG1,G2 (Xn\n\n1 , Xn\n\n2 ) .\n\n(9)\n\nThe ML decoder is optimal under the exact recovery criterion, i.e., when d = 0 Santhanam and\nWainwright [2012]. Under approximate recovery, however, no error is declared if the estimates\nof the two graphs using the ML decoder lie within d distortion level of the true graphs. We use\nlarge deviations analysis of the probability of error of (9) under approximate recovery to analyze its\nperformance.\nTheorem 1 (Class I\u03b7). Consider a pair of \u03b7\u2212similar graphs G1 and G2 in class I\u03b7. If the sample\nsize n satis\ufb01es\n\nn \u2265 r max{A1, 2A2} ,\n\nwhere we have de\ufb01ned\n\n(cid:27)\n\n5\n\n,\n\nr (cid:44) 3\u03b6 2 + 1\nsinh2(\u03bb/4)\n(2k(cid:48) + \u03b3k) + log(2k(cid:48) \u2212 d) +2(k(cid:48) + 1) log p + log\n\n4\n\u03b4\n(2k(cid:48) + \u03b3k) + log(2\u03b3k \u2212 d) +2(\u03b3k + 1) log q + log\n\nA1 (cid:44)(cid:104)\nA2 (cid:44)(cid:104)\n\n(cid:26)\n\nk(cid:48) (cid:44) min\n\nk,\n\n\u00afq(\u00afq \u2212 1)\n\n2\n\n+ q \u00afq\n\n,\n\n(cid:21)\n\n2\n\u03b4\n\n,\n\n(cid:21)\n\n,\n\n(10)\n\n(11)\n\n(12)\n\n(13)\n\n(14)\n\nthen there exists a graph decoder \u03c8 : X n\u00d7p \u00d7 X n\u00d7p \u2192 I\u03b7 that achieves P(I\u03b7, d) \u2264 \u03b4.\n\n\fNote that k(cid:48) de\ufb01ned in (14) counts the maximum number of edges that can exist in the graphs after\nexcluding those in the shared identical subgraphs.\nIn order to gain more insight into the suf\ufb01cient condition in (10), we evaluate the scaling behavior of\nthe suf\ufb01cient conditions for n in terms of parameters \u03bb (parameter of Ising model in (3)), \u03b6 (controls\nmaximum neighborhood bound), and k (maximum number of edges in each graph). In all these\nregimes, it is assumed that k is increasing with the graph size p. Furthermore, it can be readily\nveri\ufb01ed that d, i.e., the number of errors tolerated by the decoder for the structure of each graph, does\nnot affect the asymptotic scaling behavior of the sample complexity.\n\n1. \u03bb = \u0398(1): When the size of identical subgraphs dominates the sizes of non-identical parts\n\nq (cid:28) 1 and \u03b3k (cid:29)(cid:0)\u00afq\n\n(cid:1) + \u00afqq, the sample complexity is dominated by 2rA2, which\n\nsuch that \u00afq\nscales according to \u2126(\u03b6 2k log p). Also, when we have k(cid:48) = k, the bound on the sample\ncomplexity scales according to \u2126(\u03b6 2k log p). Therefore, in this regime, under \ufb01xed \u03b4, the\nbound on sample complexity is always dominated by a term that has a scaling behavior\ngiven by \u2126(\u03b6 2k log p).\n\n2\n\n\u221a\n\n\u221a\n\n2. \u03bb = O(\n\nk\u22121): By noting that sinh(\u03bb/4) = \u2126(\u03bb), in this regime, Theorem 1 implies that\nthere exists a constant c > 0 such that when n > c \u00b7 \u03b6 2k2 log p, there always exists a graph\n\u221a\ndecoder that achieves P(I\u03b7, d) \u2264 \u03b4 . If \u03b6 = O(exp(\u03bb\nk)), then the bound on sample\ncomplexity scales as \u2126(k2 log p) for \ufb01xed \u03b4.\n\n3. \u03bb = \u0398(\n\nk):\n\nIn this regime, when both \u03bb and k are increasing with p, the bound\nk) and\nk = \u03c9(log(log p)), the bound on sample complexity scales exponentially according to\n\n\u221a\non the sample complexity scales as \u2126(\u03b6 2 log p). When we have \u03b6 \u2265 exp(\u03bb\n\u221a\n\u221a\n\u03bb\nexp(\u03bb\n\nk).\n\n4.2 Necessary Conditions\n\nFor describing the results in this subsection, we denote the binary entropy function by\n\nh(\u03b8) (cid:44) \u2212\u03b8 log \u03b8 \u2212 (1 \u2212 \u03b8) log(1 \u2212 \u03b8),\n\n(15)\nTheorem 2 (Class I\u03b7 with k \u2264 p/4). Consider a pair of \u03b7\u2212similar graphs G1 and G2 in the class\n2k . For any graph decoder \u03c8 : X n\u00d7p \u00d7 X n\u00d7p \u2192 I\u03b7 that achieves\nI\u03b7, such that, k \u2264 p/4 and \u03b3 \u2264 q\n(16)\n\nfor \u03b8 \u2208 (0, 1) .\n\nP(I k\n\n\u03b7 , d) \u2264 \u03b4 ,\n\nfor d = \u03b8k, for some \u03b8 \u2208 (0, 1\n\n4 ), the sample size n should satisfy\nn \u2265 max{B1, B2} (1 \u2212 \u03b4 \u2212 o(1)) ,\n\nwhere we have de\ufb01ned\n\nB1 (cid:44) 2(1 \u2212 \u03b3) log \u00afq + \u03b3 log q \u2212 2\u03b8 log p\nB2 (cid:44)\n\n\u03b32 exp(\u2212\u03bb((cid:112)\u03b3k) \u2212 1)/2) + \u00af\u03b32 exp(\u2212\u03bb(\n\n(1 \u2212 \u03b3/2) log 2 \u2212 h(\u03b8)\n\n\u03bb tanh \u03bb\n\n(cid:16)\n\n3\u03bbk\n\n,\n\n(17)\n\n(18)\n\n(19)\n\n(cid:17) .\n\n\u221a\n\n\u00af\u03b3k \u2212 1)/2)\n\nNext, we discuss the different scaling behavior of the necessary conditions on sample complexity\nfrom Theorem 2. Note that B1 and B2 have different scaling behavior in terms of \u03bb, k and p. In all\nthe following regimes, we assume that k is increasing with p.\n\n\u221a\n\n\u221a\n2. \u03bb = O(\n\n1. \u03bb = \u0398(1): In this regime, B1 scales as log p and B2 scales as e\nthe lower bound on the sample complexity if k = \u03c9(log(log p)).\n\nk. Clearly, B2 dominates\n\nscales as 1/k. Clearly, the sample complexity is dominated by B1.\n\nk\u22121): By noting that tanh(\u03bb) = O(\u03bb) we \ufb01nd that B1 scales as \u2126(k log p). B2\n\u221a\n\nk) : In this regime, B1 scales as O( log p\n\n/k1.5. If\nwe have k = \u03c9(log(log p)), B2 dominates the sample complexity and scales exponentially\nin k1.5.\n\n\u03bb tanh \u03bb ) and B2 scales as ek1.5\n\n3. \u03bb = \u0398(\n\n6\n\n\fTheorem 3 (Class I\u03b7 with k = \u2126(p)). Consider a pair of \u03b7\u2212 similar graphs G1 and G2 in the class\n4k }. For\nI\u03b7, such that, k = (cid:98)cp1+\u00b5(cid:99) for given constants c > 0 and \u00b5 \u2208 [0, 1), and \u03b3 \u2264 min{\u03b7, \u03b72p2\nany graph decoder \u03c8 : X n\u00d7p \u00d7 X n\u00d7p \u2192 I k\n\n\u03b7 that achieves\nP(I\u03b7, d) \u2264 \u03b4 ,\n\nfor d = \u03b8k where \u03b8 \u2208 (0, 1\n\n4 ), the sample size n should satisfy\n\nn \u2265 max{B3, B2} (1 \u2212 \u03b4 \u2212 o(1)) ,\n\nwhere we have de\ufb01ned B2 in (19), and\n\nB3 (cid:44) [(1 \u2212 \u03b3/2) log 2 \u2212 h(\u03b8)] \u00b7 \u03bb\u22121 exp(2\u03bb) cosh(4\u03bbcp\u00b5) + 1\nexp(2\u03bb) cosh(4\u03bbcp\u00b5) \u2212 1\n\n.\n\n(20)\n\n(21)\n\n(22)\n\nTo analyze the asymptotic scaling behavior of the necessary condition, we note that B2 depends on\n\u03bb and k, and B3 depends on \u03bb and p. Therefore, depending on the variations of \u03bb with respect to\np and k, we characterize the scaling behavior of the suf\ufb01cient condition in terms of k and p. In the\nfollowing regimes, we assume that k is increasing with p.\n\n1. \u03bb = \u0398(1):\n\n\u221a\nIn this regime, B2 scales as exp(\n\n\u221a\n2. \u03bb = O(\n\nk). On the other hand, we have\nexp(2\u03bb) cosh(4\u03bbcp\u00b5)\u22121\nexp(2\u03bb) cosh(4\u03bbcp\u00b5)+1 = \u0398(1) and therefore, B3 = \u0398(1) as p \u2192 \u221e. Clearly, B2 dom-\ninates the bound on sample complexity.\nk\u22121):\n\u221a\nIn this regime, B2 scales as 1/k. The analysis of B3 shows that\nexp(2\u03bb) cosh(4\u03bbcp\u00b5)\u22121\nexp(2\u03bb) cosh(4\u03bbcp\u00b5)+1 = O(max{1/\nk, k/p2}). Therefore, B3 scales according to\n\u221a\n\u2126(min{k, p2/\n\u221a\nk). Note that when we have k = \u2126(p) and k = O(p4/3), we have\nmin{k, p2/\nk} = k and therefore, the bound on sample complexity scales as \u2126(k). When\nk = \u2126(p4/3), the bound on the sample complexity scales as \u2126(p2/\n\u221a\n\nk): In this regime, B2 scales as ek1.5 and B3 \u2192 0 as k \u2192 \u221e. Therefore, the\n\nk) asymptotically.\n\n\u221a\n\n3. \u03bb = \u0398(\n\nlower bound on sample complexity scales exponentially in k1.5.\n\nThe analysis of the results in Theorem 1 and Theorem 2 reveals that the suf\ufb01cient and the necessary\nbounds on the sample complexity scale at the same rate (non-exponential) for the class I\u03b7 under a\nparticular regime, as described in Corollary 1.\nCorollary 1 (Optimal Sample Complexity). When the maximum number of edges is \ufb01xed and satis\ufb01es\nk \u2264 p/4, and we have\n\n(cid:26) q\n\n(cid:27)\n\n\u03b3 \u2264 min\n\n,\n\n\u03b72p2\n4k\n\n2k\n\nand\n\n\u03bb = O(1/p) ,\n\n(23)\n\nTheorem 1 indicates that when n > c2p2 log p, for a constant c2, there exists a graph decoder that\nrecovers both graphs with P(I\u03b7, d) \u2264 \u03b4. On the other hand, in this regime, Theorem 2 indicates that\nfor any graph decoder to achieve P(I\u03b7, d) \u2264 \u03b4 we should have n > c3p2 log p, for some constant\nc3 > 0. Therefore, in this regime, the graph decoder that satis\ufb01es Theorem 1 achieves the optimal\nsample complexity up to constant factors.\n\nFurthermore, we comment that the extreme case of \u03b7 = 0 corresponds to recovering two independent\ngraphs the other extreme case of \u03b7 = 1 corresponds to recovering two identical graphs. In both these\nextreme cases, the problem analyzed in this paper simpli\ufb01es to the problem of structure learning\nof one graph studied in Scarlett and Cevher [2016] (for approximate recovery) and in Santhanam\nand Wainwright [2012] (for exact recovery, i.e., d = 0). In general, however, when we depart from\nthese special cases, the analysis techniques in the context of single graphs in existing literature do not\nextend directly to the context of recovering a pair of graphs with structural similarity. Speci\ufb01cally,\nwe use novel ensemble constructions for a pair of graphs that accommodate structural similarity in\ndifferent regimes of k and analyze the pairwise KL divergences for the graph pairs to recover the\nnecessary conditions in Theorems 2 and 3. Moreover, our analysis of an ML decoder also recovers\nthe hitherto uninvestigated suf\ufb01cient conditions for the sample complexity under the approximate\nrecovery criterion. Hence, the results provided in this paper are completely different. This observation\nis formalized in the following corollary.\n\n7\n\n\fCorollary 2 (Special Cases). The necessary and suf\ufb01cient conditions on sample complexity in the\nextreme cases of \u03b7 = 0 and \u03b7 = 1 subsume the existing results for structure learning in single graphs.\n\nIn the context of the asymptotic scaling behaviors summarized in Table 1, we comment that the\nnecessary condition bounds for approximate recovery are not any looser with respect to that for exact\nrecovery than those in the context of single-graph recovery. Also, in some regimes the gap between\nthe necessary conditions and the suf\ufb01cient conditions on the sample complexity is tighter than others.\nFor instance, when k = O(p) the mismatch is only a factor k. The mismatch between the necessary\nconditions and suf\ufb01cient conditions is more profound in denser graphs, i.e., when k = \u2126(p).\nFurthermore, the analysis of the necessary conditions in Theorems 2 an 3 and suf\ufb01cient conditions in\nTheorem 1 reveals that d does not affect the asymptotic scaling rate of their respective bounds even\nwhen d scales as fast as linearly with k. For instance, in Theorem 1, d appears only in a logarithmic\nfactor scaling at most at the rate of log k which is dominated by k log p in A1 and A2. Therefore, the\nresults in Table 1 do not depend on d.\n\n5 Numerical Evaluations\nIn this section, we evaluate the tradeoffs between decision reliability captured by P(I\u03b7, d) de\ufb01ned\nin (7) and the necessary and suf\ufb01cient conditions on the sample complexities established. We\nevaluated these tradeoffs for different approximate recovery levels controlled by d as well as similarity\nlevels of the two graphs speci\ufb01ed by \u03b7. In general, the implementation of an ML decoder may become\ninfeasible as the size of the graphs grow. Therefore, to gain meaningful insights in the sample\ncomplexity with increasing size of graphs, the evaluations were performed on an ensemble of graphs\nthat contains graphs with many isolated edges. In this ensemble, we set the size of the graphs to\np, with q = (cid:98)\u03b7p(cid:99) nodes in the shared subgraph. We assumed that each graph contains \u03b1 isolated\nedges, with (cid:98)\u03b7\u03b1(cid:99) edges lying in the shared subgraph. Furthermore, the graphs in this ensemble were\nconstructed in the following manner. We grouped the non-shared cluster with size (p \u2212 q) vertices in\n(p \u2212 q)/2 \ufb01xed pairs and randomly connected the vertices in (\u03b1 \u2212 (cid:98)\u03b7\u03b1(cid:99)). Similarly, the q vertices\nof the shared subgraph were grouped into q/2 \ufb01xed pairs and (cid:98)\u03b7\u03b1(cid:99) pairs were selected randomly\nto be connected. For this ensemble, the implementation of ML decoder can be readily shown to be\nequivalent to a counting scheme that counts the number of agreements in the states of different nodes\nin the data. This allowed us to visualize the behavior of the sample complexity for ML decoder as the\nsize of the graphs was increased.\n\nFigure 2: Reliability (P(I\u03b7, d)) versus sample\ncomplexity (n) for different values of \u03b7 and d.\nSolid and dashed curves represent the suf\ufb01cient\nnumber of samples (based on ML decoder) and\nnecessary number of samples, respectively.\n\nFigure 3: Reliability (P(I\u03b7, d)) versus graph size\n(p) for different values of d, where d represents\nthe tolerance to distortion in the recovered graphs\nwith respect to the true graphs.\n\nWe \ufb01rst considered a graph with p = 100 vertices and \u03b1 = 20. Figure 2 depicts the variations of the\nerror probability P(I\u03b7, d) versus n. For each value of P(I\u03b7, d), the \ufb01gure speci\ufb01es the necessary\n(shown by dashed curves) and suf\ufb01cient conditions (shown by solid curves) on the number of samples\nn. The suf\ufb01cient conditions are obtained by simulations of the ML decoder. The \ufb01gure shows\n\n8\n\n050100150200250Number of samples00.20.40.60.81Probability of error = 0.3, d = 3 = 0.3, d = 1 = 0.1, d = 3 = 0.1, d = 1 = 0.3, d = 3 = 0.3, d = 1 = 0.1, d = 3 = 0.1, d = 1100200300400500Size of graphs10-210-1100Probability of errord = 0d = 2d = 4\fthese variations for different levels of graph similarity \u03b7 = 0.1, 0.3 and different values of recovery\napproximation d = 1, 3. The probability of error was evaluated empirically over 6000 trials.\nIn Corollary 1, we have provided a regime in which the scaling behaviors of the necessary and\nsuf\ufb01cient conditions on the sample complexity coincide, establishing the exact sample complexity. In\nthis regime, as a result, the ML decoder achieves an optimal structure learning rule. We used the ML\nrule to characterize the variations of decision reliability P(I\u03b7, d) as the size of the graph varied in the\nrange p \u2208 [50, 500] for \ufb01xed number of edges. Figure 3 depicts these variations. For the results in\nthis \ufb01gure we have \ufb01xed \u03b1 = 20 and \u03b7 = 0.5, and have evaluated the performance based on n = 40\nsamples from each graph.\nIn Fig. 3, we observe that the decision reliability measure P(I\u03b7, d) achieves lower error rate with\nincrease in d for the same number of samples. It is important to note that increase in d signi\ufb01es a\nrise in tolerance to errors in the structure recovery by the graph decoder, and therefore, the decline in\nquality of structure recovery decisions with respect to the ground truth. Also, as stated in Corollary 1,\nwe observe that graph recovery becomes more dif\ufb01cult as the graph size increases while k remains\n\ufb01xed.\n\n6 Conclusion\n\nIn this paper, we have considered the problem of structure learning in the presence of side information\nabout the structure. This is posed, naturally, as jointly recovering the structures of two graphs with\npartial internal structural similarities. Speci\ufb01cally, it is assumed that both graphs share an identical\nsubgraph. Any inference about the structure of this subgraph from either of the graphs serves as\nthe side information for recovering the structure of the other graph. A general recovery criterion\nthat encompasses both exact and partial recovery of the graphs is considered. We have established\nnecessary (information-theoretic) and suf\ufb01cient (algorithmic) bounds on the sample complexity\nfor achieving a bounded probability of error in structure recovery. The scaling behaviors of these\nconditions are analyzed in different regimes. We have also identi\ufb01ed a regime in which the necessary\nand suf\ufb01cient conditions coincide, establishing the optimal sample complexity. We have also provided\nnumerical evaluations to illustrate the interplay among the various parameters involved.\nThe setting studied in this paper has been motivated from applications in a broad range of domains\nlike social networks, genetics, and behavioral analysis. While the existing works have primarily\nfocused on context speci\ufb01c algorithmic frameworks for joint inference, our results have established\nthe information-theoretic benchmarks on the sample complexity in different regimes characterized by\nthe properties of the graph structures.\n\nReferences\nO. Banerjee, L. E. Ghaoui, and A. d\u2019Aspremont. Model selection through sparse maximum likelihood\nestimation for multivariate Gaussian or binary data. Journal of Machine learning research, 9:\n485\u2013516, Jun. 2008.\n\nX. Chen, F. J. Slack, and H. Zhao. Joint analysis of expression pro\ufb01les from multiple cancers\nimproves the identi\ufb01cation of microRNA\u2013gene interactions. Bioinformatics, 29(17):2137\u20132145,\n2013.\n\nD. M. Chickering. Learning Bayesian networks is NP-complete. Learning from data, 112:121\u2013130,\n\n1996.\n\nP. Danaher, P. Wang, and D. M. Witten. The joint graphical lasso for inverse covariance estimation\nacross multiple classes. Journal of the Royal Statistical Society: Series B (Statistical Methodology),\n76(2):373\u2013397, Mar. 2014.\n\nA. K. Das, P. Netrapalli, S. Sanghavi, and S. Vishwanath. Learning Markov graphs up to edit distance.\nIn IEEE International Symposium on Information Theory, pages 2731\u20132735, Cambridge, MA, Jul.\n2012.\n\nL. Devroye, A. Mehrabian, and T. Reddad. The minimax learning rate of Normal and Ising undirected\n\nGraphical models. arXiv preprint arXiv:1806.06887, 2018.\n\n9\n\n\fA. Dobra, C. Hans, B. Jones, J. R. Nevins, G. Yao, and M. West. Sparse graphical models for\n\nexploring gene expression data. Journal of Multivariate Analysis, 90(1):196\u2013212, 2004.\n\nK. Dvijotham, M. Chertkov, P. V. Hentenryck, M. Vuffray, and S. Misra. Graphical models for\n\noptimal power \ufb02ow. Constraints, 22(1):24\u201349, 2017.\n\nJ. Fang, L. S. Dongdong, S. Charles, Z. Xu, V. D. Calhoun, and Y.-P. Wang. Joint sparse canonical\ncorrelation analysis for detecting differential imaging genetics modules. Bioinformatics, 32(15):\n3480\u20133488, 2016.\n\nD. E. Fishkind, S. Adali, H. G. Patsolic, L. Meng, D. Singh, V. Lyzinski, and C. E. Priebe. Seeded\n\ngraph matching. Pattern Recognition, 87:203 \u2013 215, Mar. 2019.\n\nA. Gangrade, B. Nazer, and V. Saligrama. Lower bounds for two-sample structural change detection\nin ising and Gaussian models. In Annual Allerton Conference on Communication, Control, and\nComputing, pages 1016\u20131025, Allerton, IL, Oct. 2017.\n\nJ. Guo, E. Levina, G. Michailidis, and J. Zhu.\n\nBiometrika, 98(1):1\u201315, 2011.\n\nJoint estimation of multiple graphical models.\n\nJ. Guo, J. Cheng, E. Levina, G. Michailidis, and J. Zhu. Estimating heterogeneous graphical models\nfor discrete data with an application to roll call voting. The Annals of Applied Statistics, 9(2):821 \u2013\n848, Jun. 2015.\n\nY. Jacob, L. Denoyer, and P. Gallinari. Learning latent representations of nodes for classifying in\nheterogeneous social networks. In Proc. ACM international Conference on Web Search and Data\nMining, pages 373\u2013382, New York, Feb. 2014.\n\nS. L. Lauritzen. Graphical models, volume 17. Clarendon Press, May 1996.\n\nV. Lyzinski, D. E. Fishkind, and C. E. Priebe. Seeded graph matching for correlated Erd\u00f6s-R\u00e9nyi\n\ngraphs. Journal of Machine Learning Research, 15(1):3513\u20133540, Jan. 2014.\n\nK. Mohan, P. London, M. Fazel, D. Witten, and S.-I. Lee. Node-based learning of multiple Gaussian\n\ngraphical models. The Journal of Machine Learning Research, 15(1):445\u2013488, 2014.\n\nM. Neykov and H. Liu. Property Testing in High Dimensional Ising models. arXiv preprint\n\narXiv:1709.06688, 2017.\n\nJ. Pearl. Causality: models, reasoning, and inference. Oxford: Cambridge University Press, 2009.\n\nC. B. Peterson, F. C. Stingo, and M. Vannucci. Bayesian inference of multiple Gaussian graphical\n\nmodels. Journal of the American Statistical Association, 110(509):159\u2013174, 2015.\n\nH. Qiu, F. Han, H. Liu, and B. Caffo. Joint estimation of multiple graphical models from high-\ndimensional time series. Journal of the Royal Statistical Society: Series B (Statistical Methodology),\n78(2):487\u2013504, 2016.\n\nP. Ravikumar, M. J. Wainwright, and J. D. Lafferty. High-dimensional Ising model selection using\n\nl1-regularized logistic regression. The Annals of Statistics, 38(3):1287\u20131319, Jun. 2010.\n\nA. J. Rothman, P. J. Bickel, E. Levina, and J. Zhu. Sparse permutation invariant covariance estimation.\n\nElectronic Journal of Statistics, 2:494\u2013515, 2008.\n\nN. P. Santhanam and M. J. Wainwright. Information-theoretic limits of selecting binary graphical\n\nmodels in high dimensions. IEEE Trans. Information Theory, 58(7):4117\u20134134, May 2012.\n\nJ. Scarlett and V. Cevher. On the dif\ufb01culty of selecting Ising models with approximate recovery. IEEE\n\nTransactions on Signal and Information Processing over Networks, 2(4):625\u2013638, Dec. 2016.\n\nS. Sihag and A. Tajer. Sample complexity of joint structure learning.\n\nIn Proc. International\nConference on Acoustics, Speech and Signal Processing (ICASSP), pages 5292\u20135296, Brighton,\nUK, May 2019.\n\n10\n\n\fR. Tandon and P. Ravikumar. On the dif\ufb01culty of learning power law graphical models. In Proc.\nIEEE International Symposium on Information Theory, pages 2493\u20132497, Istanbul, Turkey, Jul.\n2013.\n\nR. Tandon, K. Shanmugam, P. K. Ravikumar, and A. G. Dimakis. On the information theoretic limits\nof learning Ising models. In Proc. Advances in Neural Information Processing Systems, pages\n2303\u20132311, Montreal, Canada, Dec. 2014.\n\nD. Vats and J. M. Moura. Necessary conditions for consistent set-based graphical model selection. In\nProc. IEEE International Symposium on Information Theory, pages 303\u2013307, Saint-Petersburg,\nRussia, Jul. 2011.\n\nW. Wang, M. J. Wainwright, and K. Ramchandran. Information-theoretic bounds on model selection\nfor Gaussian Markov random \ufb01elds. In Proc. IEEE International Symposium on Information\nTheory, Austin, Texas, Jun. 2010.\n\nC. S. Won and H. Derin. Unsupervised segmentation of noisy and textured images using Markov\n\nrandom \ufb01elds. CVGIP: Graphical models and image processing, 54(4):308\u2013328, 1992.\n\nS. Yang, Z. Lu, X. Shen, P. Wonka, and J. Ye. Fused multiple graphical lasso. SIAM Journal on\n\nOptimization, 25(2):916\u2013943, 2015.\n\nM. Yuan and Y. Lin. Model selection and estimation in the Gaussian graphical model. Biometrika,\n\n94(1):19\u201335, 2007.\n\n11\n\n\f", "award": [], "sourceid": 8139, "authors": [{"given_name": "Saurabh", "family_name": "Sihag", "institution": "Rensselaer Polytechnic Institute"}, {"given_name": "Ali", "family_name": "Tajer", "institution": "Rensselaer Polytechnic Institute"}]}