{"title": "Finding Latent Causes in Causal Networks: an Efficient Approach Based on Markov Blankets", "book": "Advances in Neural Information Processing Systems", "page_first": 1249, "page_last": 1256, "abstract": "Causal structure-discovery techniques usually assume that all causes of more than one variable are observed. This is the so-called causal sufficiency assumption. In practice, it is untestable, and often violated. In this paper, we present an efficient causal structure-learning algorithm, suited for causally insufficient data. Similar to algorithms such as IC* and FCI, the proposed approach drops the causal sufficiency assumption and learns a structure that indicates (potential) latent causes for pairs of observed variables. Assuming a constant local density of the data-generating graph, our algorithm makes a quadratic number of conditional-independence tests w.r.t. the number of variables. We show with experiments that our algorithm is comparable to the state-of-the-art FCI algorithm in accuracy, while being several orders of magnitude faster on large problems. We conclude that MBCS* makes a new range of causally insufficient problems computationally tractable.", "full_text": "Finding Latent Causes in Causal Networks: \n\nan Efficient Approach Based on Markov Blankets \n\nJean-Philippe Pellet1,2 \njep@zurich . ibm . com \n\n1 Pattern Recognition and Machine Learning Group \n\nSwiss Federal Institute of Technology Zurich \n\n8092 Zurich, Switzerland \n\nAndre Elisseeff2 \n\nae l@ zurich.ibm .com \n2 Data Analytics Group \nIBM Research GmbH \n\n8803 Rlischlikon, Switzerland \n\nAbstract \n\nCausal structure-discovery techniques usually assume that all causes of more than \none variable are observed. This is the so-called causal sufficiency assumption. \nIn practice, it is untestable, and often violated. In this paper, we present an ef(cid:173)\nficient causal structure-learning algorithm, suited for causally insufficient data. \nSimilar to algorithms such as IC* and FCI, the proposed approach drops the \ncausal sufficiency assumption and learns a structure that indicates (potential) latent \ncauses for pairs of observed variables. Assuming a constant local density of the \ndata-generating graph, our algorithm makes a quadratic number of conditional(cid:173)\nindependence tests w.r.t. the number of variables. We show with experiments \nthat our algorithm is comparable to the state-of-the-art FCI algorithm in accuracy, \nwhile being several orders of magnitude faster on large problems. We conclude \nthat MBCS* makes a new range of causally insufficient problems computationally \ntractable. \nKeywords: Graphical Models, Structure Learning, Causal Inference. \n\n1 Introduction: Task Definition & Related Work \n\nThe statistical definition of causality pioneered by Pearl (2000) and Spirtes et al. (2001) has shed \nnew light on how to detect causation. Central in this approach is the automated detection of cause(cid:173)\neffect relationships using observational (i.e., non-experimental) data. This can be a necessary task, \nas in many situations, performing randomized controlled experiments to unveil causation can be im(cid:173)\npossible, unethical, or too costly. When the analysis deals with variables that cannot be manipulated, \nbeing able to learn from data collected by observing the running system is the only possibility. \n\nIt turns out that learning the full causal structure of a set of variables is, in its most general form , \nimpossible. If we suppose that the \"causal ground truth\" can be represented by a directed acyclic \ngraph (DAG) over the variables to analyze, denoted by V, where the arcs denote direct causation, \ncurrent causal structure-learning algorithms can only learn an equivalence class representing statis(cid:173)\ntically indistinguishable DAGs. This class can be represented by a partially directed acyclic graph \n(PDAG), where arcs between variables may be undirected, indicating that both directions are equaJly \npossible given the data. This is know as the problem of causal underdetermination (Pearl, 2000). \n\nCommon to most structure-learning algorithms are three important assumptions which ensure the \ncorrectness of the causal claims entailed by the returned PDAG (see Scheines, 1997, for a more \nextensive discussion of these assumptions and of their implications). First, the causal Markov con(cid:173)\ndition states that every variable is independent of its non-effects given its direct causes. It implies \nthat every dependency can be explained by some form of causation (direct, indirect, common cause, \nor any combination). Second, the faithfulness condition demands that the dependencies be DAG(cid:173)\nisomorphic; i.e., that there be a DAG whose entailed variable dependencies coincide exactly with \n\n\fthe dependencies found in the data. Third, causal sufficiency of the data states that every common \ncause for two variables in V is also in V. Causal sufficiency often appears as the most contro(cid:173)\nversial assumption as it is generally considered impossible to ensure that all possible causes are \nmeasured-there is no such thing as a closed world. In this paper, we are interested in relaxing \ncausal sufficiency: we do not require the data to contain all common causes of pairs of variables. \n\nSome of the few algorithms that relax causal sufficiency are Inductive Causation* (IC*) by Pearl and \nVerma (1991); Pearl (2000), and Fast Causal Inference (FCI) by Spirtes et al. (1995, 2001). The kind \nof graph IC* and FCI return is known as a partial ancestral graph (PAG), which indicate for each link \nwhether it (potentially) is the manifestation of a hidden common cause for the two linked variables. \nAssuming continuous variables with linear causal influences, Silva et al. (2006) recover hidden \nvariables that are the cause for more than two observed variables, to infer the relationships between \nthe hidden variables themselves. They check additional constraints on the covariance matrix, known \nas tetrad constraints (Scheines et aI., 1995), entailed by special kinds of hidden structures. \n\nThere are more specialized techniques to deal with hidden variables. Elidan et al. (2001) look for \nstructural signatures of hidden variables in a learned DAG model. Boyen et al. (1999) describe a \ntechnique that looks for violation of the Markov condition to infer the presence of latent variables in \nBayesian networks. Once a hidden variable is identified, Elidan and Friedman (2001) discuss how \nto assign it a given dimensionality to best model its interactions with the observed variables. \n\nIn this paper, we describe recent advances in making the PAG-learning task tractable for a wider \nrange of problems, and present the Markov blanketlcollidet set (MBCS*) algorithm. In Section 2, \nwe formally describe the PAG-Iearning task and motivate it with an example. Section 3 describes \nFCI. We then present MBCS* in Section 4 and compare it experimentally to FCI in Section 5. We \nfinally conclude in Section 6. Correctness proofs are provided in the supplemental material l . \n\nNotation Throughout this paper, uppercase capitals such as X and Y denote variables or nodes in \na graph and sets of variables are set in boldface, such as V. Hand L (possibly with indices) denote \nlatent (unobserved) variables. Bold lowercase greek characters such as 7r are paths (ordered list of \nnodes), while the calligraphic letter 9 refers to a graph. Finally, we denote conditional independence \nof X and Y given Z by the notation (X Jl Y I Z). \n\n2 Mixed Ancestral Graphs & Partial Ancestral Graphs \n\nIn this section, we first introduce the notation of mixed ancestral graphs (MAGs) and partial ancestral \ngraphs (pAGs) used by Spirtes et al. (1996) and describe how to learn them on a high level. We first \nreview the definition of a V-structure. \n\nDefinition 2.1 (V-structure) In a causal DAG, a V-structure is a triplet X ---> Z f - Y, where X \nand Yare nonadjacent. Z is then called an unshielded collider for X and Y. Its presence implies: \n\n::JS Xy<;;; V \\{X , Y ,Z}: ((X JlY ISxy) and (X.,ilY I Sxy u{Z})), \n\n(1) \n\nIn a V-structure, two causes X and Y, which are made independent by S xy, become dependent \nwhen conditioned on a common effect Z (or one of its descendants). This is the base fact that allows \ninitial edge orientation in causal structure learning. \n\nLet us now suppose we are learning from data whose (unknown) actual causal DAG is: \n\nFurther assume that HI and H2 are hidden. Assuming the adjacencies X - Y - Z - W have \nbeen found, conditional-independence tests will reveal that (X Jl Z) and (X .,il Z I Y), which is \na sufficient condition for the V-structure X ---> Y f - Z. Similarly, (Y Jl W) and (Y .,il W I Z) is \na sufficient condition for the V-structure Y ---> Z f - W. In a DAG like in a PDAG, however, those \ntwo overlapping V-structures are incompatible. The simplest DAG compatible with those findings \nneeds the addition of an extra variable H: \n\nX ---> Y f - H ---> Z f - W. \n\n(3) \n\nI Available at http : //j p.pellet.name/publis/pellet08nips_supplement . pdf. \n\n(2) \n\n\fActually, (3) is the projection of the latent structure in (2). In the projection of a latent structure as \ndefined by Pearl (2000), all hidden variables are parentless and have only two direct effects. Verma \n(1993) proved that any hidden structure has at least one projection. Notice that we cannot recover \nthe information about the two separate hidden variables HI and H 2 . In the projection, information \ncan thus be lost with respect to the true latent structure. \n\nWhereas causally sufficient datasets are represented as DAGs and learned as PDAGs to represent \nindependence-equivalent DAGs, the projection of latent structures is represented by special graphs \nknown as mixed ancestral graphs (MAGs) (Spirtes et aI., 2001), which allow for bidirected arrows to \nrepresent a hidden cause for a pair of variables. Independence-equivalent MAGs are represented by \npartial ancestral graphs (PAGs). PAGs are thus to MAGs what PDAGs are to DAGs, and structure(cid:173)\nlearning algorithms like FCI return a PAG. \n\nPAGs allow four kinds of arrows: ----7, 0----7, <J-O, and +----+. X ----7 Y in the PAG denotes true causation \nX ----7 Y in the projection; X +----+ Y indicates the presence of a latent cause X +-- H ----7 Y (without \nexcluding direct causation); X 0----7 Y denotes either true causation X ----7 Y or a latent cause X +-(cid:173)\nH ----7 Y (or a combination of both); finally, X <J-O Y denotes potential causation from X ----7 Y \nor Y ----7 X and/or a latent common cause X +-- H ----7 Y in the projection, and is thus the most \n\"agnostic link.\" An asterix as an arrowhead is a wildcard for any of the three possible endpoints of \na link, such that X ...--+ Y, for instance, means any of X ----7 Y, X 0----7 Y, and X +----+ Y. Additionally, \nwe also use the notation X <--* ~ <--* Y to indicate that Z is a definite noncollider for X and Y, such \nthat any of X ...--+ Z ----7 Y, X+-- Z f-* Y , or X+-- Z ----7 Y can occur, but not X ...--+ Z f-* Y. \n\nTo illustrate how MAGs and PAGs are related to a latent structure, consider the causal graph shown \nin Figure I (i). There, the hidden variable H is a cause for 3 observed variables, and L is a hidden \nvariable in the causal chain from Z to W. All other variables are observed. In (ii), we show the \nprojection of (i): note that we lose information about L and about the fact that HI, H 2 , and H 3 \nare actually the same variable. The corresponding MAG is shown in (iii), and in (iv) the PAG that \nrepresents the class of independence-equivalent MAGs of which (iii) is a member. Note how the \ncausal-underdetermination problem influences PAG learning: for instance, the model shown in (vi) , \nif learned as a PAG, will also be represented as in (iv). (v) is commented on later in the text. \n\nS +-H \n\nx~ /1 \nZ+-Y \nI \n\nL ~W \n\nS+-HI \n\nxi? E2 1 x~l~ x~!~ x~ x~l~ \n\n\\\" \n\nS \n\nS \n\n/ \nZ+-Y \ntB?' \nW \n\n3 \n\nZ~Y z~ \nt \nW \n\nt \nW \n\nS \n\nI \nW \n\n(v) \n\nS \n\nZ +-Y \nt \nW \n\n(vi) \n\n(i) \n\n(ii) \n\n(iii) \n\n(iv) \n\nFigure 1: (i) Example of a causal structure with the hidden variables Hand L. (ii) Projection of (i). \n(iii) MAG representing the projection (ii). (iv) PAG representing the class of projections \nthat are independence-equivalent to (iii). (v) The moral graph of (iii). (vi) Another struc(cid:173)\nture with no hidden variable whose learned PAG is (iv). \n\n3 Learning PAGs with the FeI Algorithm \n\nThis section now turns to the task of learning PAGs with conditional-independence tests and de(cid:173)\nscribes shortly the reference algorithm, FCI. \n\nIn principle, learning the structure of a PAG is not much different from learning the structure of a \nPDAG. The main difference is that instead of creating V-structures in a PDAG, we now just add \narrow heads into the identified colliders, independently of what the other arrow endpoints are. A \nPAG-Iearning algorithm could thus operate this way: \n\n1. Adjacencies: insert the \"agnostic link\" X <J-O Y if IfS c:;;; V \\ {X, Y} : (X )l Y I S); \n2. V-structures: when the condition (1) holds for triplet (X , Z , Y) , add arrow heads into Z; \n3. Orientations: use rules to further orient \"agnostic\" endpoints wherever possible. \n\nThe second difference w.r.t. PDAG learning is in the set of rules applied in Step 3 to further orient \nthe graph. Those rules are detailed in the next subsection . \n\n\fTo the best of our knowledge, the FCI algorithm is regarded as the state-of-the-art implementa(cid:173)\ntion of a PAG-Iearning algorithm. We list its pseudocode in Algorithm]. The notation Nb(X) \nstands for the set of direct neighbors of X in the graph being constructed g (and potentially \nchanges at each iteration). The set ExtDSep(X, Y) is the union of Possible-D-Sep(X, Y) and \nPossible-D-Sep(Y, X). Possible-D-Sep(X, Y) is the set of nodes Z where there is an undi(cid:173)\nrected path 7r between X and Z such that for each subpath 8 ...... W ...... T of 7r, either (a) W is a \ncollider; or (b) W is not marked as a noncollider and 8, W, T are a triangle. (A triangle is a set of \nthree nodes all adjacent to one another.) \n\nWe list the orientation rules as a separate procedure in Algorithm 2, as we reuse them in our algo(cid:173)\nrithm. Rule I preserves acyc1icity. Rule 2 honors the noncollider constraint when one of the two \nendpoints is an arrowhead. Rule 3 orients double-triangle structures; for instance; it orients 8 o----t Z \nin Figure 1 (iii). Rule 4 needs the following definition (Spirtes et al., 1995). \n\nDefinition 3.1 (DDP) In a PAG g, 7r is a definite discriminating path (DDP) between 8 and Y \n(8 , Y nonadjacent) for Z (Z -=I- 8, Y) if and only if 7r is an undirected path between 8 and Y \ncontaining Z, Z precedes Y on 7r, every vertex V between 8 and Z on 7r is a collider or a definite \nnoncollider on 7r, and.' \n\n(i) if V and V' are adjacent on 7r and Viis between V and Z, then V +---+ V' on 7r; \n(ii) if V is between 8 and Z on 7r and V is a collider on 7r, then V ~ Yin g, else Y +---+ V in g. \n\nFigure 2 shows an example for a DDP and for Rule 4, wruch produces the orientation Z +---<> Y. For \na more extensive justification and a proof of those rules, see Spirtes et al. (1995, 2001). \n\nThe time complexity of FCI makes it non-scalable for larger networks. In particular, the two subset \nsearches at lines 5 and 19 of Algorithm I are computationally costly in dense networks. In the next \nsection, we present an algorithm that takes another approach at PAG learning to tackle problems \nlarger than those that FCI can handle. \n\nFigure 2: Path 7r = (8 , V , X , Z , Y ) is a DDP for Z. Rule 4 adds an arrow head into Z if Z tj. S SY . \n\n4 Efficient Structure Learning with the MBCS* Algorithm \n\nIn this section, we propose a PAG-Iearning algorithm, MBCS*, which is more efficient than FCI in \nthe sense that it performs much fewer conditional-independence tests, whose average conditioning(cid:173)\nset size is smaller. We show in Section 5 that MBCS* compares very favorably to FCI on test \nnetworks in terms of computational tractability, while reaching similar accuracy. Pseudocode for \nMCBS* is listed in Algorithm 3. \n\nMBCS* proceeds in three steps: first, it detects the Markov blankets for each variable; second, \nit examines the triangle structures to identify colliders and noncolliders; finally, it uses the same \norientation rules as FCI to obtain the maximally oriented PAG. We detail the first two steps below; \nthe orientation rules are the same as for FCI. \n\n4.1 Step 1: Learning the Markov Blanket \n\nThe first phase of MBCS * builds an undirected graph where each variable is connected to all mem(cid:173)\nbers of its Markov blanket. \n\nDefinition 4.1 (Markov blanket) The Markov blanket of a node X is the smallest set of variables \nMb(X) such that 'VY E V \\ Mb(X) \\ {X} : (X Jl Y \n\nI Mb(X)). \n\nAssuming faithfulness, Mb(X) is unique. In a DAG, it corresponds to the parents, children, and \nchildren's parents (spouses) of X. We extend trus to MAGs. \n\n\fAlgorithm 1 9 = FCI(V, J) \nInput: \n\nV : \nI : \nOutput: g: \n1: 9 f-- fully connected graph over V \n2: i f-- 0 \n\nII Detect adjacencies \n\nif (X JL Y I S) then \n\nfor each S <:;; Nb (X) \\ {Y} of size i do \n\nfor each X - Y s.t. INb(X)1 > i do \n\nremove link X - Y from 9 \nS Xy, S yX f-- S \nbreak from loop line 4 \n\n3: while :3(X - Y) s.t. INb(X)1 > i do \n4 : \n5: \n6: \n7: \n8: \n9: \n10: \n11: \n12: \n13: \n14: end while \n15 : for each X - Z - Y s.t. X , Y nonadjacent do \n16: \n17: end for \n\nif Z tt S XY then orient as X --> Z f-- Y \n\nend for \nif--i+l \n\nend for \n\nend if \n\nset of observed variables \na conditional-independence oracle, called with the notation ( . JL . I .) \nmaximally oriented partial ancestral graph \n\nII Detect additional adjacencies \n\nend if \n\nend for \n\nif(X JL Y IS) then \n\nfor each S <:;; ExtDSep(X,Y)\\{X,Y} do \n\nremove link X - Y from 9 \nS Xy, S yxf-- S \nbreak from loop line 18 \n\n18: for each pair of adjacent variables X , Y do \n19: \n20: \n21: \n22: \n23: \n24: \n25: \n26: end for \n27: orient every link as 0-0 \nII Orient V-structures \nif Z tt S XY then orient as X ....... Z f-* Y \nelse mark Z as noncollider: X ...... Z ...... Y \n\n28: for each X ...... Z ...... Y s.t. X , Y nonadjacent do \n29: \n30: \n31: end for \n32: return ORIENTMAXIMALLY(9 , \\f(X,Y): S XY ) \n\nAlgorithm 2 9 = ORIENTMAXIMALLY(9 , a list of sets SXY) \nInput: \n\npartial ancestral graph \nfor (some) nonadjacent pairs (X, Y): a d-separating set of variables \nmaximally oriented partial ancestral graph \n\n9 : \nS XY : \n\nOutput: g: \n1: while 9 is changed by some rule do \n2: \n3: \n4 : \n5: \n6: \n7: \n8: \n9: \nend for \n10: \n11: end while \n\nend if \n\nfor each X *-0 Y such that there is a directed path from X to Y do orient as X ....... Y \nfor each X ....... Z 0-;< Y do orient as X ....... Z --> Y \nfor each X ....... Z f-* Y with S *-0 Z and S E S XY do orient as S ....... Z \nfor each defi nite discriminating path 11\" between Sand Y for Z do \nif S SY exists and Z tt S SY then orient as X ....... Z f-* Y \nelse mark Z as a noncollider X ...... Z ...... Y \n\nif X ...... Y where X is adjacent to Z on 11\" and X , Z, Yare a triangle then \n\nII Rule 1 \nII Rule 2 \nII Rule 3 \nII Rule 4 \n\nAlgorithm 3 9 = MBCS*(V, J) \nInput: \n\nV : \nI : \n\nset of observed variables \na conditional-independence oracle, called with the notation ( . JL . I .) \n\nOutput: g: maximally oriented partial ancestral graph \n\nII Initialization \n\n1: 9 f-- empty graph over V \n\nII Find Markov blankets (Grow-Shrink) \n\n2: for each X E V do \n3: \n4: \n5: \n6: \n7: \n8: \n9: end for \n\nadd Y to S \n\nS f-- empty set of Markov blanket variables \nwhile:3Y E V \\ {X} s.t. (X .,It Y IS) do \n\nwhile :3Y E S s.t. (X JL Y I S \\ {Y} ) do \n\nremove Y from S \n\nfor each Y E S do add link X 0-0 Y \n\nII Add noncollider constraints \n\n10: for each X 0-0 Z 0-0 Y s.t. X , Y nonadjacent do \n11 : \n12: end for \n\nmark as noncollider X 0-0 Z 0-0 Y \n\nII Adjust local structures (Collider Set search) \nC f-- empty list of collider-orientation directives \nfor each X ...... Y in a fully connected triangle do \n\nif :3collider set Z <:;; Tri(X - Y) then \nS XY f-- d-separating set for (X, Y) \nremove link X ...... Y from 9 \nfor each Z E Z do \n\nadd ordered triplet (X, Z, Y) to C \n\nfor each Z E S XY do \n\nmark as noncollider X ...... Z ...... Y \n\nend if \n\nend for \nfor each orientation directive (X, Z , Y) E C do \nif X ...... Z ...... Y then orient as X ....... Z f-* Y \n\nend for \nreturn ORIENTMAXIMALLY(9 , \\f(X,Y): S XY ) \n\n13: \n14: \n15: \n16: \n17: \n18: \n19: \n20: \n21: \n22: \n23: \n24: \n25: \n26: \n27: \n\n\fProperty 4.2 In a faithful MAG, the Markov blanket Mb(X) of a node X is the set of parents, \nchildren, children 's parents (spouses) of X, as well as the district of X and of the children of X, \nand the parents of each node of these districts, where the district of a node Y is the set of all nodes \nreachable from Y using only bidirected edges. (Proofin supplemental material.) \n\nWe use algorithmic ideas from Margaritis and Thrun (1999) to learn the Markov blanket of a node \nwith a linear number of conditional-independence tests (proof in the supplemental material of Mar(cid:173)\ngaritis and Thrun, 1999). This technique is used in lines 3 to 6 of Algorithm 3. The resulting graph is \nan undirected graph called moral graph where each node is connected to its Markov blanket. There(cid:173)\nfore, it contains spurious links to its spouses, to members of its district, to members of its children's \ndistrict, and to parents of nodes in those districts, which we all call SD links (for Spouse/District). \nRemoval of those links is done in the second step of MBCS *. \n\n4.2 Step 2: Removing the SD Links \n\nIn the second step of MBCS*, each undirected edge must be identified as either an SD link to be \nremoved, or a true link of the original MAG to be kept. Direct parents and children are dependent \ngiven any conditioning set, while spouses and district members (and their parents) can be made \nindependent. For each link X - Y, a search is thus performed to try to d-separate the two connected \nnodes. This search can be limited to the smallest of the Markov blankets of X and Y, as by definition \nthey contain all nodes that can minimally make them independent from each other, provided they \nare linked by an SD link. If such a d-separating set S Xy is found, the link is removed. Interestingly, \nidentifying a d-separating set SXY also identifies the collider set for X and Y. \n\nDefinition 4.3 (Collider set) In an undirected graph 9 over V, let Thi(X - Y) (X , Y adjacent) \nbe the set of all vertices that form a triangle with X and Y. Suppose that 9 is the moral graph \nof the DAG or MAG representing the causal structure of a faithful dataset. A set of vertices Z ~ \nThi (X - Y) then has the Collider set property for the pair (X, Y) if it is the largest set that fulfills \n\n:3SXy ~ V \\ {X, Y} \\ Z:(X Jl Y ISxy) \n\nand VZ E Z : (X .,Ii Y I S X Y U {Z}). \n\nCollider sets are useful because each node in them satisfies the property of a collider (1) and reveals \na V-structure. Suppose (X Jl Y I SXy): then each node Z (connected by a non-SD link to both X \nand Y) not in S Xy is a collider. This follows from the fact that for each path X ....... Z ....... Y where \nZ rf. S xy , the only structural possibility is to have arrow head pointing into Z by the definition of d(cid:173)\nseparation (Pearl, 1988). Similarly, if Z E SXY, then any orientation is possible save for a collider. \nThose two types of constraints appear in lines 19 and 21 of Algorithm 3, respectively. Note that \nmore noncollider constraints are added in line 11: in the case X 0-0 Z 0-0 Y with X , Y nonadjacent, \nwe know that Z cannot be a V-structure owing to the following lemma. \n\nLemma 4.4 In the moral graph gm of a DAG or a MAG g, whenever the pattern X ;<--7 Z +---> Y \noccurs in g, then X and Yare linked in gm. (Proof in supplemental material.) \n\nIn practice, the search for collider sets and simultaneously for d-separating sets in lines 15 and 16 \nis performed following the implementation proposed by Pellet and Elisseeff (2008). They also \ndiscuss why V-structure orientations must be delayed to line 25 instead of being made immediately \nin line 19. \n\nIn the supplemental material to this paper, we prove that MBCS* correctly identifies all adjacencies \nand V-structures. The final orientation step (Algorithm 2) requires d-separating-set information for \nRules 3 and 4: we also prove that MBCS* provides all necessary information. \n\n5 Experimental Evaluation \n\nWe now compare FCI and MBCS* with a series of experiments. We took two standard benchmark \nnetworks, ALARM and HAILFINDER, and for each of them, chose to hide 0, 1, 2, and 3 variables, \ncreating in total 8 learning problems. On a first series on experiments, the algorithms were run with \na d-separation oracle, which is equivalent to perfect conditional-independence tests. Conditioning \n\n\fTable 1: Comparison of MBCS* and FCI where conditional-independence tests are done using a \nd-separation oracle. We report the number of tests t; the weighted number of tests wt, \nwhere each test contributes to wt a summand equal to the size of its conditioning set; and \nthe ratio of t for FCI over the t for MBCS*: r ~ t (FCI) It (MBCS*). \n\nAlg. \n\nMBCS* \n\nFCI \n\nMBCS * \n\nFCI \n\nMBCS * \n\nFCI \n\nMBCS* \n\nFCI \n\n#hid. v. \n\nALARM \n\nt \n\nwt \n\nr \n\n#hid. v. \n\nHAILFINDER \n\nt \n\nwt \n\nr \n\no \n\n2 \n\n3 \n\n2,237 \n9,340 \n\n3,397 \n21,497 \n\n5,208 \n31,018 \n\n12,123 \n27,666 \n\n18,113 \n95,497 \n\n27,576 \n145,322 \n\n7,527 \n231,096 \n\n42,133 \n1,612,106 \n\n4 \nI \n-\n6 \nl \n-\n6 \nl \n-\n30 \n\no \n\n2 \n\n3 \n\n5,333 \n2,254,774 \n6,516 \n2,302,707 \n\n35,841 \n20,153,894 \n\n42,379 \n20,448,775 \n\n7,205 \n2,324,503 \n\n46,291 \n20,608,841 \n\n18,244 \n2,622,312 \n\n117,209 \n22,888,622 \n\nl \n423 \nI \n\n353 \n\nl \n\n322 \n\n143 \n\n50 0==0 - - - - - - - - - , \n\nHailfinder \n\n40 \n\nAlarm \n\n_\n\nEdgeerrors \n\n35 c=J Orientation errors \n\n'\" e 30 \" '0 25 \" ~ 20 \n\n:J \nC \nQl 15 \n~ \n~ 10 \n\n5 \no \n\nMBCS' FCI MBCS' FCI MBCS' FCI MBCS' FCI \n\nMBCS'FCI MBCS'FCI MBCS' FCI MBCS' FCI \n\no hid.v. \n\n1 hid.v. \n\n2 hid.v. \n\n3 hid.v. \n\no hid.v. \n\n1 hid.v. \n\n2 hid.v. \n\n3 hid.v. \n\nFigure 3: Comparison of MBCS* and FCI where conditional-independence tests are done using \n\nFisher's z-test. We compare the number of edge errors (missing/extraneous) and orienta(cid:173)\ntion errors (including missing/extraneous hidden variables). Error bars show the standard \ndeviation over the 5 runs. \n\non hidden variables was prohibited. The results are listed in Table 1. In a second series, multivariate \nGaussian datasets (with 500 datapoints) were sampled from the networks and data corresponding \nto the hidden variables were removed. The algorithms were run with Fisher's z-test on partial \ncorrelation as conditional-independence test. This was repeated 5 times for each learning problem. 2 \nFor FCI, we used the authors' implementation in TETRAD (Scheines et aI., 1995). MBCS* was \nimplemented in Matlab. See Figure 3 for the comparison. \nTable 1 shows in the columns named t that MBCS* makes up to 3 orders of magnitude fewer \nconditional-independence tests than FCI on the tested networks. As the number of tests alone does \nnot reflect the quality of the algorithm, we also list in the wt column a weighted sum of tests, where \neach test is weighted by the size of its conditioning set. As the ALARM network becomes denser \nby hiding certain variables, the difference between FCI and MBCS* becomes even more apparent. \nThe inverse phenomenon is to be observed for HAILFINDER, where the difference between FCI and \nMBCS* gets smaller: this is because this network is more densely connected, and both algorithms \nexhibit a behavior gradually evolving towards the worst case of the fully connected graph. FCI \nslowly \"catches up\" with MBCS* in those circumstances. \n\n2We would have liked to both vary the number of samples for each dataset and include more test networks, \nbut the running times of Fer on the larger instances, even when run with an upper limit on the maximum size \nof conditioning sets, were prohibitive, ranging up to a week on dense networks on a 2 GHz machine. \n\n\fFigure 3 essentially shows that the difference of accuracy between FCI and MBCS* is not significant \nin either way. On each learning problem, the returned PAGs have been checked for correctness with \nrespect to the maximally oriented PAG go theoretically obtainable (as returned by the first series of \nexperiments). The discrepancies were classified either as edge errors (when an arc was missing or \nextraneous in the returned PAG W.r.t. go), or orientation errors (when a predicted arc in the returned \nPAG was indeed present in go, but had a reversed direction or different end points). On aIlS learning \nproblems, both edge and orientation errors are similar within the margin indicated by the standard(cid:173)\ndeviation error bars. \n\nNote that the overall relatively high error rate comes from the failure of statistical tests with lim(cid:173)\nited sample size. This indicates that structure learning is a hard problem and that low-sample-size \nsituations where tests typically fail must be investigated further. \n\n6 Conclusion \n\nWith the formalism of MAGs and PAGs, it is possible to learn an independence-equivalence class \nof projections of latent structures. We have shown an algorithm, MBCS*, which is much more \nefficient than the reference FCI algorithms on networks that are sufficiently sparse, making up to \nthree orders of magnitude fewer conditional-independence tests to retrieve the same structure. We \nhave experimental evidence that structural accuracy of MBCS* is as good as that of FCI. MBCS* \nis based on a first phase that identifies the Markov blanket of the underlying MAG, and then makes \nlocal adjustments to remove the spurious links and identify all colliders. The last step involving \norientation rules is the same as for FCI. The reduced practical complexity makes MBCS* solve in \nminutes problems that FCI would need several days to solve. In that sense, MBCS* makes a whole \nnew range of problems computationally tractable. \n\nReferences \n\nX. Boyen, N. Friedman, and D. Koller. Discovering the hidden structure of complex dynamic systems. In \n\nProceedings of the 15th Conference on Uncertainty in Artijicial1ntelligence, 1999. \n\nG. Elidan and N. Friedman. Learning the dimensionality of hidden variables. In Proceedings of the 17th \n\nConference in Uncertainty in Artijicial1ntelligence, pages 144-151 , 2001. \n\nG. Elidan, N. Lotner, N. Friedman, and D. Koller. Discovering hidden variables: A structure-based approach. \n\nIn Proceedings of the 13th Conference on Advances in Neural Information Processing Systems, 2001. \n\nD. Margaritis and S. Thrun. Bayesian network induction via local neighborhoods. In Advances in Neural \n\nInformation Processing Systems 12, 1999. \n\n1. Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, 2000. \n1. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, \n\nLos Altos, 1988. \n\n1. Pearl and T. Verma. A theory of inferred causation. In Proc. of the Second Int. Con! on Principles of \n\nKnowledge Representation and Reasoning. Morgan Kaufmann, 1991. \n\nJ.-P. Pellet and A. Elisseeff. Using Markov blankets for causal structure learning. Journal of Machine Learning \n\nResearch, 9: 1295-1342, 2008. \n\nR. Scheines. An introduction to causal inference. In V. McKim and S. Turner, editors, Causality in Crisis?, \n\npages 185- 200. Univ. of Notre Dame Press, 1997. \n\nR. Scheines, P. Spirtes, C. Glymour, C. Meek, and T. Richardson. The TETRAD project: Constraint based aids \n\nto causal model specification. Technical report, Carnegie Mellon University, Dpt. of Philosophy, 1995. \n\nR. Silva, R. Scheines, C. Glymour, and P. Spirtes. Learning the structure of linear latent variable models. \n\nJournal of Machine Learning Research, 7: 191-246,2006. \n\nP. Spirtes, C. Meek, and T. Richardson. Causal inference in the presence of latent variables and selection \nbias. In Philippe Besnard and Steve Hanks, editors, Proceedings of the 11th Conference on Uncertainty in \nArtijicialIntelligence, pages 491--498, San Mateo, CA, 1995. Morgan Kaufmann . \n\nP. Spirtes, T. Richardson, and C. Meek. Heuristic greedy search algorithms for latent variable models. In \n\nProceedings of the 6th International Workshop on Artijiciallntelligence and Statistics, 1996. \n\nP. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search, Second Edition. The MIT Press, \n\n200 I. ISBN 0262194406. \n\nT. Verma. Graphical aspects of causal models. Technical Report R-191, Cognitive Systems Laboratory, UCLA, \n\n1993. \n\n\f", "award": [], "sourceid": 509, "authors": [{"given_name": "Jean-philippe", "family_name": "Pellet", "institution": null}, {"given_name": "Andr\u00e9", "family_name": "Elisseeff", "institution": null}]}