{"title": "Linear-time inference in Hierarchical HMMs", "book": "Advances in Neural Information Processing Systems", "page_first": 833, "page_last": 840, "abstract": null, "full_text": "Linear Time Inference in Hierarchical HMMs\n\nKevin P. Murphy and Mark A. Paskin\n\nComputer Science Department\n\nUniversity of California\nBerkeley, CA 94720-1776\n\n murphyk,paskin\n\n@cs.berkeley.edu\n\nAbstract\n\nThe hierarchical hidden Markov model (HHMM) is a generalization of\nthe hidden Markov model (HMM) that models sequences with structure\nat many length/time scales [FST98]. Unfortunately, the original infer-\nis\nence algorithm is rather complicated, and takes\nthe length of the sequence, making it impractical for many domains. In\nthis paper, we show how HHMMs are a special kind of dynamic Bayesian\nnetwork (DBN), and thereby derive a much simpler inference algorithm,\nwhich only takes\ntime. Furthermore, by drawing the connection\nbetween HHMMs and DBNs, we enable the application of many stan-\ndard approximation techniques to further speed up inference.\n\ntime, where\n\n\u0002\u0004\u0003\u0006\u0005\b\u0007\n\t\n\n\u0002\u0004\u0003\u0006\u0005\u000b\t\n\n1 Introduction\n\nThe Hierarchical HMM [FST98] is an extension of the HMM that is designed to\nmodel domains with hierarchical structure, e.g., natural language, XML, DNA sequences\n[HIM\n00], handwriting [FST98], plan recognition [BVW00], visual action recogntion\n[IB00, ME01, Hoe01], and spatial navigation [TRM01, BVW01]. HHMMs are less ex-\npressive than stochastic context free grammars (SCFGs), since they only allows hierarchies\nof bounded depth, but they are more ef\ufb01cient and easier to learn. Unfortunately, the original\ninference algorithm\n\u0002\u0004\u0003\u0006\u0005\b\u0007\n\u000e\u0010\u000f\u0011\t\ntime, where\nis the\n(maximum) number of states at each level of the hierarchy. In this paper, we show how to\nrepresent an HHMM as a dynamic Bayesian network (DBN), and thereby derive a much\nsimpler and faster inference algorithm, which takes at most\ntime; empirically,\nwe \ufb01nd it takes only\ntime using the junction tree algorithm. Furthermore,\nby drawing the connection between HHMMs and DBNs, we enable the application of ap-\nproximate inference techniques such as belief propagation, which can perform inference in\n\ndescribed in [FST98] is somewhat complicated, and takes\n\nis the depth of the hierarchy, and\n\nis the length of the sequence,\n\n\u0002\u0004\u0003\u0006\u0005\u000b\u000e\u0014\u0013\n\n\u0002\u0004\u0003\u0015\u0005\u0016\u0012\u0017\u000e\u0019\u0018\u001a\r\u001c\u001b\n\n\u001d\u001e\u000f \u001f!\t\n\ntime.\n\n\u0002\u0004\u0003\u0006\u0005\u000b\u0012\n\" By inference, we mean of\ufb02ine smoothing, i.e., conditioning on a \ufb01xed-length observation se-\nquence. This is needed as a subroutine for EM. Once the model has been learned, it will typically be\nused for online inference (\ufb01ltering).\n\n\u0001\n\u0005\n\f\n\n\u0005\n\u0012\n\u000e\n\u000f\n\t\n\u0013\n\u000e\n\u0013\n\t\n\fend\n\nend\n\n0\n\n1\n\n3\n\n2\na\n\n4\nb\n\nend\n\n8\nx\n\nend\n\n7\nd\n\n6\n\nend\n\n5\nc\n\n9\ny\n\nFigure 1: A 3-level hierarchical automaton representing the regular expression\n.\nSolid lines represent horizontal transitions, dotted lines represent vertical transitions. Letters below a\nproduction state represent the symbol that is emitted. The unnumbered root node is considered level\n0, and could be omitted if we fully interconnected states 0 and 1.\n\n\u0001\u0003\u0002\u0005\u0004\u0007\u0006\t\b\u000b\n\r\f\n\n\u000e\u000f\u0001\u0003\u0002\u0005\u0004\u0007\u0006\u0010\b\u000b\u0011\n\ninference algorithm in\nWe will describe HHMMs in Section 2, and the original\nSection 3. The main contribution of the paper is in Section 4, where we show how to\nrepresent an HHMM as a DBN. In Section 5, we discuss how to do ef\ufb01cient inference in\nthis DBN, and in Section 6, we discuss related work. In the full version of this paper, we\ndiscuss how to do parameter and structure learning using EM.\n\n\u0002\u0004\u0003\u0015\u0005\n\n2 Hierarchical HMMs\n\nHHMMs are like HMMs except the states of the stochastic automaton can emit single\nobservations or strings of observations.\n(For simplicity of exposition, we shall assume\nall observations are discrete symbols, but HHMMs can easily be generalized to handle\ncontinuous observations, as we discuss in Section 4.1.) Those that emit single symbols\nare called \u201cproduction states\u201d, and those that emit strings are termed \u201cabstract states\u201d. The\nstrings emitted by abstract states are themselves governed by sub-HHMMs, which can be\ncalled recursively. When the sub-HHMM is \ufb01nished, control is returned to wherever it was\ncalled from; the calling context is memorized using a depth-limited stack.\n\nWe illustrate the generative process with Figure 1, which shows the state transition diagram\nof an example HHMM which models the regular expression\n. We start\nin the root state, and make a \u201cvertical transition\u201d to one of its children, say state 0. From\nhere, we make another vertical transition to state 2. Since state 2 is a production state, it\nemits \u201ca\u201d and then makes a \u201chorizontal transition\u201d to state 3. State 3 calls its sub-HMM,\nwhich emits x\u2019s and y\u2019s until it enters its end state; then control is returned to the calling\nstate, in this case state 3. We then make a horizontal transition to state 4, emit \u201cb\u201d, and\nenter the end state, thereby returning control to state 0. Finally, from state 0, we return\ncontrol to the root, and optionally start again.\n\n\f\u0018\u0017\u001a\u0019\n\n\u0003\u0014\u0013\u0016\u0015\n\n\u0003\u001c\u0013\u001d\u0015\n\n\f\u0018\u001e\n\nAny HHMM can be converted to an HMM by creating a state for every possible legal stack\ncon\ufb01guration\n. If the HHMM transition diagram is a tree, there will be one HMM state\nfor every HHMM production state. If the HHMM transition diagram has shared substruc-\nture (such as the sub-expression\n), this structure must be duplicated in the HMM,\ngenerally resulting in a larger model. It is the ability to reuse sub-models in different con-\n\n\u000e\u0014\r \u001f\n\n\u0003\u0014\u0013\u0016\u0015\n\n\n\u0007\n\t\n\u0012\n\t\n\u001b\n\t\n\t\n\u000f\n!\n\t\n\f\n\ftexts that makes HHMMs more powerful than standard HMMs. In particular, the parame-\nters of such shared sub-models only need to be learned once. (Given segmented data, we\ncan train the sub-HMMs separately, and then \u201cglue them together\u201d, but it is also possible\nto train the HHMM on unsegmented data; see the full version of this paper for details.)\n\n3 Cubic-time inference\n\nThe inference algorithm for HHMMs presented in [FST98] runs in\ntime and is\nbased on the Inside-Outside algorithm [LY90], an exact inference algorithm for stochastic\ncontext-free grammars (SCFGs) which we now describe.\n\n\u0002\u0004\u0003\u0006\u0005\b\u0007\n\n\u0003\u0001\n\n\u0007\u0006\b\n\t\n\n\u0002\u0001\n\u0003\u0001\n\nIn an SCFG, sequences of observations are generated by a set of stochastic production\nrules. Each production rule stochastically rewrites a non-terminal symbol\ninto either\n) or a pair of nonterminal symbols (\n).\na symbol of the alphabet (\nObservation strings are generated by starting with the distinguished \u201cstart\u201d nonterminal\n, and continually re-writing all non-terminals using stochastic production rules until,\n\f\u000b\n\ufb01nally, only symbols of the alphabet remain.\nThe Inside-Outside algorithm computes\nis a subsequence. This can then be used to compute the expected\n!\u0011\u0010\nsuf\ufb01cient statistics needed by the EM algorithm to learn the parameters of the model. If\nthere are\n, then the\nInside-Outside algorithm requires\ntime. To see why, note that we must compute\nsuch that\nand\n\u0003\u0014\n\u0001\n,\ngenerates\n\u0007\u0006\nand\n\u001a!\u001c\n\u0007\u0006\nand\n\n\t\nThe inference algorithm for HHMMs presented in [FST98] is based upon a straight-\nThe algorithm computes\nforward adaptation of\n, that a\n. Hence the algorithm\n\n, and for all midpoints\n\u2014 the three degrees for freedom\n,\n\u0003\u0001\n\n\u001a\u001d\u001c\nterm arises because we must consider all\n\f \u0015\b\u001e\n\nthe Inside-Outside algorithm.\nby assuming that sub-state\n\nnon-terminals in the grammar and the training sequence is of length\n\n\u0002\u0004\u0003\u000e\n\u0005\u0016\u0007\nfor all end points\n\n\t\nterm. The\n\ngenerates\n\nand\n\f\u0016\u0015\ngives rise to the\n\f\u0016\u0015\u001f\u001e\n.\n\noccurs, and that\n\nin state\n\nat time\n\n\u0017\u0019\u0018\u001b\u001a\n\f\u0016\u0015\n\ngenerates\n\ngenerates\n\n\u0007\u0006\b\u0003\t\n\n\u0010\b\u0012\u0013\u0012\u0013\u0012\b\u0010\n\n\u0007\u0006\b\n\t\n\n\u0003\u000e\u0003\u0001\n\n, where\n\n\u0005\u000b\u0007\n\n\"$#\n\ntransition to state\ntakes\n\n\f\u0016\u0015\ntime, where\n\n\u0002\u0004\u0003\u0014\n\nis the total number of states.\n\f \u0015\n\n\f\u0016\u0015\n\n\u0005\u0016\u0007\n\t\nWe can always \u201c\ufb02atten\u201d an HHMM into a regular HMM and hence do inference in\n. Unfortunately, this \ufb02at model cannot represent the hierarchical structure, yet\n\u0002\u0004\u0003\u0014\nalone learn it. In the next section, we show how to represent the HHMM as a DBN, and\nthereby get the best of both worlds: low time complexity without losing hierarchical struc-\nture.\n\n\u0005\u000b\t\n\n\f \u0015\n\n4 Representing the HHMM as a DBN\n\nWe can represent the HHMM as a dynamic Bayesian network (DBN) as shown in Figure 2.\n(We assume for simplicity that all production states are at the bottom of the hierarchy;\nthis restriction is lifted in the full version of this paper.) The state of the HMM at level\nand time\n\n! . The state of the whole HHMM is encoded by the vector\n\u000e&#\n\n; intuitively, this encodes the contents of the stack, that speci\ufb01es the\n\nis represented by\n\u0010\u0013\u0012\u0013\u0012\b\u0012(\u0010\n\ncomplete \u201cpath\u201d to take from the root to the leaf state.\n\n\u000e\u0010\u000f\n\nis an indicator variable that is \u201con\u201d if the HMM at level\n\nand time\n\n(i.e., is about to enter an end state), otherwise it is off. Note that if )\n\nhas just \u201c\ufb01nished\u201d\n\u000f+* , then )\n\u000f,*\n\n\t\n\u0004\n\u0005\n\u0012\n\u0004\n\u0005\n\n\u0005\n\u0019\n\u0002\n!\n\u001f\n!\n\f\n\t\n\t\n\u0002\n!\n\u001f\n!\n\f\n\t\n\u000f\n\u0002\n\u0002\n!\n\f\n\n\u0002\n!\n\f\n\t\n\n\u0005\n\u0007\n\t\n\n\u0005\n\u0019\n\u0002\n!\n\u001f\n!\n\t\n\u0017\n\u0002\n!\n\u001f\n!\n\u0002\n!\n\f\n\n\u001f\n!\n\u0017\n\u001a\n\n\u0007\n\n\u0003\n\u0001\n\u0017\n\u0019\n\u0002\n!\n\u001f\n!\n\t\n\"\n#\n\f\n\n\u0006\n\u0002\n!\n\u001f\n!\n\u001e\n%\n\"\n#\n\f\n\n\t\n\u0002\n!\n\u001e\n\f\n\n\u001f\n!\n\n\u0013\n\u001e\n\u0017\n'\n\u000e\n!\n\u000f\n\u0003\n\u000e\n\n!\n!\n\t\n)\n#\n!\n\u001e\n\u0017\n#\n!\n#\n\u001e\n!\n\f\u000e\u0014\n\n\u000e\u0010\u0007\n\n\u000e\u0014\n\n\u000e\u0010\u0007\n\n\u000e\u0010\u0007\n\n\u000f\u0003\u0002\n\nFigure 2: An HHMM represented as a DBN.\nHMM at level \u0011 has \ufb01nished (entered its exit state), otherwise\n\nif the\n. Shaded nodes are observed;\nthe remaining nodes are hidden. We may optionally clamp\n, where\nis the length of the\nobservation sequence, to ensure all models have \ufb01nished by the end of the sequence. (A similar trick\nwas used in [Zwe97].)\n\nis the state at time\n\n, level \u0011 ;\n\n\t\n\u0005\n\u0007\u0003\u000b\r\f\n\n\u000b\u0010\u000f\n\u000b\u0012\f\n\n\u0004\u0006\u0005\n\n\t\u000e\u0005\n\nfor all\nof the \u201ccontext stack\u201d, i.e., which level of the hierarchy we are currently on.\n\n; hence the number of )\n\nnodes that are \u201coff\u201d represents the effective height\n\n\u001c\u0015\u0014\n\nvariables represent the fact that a state \u201ccalls\u201d a\nThe downward going arcs between the\nvariables enforce the fact that a higher-\nsub-state. The upward going arcs between the )\nlevel HMM can only change state when the lower-level one is \ufb01nished. This ensures proper\nnesting of the parse trees, and is the key difference between an HHMM and a hidden\nMarkov decision tree [JGS96].\n\nWe will de\ufb01ne the conditional probability distributions (CPDs) of each of the node types\nbelow, which will complete the de\ufb01nition of the model. We consider the bottom, middle\nand top layers of the hierarchy separately (since they have different local topology), as well\nas the \ufb01rst, middle and last time slices.\n\n4.1 De\ufb01nition of the CPDs\n\nConsider the bottom level of the hierarchy.\nfollows a Markov chain with parameters\ndetermined by its position in the automaton, which is encoded by the vector of higher-up\nstate variables\nWhen\n\n, which we will represent by the integer\n\n.\n\n \u001f\n\n\u000f\n\u0016\n\n\u0010\u0013\u0012\u0013\u0012\b\u0012(\u0010\n\n\u000f\n\u0016\n\n\u0017 If the topology is sparse, this distribution will be 0 for many values of\n\n. This will be discussed\n\nin Section 4.2.\n\n)\n\n)\n\u0013\n\n)\n\u0007\n\n\u000e\n\u0013\n\n\u0013\n\u0013\n\u000e\n\u0013\n\u0013\n)\n\u0013\n\u0013\n)\n\n\u0013\n\u000e\n\n\u0007\n)\n\n\u0007\n)\n\u0013\n\u0007\n)\n\u0007\n\u0007\n\u0007\n\u000e\n\u0013\n\u0007\n)\n\u0007\n\u0013\n\u0002\n\n\u0002\n\u0013\n\u0002\n\u0007\n\u0017\n\u000f\n*\n\u0017\n\u000f\n\u0001\n\u0017\n\n\u0007\n\b\n\u0007\n\t\n\u0005\n\u0011\n\u0013\n\u001e\n\u001e\n\u000e\n\u000e\n\u000f\n\u000e\n\n!\n\u000f\n\u000e\n\n!\n\u000e\n\n!\n%\n\u0013\n\u0018\n\fenters its end state, it will \u201cturn on\u201d )\n\n, to mean it is \ufb01nished; this will be a signal\nthat higher-level HMMs can now change state. In addition, it will be a signal that the next\nvalue of\nshould be drawn from its prior distribution (representing a vertical transition),\ninstead of its transition matrix (representing a horizontal transition). Formally, we can write\nthis as follows:\n\n\u000f\u0001\n\nwhere we have assumed\u0002\n\nparent variables are in state\nthe initial distribution for level\nfor )\n\nis simply\n\n\u000f\u0003\u0002\n\r\f\n\n\u000f\u0003\u0004\nend.\t\n, and \u0007\n\n\u000f\n\u0016\n\n\u000f\u0006\u0005\b\u0007\nis just a rescaled version of\t\n\nis the transition matrix for level\n.\n\ngiven that the parent variables are in state\n\nif\u0004\n\u000f\u0003\n\nif\u0004\nSimilarly,\u000b\n\ngiven that the\nis\n. The equation\n\n \u001f\n\n\u000f\u000e\u0002\n\n\u0010 end\n\nfollows a Markov chain with param-\nNow consider the intermediate levels. As before,\nspeci\ufb01es whether we should use the transition matrix\neters determined by\n, specifying\nor the prior. The difference is that we now also get a signal from below, )\nwhether the sub-model has \ufb01nished or not; if it has, we are free to change state, otherwise\nwe must remain in the same state. Formally, we can write this as follows:\n\n, and )\n\n\u000e\u0014\r \u001f\n\n\u000f\u000f\n\n\u000f\u0003\u0002\n\n\u000f\u0010\u0004\n\n \u001f\n\n\u0014\u0003\u0015\n\n\u000f\u0012\u0011\n\nshould \u201cturn on\u201d only if\n\ndepends on the current context\n\nis \u201callowed\u201d to enter a \ufb01nal state, the probability of which\n\r \u001f\n\n. Formally, we can write this as follows:\n\n\u000f\u000e\n\n* and\u0004\n* and\u0004\n\nif\nif\nif\n\n\u000f\u000e\n\n\u000f\u000e\n\nif\nif\n\n\u0010 end\nnode has no\n\n\u000f\u000e\u0002\n\n\u000f\u0016\n\nparent to\nThe top level differs from the intermediate levels in that the\nspecify which distribution to use. The equations are the same as above, except we eliminate\n. (Equivalently, we can imagine a dummy top layer HMM,\nthe conditioning on\nwhich is always in state 1:\n* . This is often how HHMMs are represented, so that this\ntop-level state is the root of the overall parse tree, as in Figure 1.)\n\n \u001f\n\n\u000f\u0017\n\n \u001f\n\nThe CPDs for the nodes in the \ufb01rst slice are as follows:\nlevel and\n\u0010\b\u0012\u0013\u0012\u0013\u0012\u0013\u0010\n\n, for\n\n.\n\nfor the top\n\nas a multinomial\nIf the observations are discrete symbols, we may represent\n(i.e., using a table), or by using any of the more parsimonious representations discussed\nin Section 4.2. If the observations are real-valued vectors, we can use a Gaussian for each\nvalue of '\n\n! , or a mixture of a smaller number of Gaussians, as in [GJ97].\n\nto turn on, and then enters a new (non-terminal) state at time\n\ncan\n. Instead,\n. This means\nthat the DBN and HHMM transition matrices are not identical, but satisfy the following relation:\nrepresents\n\n\u0018 Unlike the automaton representation, the DBN never actually enters an end state (i.e.,\nnever taken on the value \u201cend\u201d), because if it did, it would not be able to emit the symbol\u0019\n\b\u001b\u001a\n\u0001 \u001f\"!$#\n\u0001 \u001f\"!$#\n\u0006&%\n\f('*)\nthe DBN transition matrix, and)\n\nis the probability of terminating from state\u001f .\n\nThe equations holds because the probability of each horizontal transition in the DBN gets multiplied\nby the probability that\nIt is easy to see that the new matrix is also stochastic, as required.\n\n\u0006 ; this product should match the original probability.\n\nrepresents the automaton transition matrix,\n\n, which is\n\ncauses\n\n\u0006\"+\n\n\u0001 \u001f\n\n\t\n\u0005\n\n\u0006,! where\u001d\n\u0001 \u001f\n\u0006.-0/$1\n\f2'3)\n\n\u0001 \u001f\"! end\u0006\n\u0001 \u001f\n\n\u000e\n\u000f\n\u000f\n\u000e\n\u000f\n\n\u0003\n\u000e\n\u000f\n!\n\u0019\n\u000e\n\u000f\n!\n\u0016\n\n\u0010\n)\n\u000f\n!\n\u0016\n\n\u0010\n\u000e\n\n\u001f\n\n!\n\u000f\n%\n\t\n\t\n\u000f\n\t\n\u0003\n\u0002\n\u0010\n\n\t\n\u000b\n\u000f\n\t\n\u0003\n\n\t\n\u000f\n*\n\u0010\n\u000f\n\u000f\n\t\n\u0012\n%\n\t\n\u000f\n\t\n\u000f\n\t\n\u0007\n\u000f\n\t\n\u0012\n%\n\u000f\n\n\u0003\n)\n\u000f\n!\n\u000f\n*\n\u0019\n\u000e\n\u000f\n\u0016\n\n!\n\u000f\n%\n\u0010\n\u000e\n\u000f\n!\n\t\n\u000f\n\t\n\u000f\n\t\n\u0003\n\u0002\n\t\n\u0012\n\u000e\n#\n#\n\u0016\n\n#\n#\n\f\n\n\u0003\n\u000e\n#\n!\n\u0019\n\u000e\n#\n!\n\u0016\n\n\u0010\n)\n#\n\f\n\n!\n\u0016\n\n\u000f\n\u0017\n\u0010\n)\n#\n!\n\u0016\n\n\u0010\n\u000e\n#\n\u0016\n\n!\n\u000f\n%\n\t\n\u0013\n\u0003\n\u0002\n\u0010\n\n\t\n\u0017\n\u0007\n\t\n#\n\t\n\u0003\n\u0002\n\u0010\n\n\t\n\u0017\n\u000f\n\u000b\n#\n\t\n\u0003\n\n\t\n\u0017\n\u000f\n\u000f\n*\n)\n#\n\u000e\n#\n\u000e\n#\n\u0016\n\n\u0003\n)\n#\n!\n\u000f\n*\n\u0019\n\u000e\n#\n!\n\u0010\n\u000e\n\n\u001f\n#\n\u0016\n\n!\n\u000f\n%\n\u0010\n)\n#\n\f\n\n!\n\u000f\n\u0017\n\t\n\u000f\n\u0005\n\n\u0017\n\t\n#\n\t\n\u0003\n\u0002\n\t\n\u0017\n\u000f\n*\n\u000e\n\u000e\n\u000e\n#\n\u0016\n\n!\n\u000f\n%\n\u000e\n\u000b\n!\n\u000f\n\n\u0003\n\u000e\n\n\t\n\u000f\n\u000b\n\n\u0003\n\n\t\n\n\u0003\n\u000e\n#\n\n\u0019\n\u000e\n#\n\u0016\n\n\u000f\n%\n\t\n\u000f\n\u000b\n#\n\t\n\u0003\n\n\t\n\u001e\n\u000f\n\u0001\n\u0012\n\n\u0003\n\u0002\n!\n\u0019\n'\n\u000e\n!\n\t\n\u000e\n\u0004\n\u0005\n\u0007\n\u0007\n\u0004\n\u0005\n\u0007\n\t\n\u0005\n\u0007\n\f\n\u001c\n\u001d\n\u0005\n\u001e\n\u0005\n\u001e\n\u000b\n\u001d\n\u0005\n\u001e\n\u001c\n\u001d\n\u0005\n\u001e\n\u000b\n\u001d\n\u0005\n\u001e\n\u0007\n\u000b\n\u000f\n\u0005\n\u001e\n\f4.2 Parsimonious representations of the CPDs\n\n\u0002\u0004\u0003\n\nas a multinomial\nThe number of parameters needed to represent\nis\n. If the state-transition diagram of the hierarchical automaton is sparse, many of\nthe entries in this table will be 0. However, when we are learning a model, we do not know\nthe structure of the state-transition diagram, and must therefore adopt a representation with\nfewer parameters. There are at least three possibilities: decision trees [BFGK96], softmax\nnodes, or representing\nas a mixture of smaller transition matrices\nat different depths c.f. [SJ99]. See the full version of this paper for details.\n\n\u000e&#\n\n \u001f\n\n\u000e&#\n\n \u001f\n\n5 Linear-time inference\n\n\u0003\u0001\n\n \u001f\n\nor\n\n! , with \u0005\n\n\u0001\u0004\u0003 parents\n\u000f\u0011\u000e\n\nWe de\ufb01ne inference to be computing\nin the DBN. These \u201cfamily\u201d marginals are needed by EM. The simplest way to do this is to\n\nfor all sets of nodes \nmerge all the hidden nodes in each slice into a single \u201cmega node\u201d, \u0005\nterm arises from the binary )\npossible values. (The \u0001\nforwards-backwards algorithm for HMMs, which takes\n\u0005\u000b\t\n\nnodes.) We can then apply the\n\u0002\u0004\u0003\u0001\u0005\n\n. (Even storing the \u0005\u0007\u0006\b\u0005\n\nUnfortunately, converting the DBN to an HMM in this way will not be tractable for reason-\nably large\ntransition matrix is likely to consume too much\nspace.) Fortunately, we can do better by exploiting the structure of the model. In [Mur01],\nwe present a way of applying the junction tree (jtree) algorithm to variable-length DBNs;\nwe give a brief sketch here. The algorithm works by performing a forwards-backwards\n-slice DBN\u201d; this is a\nsweep through a chain of jtrees. Each jtree is formed from a \u201c *\nDBN that contains all the nodes in slice 1 but only the interface nodes from slice 2. The\ninterface nodes are those nodes in slice 2 that have an incoming temporal arc, plus parents\nof nodes that have incoming temporal arcs. In the case of an HHMM, the interface is all\nthe\n\nnodes.\n\ntime.\n\nThe cost of doing inference in each jtree depends on the sizes of the cliques. Minimizing\nthe maximal clique size is NP-hard, so we used a standard one-step look-ahead (greedy)\nalgorithm [Kja90]. The resulting cliques are hard to interpret, but we can still analyze the\ncomplexity. Let\nbe the\n\n\t\u0016\u0003\n\u0012\u0017\t\nnumber of ) nodes, and let\nbe the number of cliques. Then the cost of inference in\na jtree is proportional to\n\nbe the number of\n\u000e\n\nnodes in clique\n\n, let\n\n\f\u000b\n\n\u0012\u0017\t\n\n\u0003\u0015\u0012\u0017\t\n\n\u000f\u0011\u0010\u0013\u0012\n\r\u0018\u0017\n\n\u0010)\u000f\u001e\u0019*\u0012\n\n\u001c\u001b\n\n\u000f\u0015\u0014\n\n\u000f\u0015\u0014\n\n\u000f\u0011\u0019\u001a\u0012\n\n\u001c\u001b\n\n\u000f\u0015\u0014\n\n\u000f\u001e\u001d\u001f\u0012\n\n\u001c\u001b\n\n\u000f\u0015\u0014! \n\n\u0003\u0015\u0012\n\nEmpirically we \ufb01nd that, for a wide range of\n\n\n\u000b\n\u0012\u0017\t\"0\n. We remind readers that the original algorithm has\n\n\t\"\u0006\u0019\u000e$#&%('\n\u000e\r\n\u0012\u0017\t10\n\u0012\u000e4 . Hence a crude upper bound on the cost of in-\n, yielding an overall time and space com-\n\u000b\n\u001b\n\u0002\u0004\u0003\u0006\u0005\b\u0007\n\u000e\u0010\u000f\u0011\t\nstates in the HHMM. The advantage of\n\n\u0012\u000e4 and ,5-/.\u0004\r!6\t\u0016\u0003\n\u0002\u0004\u0003\u0015\u0005\u0016\u0012\u0017\u000e\u0019\u0018\u001a\n\n\u001273\nference in each jtree is\n\u000f \u001f\nplexity of\ntime complexity, since there can be up to\nthe new algorithm in practice is clearly illustrated in Figure 3.\n\n\u0002\u0004\u0003\u001e\u0003\u0015\u0012\n\u000f \u001f\n\n\u000f \u001f\n\n\u0003\u0015\u0012\n\n\u0018\u001a\n\n,\n\n\u001c\u001b\n\n\u000f\u0015\u0014\n\n\u0010)\u000f\u001e\u001d+\u0012\n#&%('\n\u0001 , ,\u000e-/.\u0004\n\nWe can reduce the time (and space) complexity from\nby\nusing approximate DBN inference techniques such as the \u201cfactored frontier (FF) algorithm\u201d\n[MW01], which is equivalent to applying \u201cloopy belief propagation\u201d to the DBN using a\nleft-right scheduling of the messages. (It is still exponential in\nbecause of the high fan-in\nof the nodes.) We can get a further speedup by using a mixture representation of the CPDs\n\n\u0002\u0004\u0003\u0015\u0005\u0016\u0012\u0017\u000e\n\n\u0002\u0004\u0003\u0015\u0005\u0016\u0012\u0017\u000e\n\n\u001d\u001e\u000f \u001f\n\n\u000f\u0011\t\n\n\u0018\u001a\r\u001c\u001b\n\nto\n\n\u0003\n\u000e\n#\n!\n\u0019\n!\n\u0016\n\n\u0010\n\u000e\n#\n\u0016\n\n!\n\u000f\n%\n\t\n\u000e\n#\n\f\n\n\t\n\n\u0003\n!\n\u0019\n\u000e\n#\n!\n\u0016\n\n\u0010\n\u000e\n#\n\u0016\n\n!\n\u000f\n%\n\t\n\n\u0001\n\u0019\n\u0002\n\u0002\n\t\n\u0001\n\u000f\n\n\u0002\n\u0003\n\u0002\n\t\n\u000f\n\u0001\n\u000f\n\u000f\n\u0013\n\u000e\n\u0012\n\n\u0013\n\u000e\n\u001b\n\u0010\n\u000e\n\u001b\n\u0003\n\u001b\n\u0010\n\u0016\n\n\u000e\n\u0006\n\u0001\n\n\n\u0006\n\u0001\n\u0012\n\t\n \n\u0012\n\u0018\n\u0003\n\u001b\n\u0010\n2\n\n\u0012\n3\n\u001b\n\u0010\n*\n\u0018\n2\n*\n\u0018\n\u0001\n\t\n\u000e\n\u001b\n\u001d\n\u0001\n\u0018\n\u001d\n\t\n\u001b\n\u001d\n\t\n\n\u000f\n\u000e\n\u000f\n\t\n\u0012\n\frunning time (seconds) vs sequence length\n\nlinear\ncubic \n\n15\n\n20\n\n25\n\n30\n\n35\n\n40\n\n40\n35\n30\n25\n20\n15\n10\n5\n0\n10\n\nFigure 3: Running time vs. sequence length. Both algorithms were implemented in Matlab. The\nHHMM has\n\n.\n\n,\n\n\u000b\u0002\u0001\n\n\u000b\u0004\u0003\n\n(see Section 4.2). In this case, we can exploit the form of the CPD to compute the required\nmessages ef\ufb01ciently [Mur99], bringing the overall complexity down to\n\n.\n\nWe remark that all of the above algorithms can also be used for online \ufb01ltering. In addition,\nby replacing the sum operator with max, we can do Viterbi segmentation in the usual way.\n\n\u0002\u0004\u0003\u0006\u0005\u000b\u0012\n\n6 Related work\n\nHidden Markov decision trees (HMDT) [JGS96] are DBNs with a structure similar to Fig-\nure 2, but they lack the )\nnodes and the upward going arcs; hence they are not able to\nrepresent the call-return semantics of the HHMM. Embedded HMMs [NI00] are a special\ncase of HHMMs in which the ending \u201ctime\u201d of the sub-HMMs is known in advance (e.g.,\nthe sub-HMM models exactly one row of pixels). ([Hoe01] calls these models \u201chierarchi-\ncal mixture of Markov chains\u201d.) A variable-duration HMM [Rab89] is a special case of a\n2-level HHMM, where the bottom level counts how long we have been in a certain state;\nwhen the counter expires, the ) node turns on, and the parent can change state.\n[BVW00] describes the \u201cAbstract HMM\u201d (AHMM), which is very closely related to HH-\nMMs. These authors are interested in inferring what abstract policy an agent is following\nby observing its effects in the world. An AHMM is equivalent to an HHMM if we consider\nrepresents\nthe concrete action, which causes the observation. We also need to add a hidden global\nnodes.\nis hidden to us as observers, but not to the agent performing the actions.) [BVW00]\nconsider abstract policies of the \u201coptions\u201d kind [SPS99], which is equivalent to assuming\nthat there are no horizontal transitions. (HAMs [PR97] generalize this by allowing hori-\nzontal transitions (i.e., internal state) within a controller.) In addition, they assume that\nonly depends on its immediate parent,\nnodes become connected by a chain. This enables them to use Rao-Blackwellized particle\n\ufb01ltering for approximate online inference: conditioned on the )\nnodes, the distribution\nover the\nnodes can be represented as a product of marginals, so they can be ef\ufb01ciently\nmarginalized out.\n\nto represent the (abstract) policy being followed at level\n\n! , which is a parent of the\n\nstate variable \n(\n\n! nodes and all the\n\n! node, all the )\n\n, but not its whole context,\n\nand time\n\n;\n\n, so the\n\n\n\u0004\n\u0013\n\u000e\n\u0013\n\t\n\u000e\n#\n!\n\u001e\n\u0017\n\u000e\n\u000f\n!\n\u0002\n\u000e\n!\n\f\n\n!\n\u000e\n#\n!\n\u000e\n#\n\u0016\n\n!\n\u000e\n\n\u001f\n#\n\u0016\n\n!\n\u000e\n\u000e\n\fAcknowledgements\n\nI would like to thank Dr Christopher Schlick for giving me his Matlab implementation of\nthe\n\nalgorithm, which was used to create part of Figure 3.\n\n\u0002\u0004\u0003\u0015\u0005\u0016\u0007\n\t\n\nReferences\n\n[BFGK96] C. Boutilier, N. Friedman, M. Goldszmidt, and D. Koller. Context-Speci\ufb01c Independence in Bayesian Networks. In\n\nUAI, 1996.\n\n[BVW00] H. Bui, S. Venkatesh, and G. West. On the recognition of abstract Markov policies. In AAAI, 2000.\n\n[BVW01] H. Bui, S. Venkatesh, and G. West. Tracking and surveillance in wide-area spatial environments using the Abstract\n\nHidden Markov Model. Intl. J. of Pattern Rec. and AI, 2001.\n\n[FST98]\n\nShai Fine, Yoram Singer, and Naftali Tishby. The hierarchical Hidden Markov Model: Analysis and applications.\nMachine Learning, 32:41, 1998.\n\n[GJ97]\n\nZ. Ghahramani and M. Jordan. Factorial hidden Markov models. Machine Learning, 29:245\u2013273, 1997.\n\n[HIM\n\n00] M. Hu, C. Ingram, M.Sirski, C. Pal, S. Swamy, and C. Patten. A Hierarchical HMM Implementation for Vertebrate\n\nGene Splice Site Prediction. Technical report, Dept. Computer Science, Univ. Waterloo, 2000.\n\n[Hoe01]\n\nJ. Hoey. Hierarchical unsupervised learning of facial expression categories. In ICCV Workshop on Detection and\nRecognition of Events in Video, 2001.\n\n[IB00]\n\nY. Ivanov and A. Bobick. Recognition of visual activities and interactions by stochastic parsing. IEEE Trans. on\nPattern Analysis and Machine Intelligence, 22(8):852\u2013872, 2000.\n\n[JGS96] M. I. Jordan, Z. Ghahramani, and L. K. Saul. Hidden Markov decision trees. In NIPS, 1996.\n\n[Kja90]\n\n[LY90]\n\n[ME01]\n\nU. Kjaerulff. Triangulation of graphs \u2013 algorithms giving small total state space. Technical Report R-90-09, Dept. of\nMath. and Comp. Sci., Aalborg Univ., Denmark, 1990.\n\nK. Lari and S. J. Young. The estimation of stochastic context-free grammars using the Inside-Outside algorithm.\nComputer Speech and Language, 4:35\u201356, 1990.\n\nD. Moore and I. Essa. Recognizing multitasked activities using stochastic context-free grammar. In CVPR Workshop\non Models vs Exemplars in Computer Vision, 2001.\n\n[Mur99]\n\nK. Murphy. Pearl\u2019s algorithm and multiplexer nodes. Technical report, U.C. Berkeley, Dept. Comp. Sci., 1999.\n\n[Mur01]\n\nK. Murphy. Applying the junction tree algorithm to variable-length DBNs. Technical report, Comp. Sci. Div., UC\nBerkeley, 2001.\n\n[MW01]\n\nK. Murphy and Y. Weiss. The Factored Frontier Algorithm for Approximate Inference in DBNs. In UAI, 2001.\n\n[NI00]\n\nA. Ne\ufb01an and M. Hayes III. Maximum likelihood training of the embedded HMM for face detection and recognition.\nIn IEEE Intl. Conf. on Image Processing, 2000.\n\n[PR97]\n\nR. Parr and S. Russell. Reinforcement learning with hierarchies of machines. In NIPS, 1997.\n\n[Rab89]\n\nL. R. Rabiner. A tutorial on Hidden Markov Models and selected applications in speech recognition. Proc. of the\nIEEE, 77(2):257\u2013286, 1989.\n\n[SJ99]\n\nL. Saul and M. Jordan. Mixed memory markov models: Decomposing complex stochastic processes as mixture of\nsimpler ones. Machine Learning, 37(1):75\u201387, 1999.\n\n[SPS99]\n\nR.S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in\nreinforcement learning. Arti\ufb01cial Intelligence, 112:181\u2013211, 1999.\n\n[TRM01] G. Theocharous, K. Rohanimanesh, and S. Mahadevan. Learning Hierarchical Partially Observed Markov Decision\n\nProcess Models for Robot Navigation. In ICRA, 2001.\n\n[Zwe97]\n\nG. Zweig. Speech Recognition with Dynamic Bayesian Networks. PhD thesis, U.C. Berkeley, Dept. Comp. Sci.,\n1997.\n\n\n\f", "award": [], "sourceid": 2050, "authors": [{"given_name": "Kevin", "family_name": "Murphy", "institution": null}, {"given_name": "Mark", "family_name": "Paskin", "institution": null}]}