{"title": "Graph Matching with Hierarchical Discrete Relaxation", "book": "Advances in Neural Information Processing Systems", "page_first": 689, "page_last": 695, "abstract": "", "full_text": "Graph Matching with Hierarchical \n\nDiscrete Relaxation \n\nRichard C.  Wilson and Edwin R.  Hancock \n\nDepartment of Computer Science,  University of York \n\nYork,  YOl  5DD,  UK. \n\nAbstract \n\nOur aim in this paper is to develop a Bayesian framework for match(cid:173)\ning hierarchical relational models.  The goal is  to make discrete la(cid:173)\nbel assignments so as to optimise a  global cost function that draws \ninformation concerning the consistency of match from different lev(cid:173)\nels  of the  hierarchy.  Our  Bayesian  development  naturally  distin(cid:173)\nguishes  between intra-level and inter-level constraints.  This allows \nthe  impact  of reassigning  a  match  to  be  assessed  not  only  at  its \nown (or peer) level ofrepresentation, but also upon its parents and \nchildren in  the hierarchy. \n\nIntrod uction \n\n1 \nHierarchical graphical structures are of critical importance in the interpretation of \nsensory or perceptual data.  For instance, following  the influential work of Marr [6] \nthere has been sustained efforts at effectively organising and processing hierarchical \ninformation in vision systems.  There are a  plethora of concrete examples which in(cid:173)\nclude pyramidal hierarchies [3]  that are concerned with multi-resolution information \nprocessing  and  conceptual  hierarchies  [4]  which  are  concerned  with  processing  at \ndifferent levels of abstraction.  Key to the development of techniques for hierarchical \ninformation  processing is  the  desire  to  exploit  not  only  the  intra-level constraints \napplying  at  the  individual  levels  of representation  but  also  inter-level  constraints \noperating between  different  levels  of the  hierarchy.  If used  effectively  these  inter(cid:173)\nlevel  constraints can  be  brought  to bear on  the  interpretation of uncertain  image \nentities in such a  way as to improve the fidelity of interpretation achieved by single \nlevel means.  Viewed as an additional information source, inter-level constraints can \nbe  used  to  resolve  ambiguities  that  would  persist  if  single-level  constraints  alone \nwere  used. \n\n\f690 \n\nR.  C.  Wilson and E.  R.  Hancock \n\nIn the connectionist literature graphical structures have been widely  used to repre(cid:173)\nsent  probabilistic causation in  hierarchical  systems  [5,  9].  Although  this literature \nhas provided a powerful battery of techniques, they have proved to be of limited use \nin practical sensory processing systems.  The main reason for  this is  that the under(cid:173)\npinning independence assumptions and the resulting restrictions on graph topology \nare rarely realised  in  practice.  In particular there are severe technical problems  in \ndealing with structures that contain loops or are not tree-like.  One way to overcome \nthis difficulty  is  to edit intractable structures to produce tractable ones  [8]. \n\nOur aim in  this  paper is  to extend this discrete relaxation framework  to hierarchi(cid:173)\ncal graphical structures.  We  develop a  label-error process to model the violation of \nboth inter-level and intra-level constraints.  These two  sets of constraints have  dis(cid:173)\ntinct probability distributions.  Because we  are concerned with  directly comparing \nthe  topology  graphical  structures  rather  than  propagating  causation,  the  result(cid:173)\ning  framework  is  not  restricted  by  the  topology  of the  hierarchy.  In ,particular, \nwe  illustrate  the  effectiveness  of the  method  on  amoral  graphs  used  to  represent \nscene-structure in an image interpretation problem.  This is  a  heterogeneous struc(cid:173)\nture  [2,  4]  in which  different  label  types  and different  classes of constraint operate \nat  different  levels  of abstraction.  This  is  to  be contrasted with  the  more  familiar \npyramidal  hierarchy  which  is  effectively  homogeneous  [1,  3].  Since  we  are  deal(cid:173)\ning  with  discrete  entities  inter-level  information  communication  is  via  a  symbolic \ninterpretation of the objects under consideration. \n2  Hierarchical  Consistency \nThe  hierarchy  consists  of  a  number  of levels,  each  containing  objects  which  are \nfully  described by their children at the level below.  Formally each level is  described \nby  an  attributed  relational  graph  GI  =  (Vi, EI, Xl),  Vi  E  L,  with  L  being  the \nindex-set  of  levels  in  the  hierarchy;  the  indices  t  and  b  are  used  to  denote  the \ntop and bottom levels  of the  hierarchy respectively.  According to our notation for \nlevel  i  of the  hierarchy,  Vi  is  the  set  of  nodes,  EI  is  the  set  of  intra-level  edges \nand  Xl  =  {~~, Vu  E  Vi}  is  a  set  of unary  attributes  residing  on  the  nodes.  The \nchildren  or  descendents  which  form  the representation  of an element  j  at  a  lower \nlevel are denoted by V j .  In other words, if U l - I  is  in Vj  then there is  a  link in  the \nhierarchy  between  element  j  at level  i  and element  u  at level  i-I. According  to \nour assumptions, the elements of Vj are drawn exclusively from  Vi-I.  The goal of \nperforming relaxation operations is  to find  the match between scene graph G1  and \nmodel graph G2 \u2022  At each individual level of the hierarchy this match is  represented \nby a  mapping function p, Vi  E L,  where II:  Vi  -t Vi. \nThe  development  of  a  hierarchical  consistency  measure  proceeds  along  a  similar \nline to the Single-level work of Wilson and Hancock [10].  The quantity of interest is \nthe MAP estimate for  the mapping function I  given the available unary attributes, \ni.e.  I  =  argmaxj P(jt, Vi  E  LIXI , Vi  E  L).  We  factorize  the  measurement  infor(cid:173)\nmation over the set of nodes  by application of Bayes rule under  the assumption of \nmeasurement independence on the nodes.  As  a  result \n\nP(/, Vi  E L/Xl , Vi  E L) = \n\n(Xl  ~i E L) {II  II p(X~ll(u))}P(fI, Vi  E L)  (1) \n\np ,  \n\nIELuEVI \n\nThe critical modelling ingredient in developing a discrete relaxation scheme from the \nabove MAP criterion is  the joint prior for  the mapping function,  i.e.  p(fl, Vi  E L) \n\n\fGraph Matching with Hierarchical Discrete Relaxation \n\n691 \n\nParents \n\nChildren \n\nA \n\nB \n\nc \n\nPossible mappings or children: \n\n1,2,3  -- A,B,C \nC,B,A \n\nFigure 1:  Example constrained children mappings \n\nwhich  represents  the  influence  of structural information on the  matching  process. \nThe joint measurement density, p(XI, 'VI  E L), on the other hand is  a fixed  property \nof the hierarchy and can be eliminated from  further consideration. \n\nRaw  perceptual  information  resides  on  the  lowest  level  of the hierarchy.  Our task \nis  to propagate this  information upwards through the hierarchy.  To  commence our \ndevelopment, we  assume that individual levels  are conditionally dependent only on \nthe immediately  adjacent  descendant  and  ancestor  levels.  This  assumption  allows \nthe factorisation of the joint probability in  a  manner analogous to a  Markov chain \n[3].  Since  we  wish  to  draw  information  from  the  bottom  upwards,  the  factorisa(cid:173)\ntion  commences  from  the  highest  level  of labelling.  The  expression  for  the  joint \nprobability of the hierarchical labelling is \n\np(fl, 'VI  E L) =  p(fb)  II  P(fl+Ill) \n\nIEL,I#t \n\n(2) \n\nWe  can now focus  our attention on the conditional probabilities P(fI+1lfl).  These \nquantities express the probability of a labelling at the level I + 1 given the previously \ndefined  labelling  at  the  descendant  level  l.  We  develop  tractable  expressions  for \nthese probabilities by decomposing the hierarchical graph into convenient structural \nunits.  Here  we  build  on  the  idea  of  decomposing  Single-level  graphs  into  super(cid:173)\ncliques that was  successfully exploited in our previous work  [10].  Super-cliques are \nthe sets of nodes connected to a  centre-object by intra-level edges.  However, in the \nhierarchical case the relational units are more complex since we  must also consider \nthe graph-structure conveyed by  inter-level edges. \n\nWe follow the philosophy adopted in the single-level case [10]  by averaging the super(cid:173)\nclique  probabilities to estimate  the conditional matching probabilities  P(fI+1lfl). \nIf r~  C  fl  denotes  the  current  match  of the  super-clique  centred  on  the  object \nj  E ~l then we  write \n\nP(f'lfl-I) =  ~l  L  p(r~lfl-l) \n\nI  I jEV' \n\n(3) \n\nIn order to model this probability, we  require a dictionary of constraint relations for \nthe corresponding graph  sub-units  (super-cliques)  from  the model graph G2 \u2022  The \nallowed mappings  between the model graph and the data graph which preserve the \ntopology of the graph structure at a  particular level  of representation are referred \n\n\f692 \n\nR.  C.  Wilson and E  R. Hancock \n\nto  as  \"structure preserving  mappings\"  or  SPM's.  It  is  important to  note that  we \nneed  only  explore  those  mappings  which  are  topologically  identical  to  the  super(cid:173)\nclique centred on object j  and therefore the possible mappings of the child nodes are \nheavily constrained by the mappings of their parents (Figure 1).  We  denote the set \nof SPM's by P.  Since the set P  is  effectively  the state-space of legal matching,  we \ncan apply  the  Bayes  theorem  to compute  the  conditional  super-clique  probability \nin  the following  manner \n\np(r~I/I-l) =  2: p(r~15,/I-l)P(5Ii-l) \n\n(4) \n\nSEP \n\nAccording to this expression, there are two distinct components to our model.  The \nfirst  involves  the  comparison  between  our  mapped  realisation  of the  super-clique \nfrom  graph  G1 ,  i.e.  q, with  the  selected  unit  from  graph  G 2  and  the  mapping \nfrom level 1 - 1.  Here we  take the view that once we  have hypothesised a  particular \nmapping 5  from P, the mapping P-l provides us  with no further information, i.e. \np(r~ 15, /1-1)  =  p(r~ 15).  The matched super-clique r~ is conditionally independent \ngiven a mapping from the set of SPM's and we  may write the first term as p(r~15). \nIn other words, this first  conditional probability models only intra-level constraints. \nThe second term is the significant one in evaluating the impact inter-level constraints \non the labelling at the previous level.  In this term the probability of the hypothesised \nmapping 5  is  conditioned according to the match of the child levell. \n\nAll  that  remains  now  is  to  evaluate  the  conditional  probabilities.  Under  the  as(cid:173)\nsumption  of  memoryless  matching  errors,  the  first  term  may  be  factorised  over \nthe  marginal  probabilities  for  the  assigned  matches  lIon the  individual  nodes  of \nthe matched super-clique q given  their counterparts Si  belonging to the structure \npreserving mapping 5.  In other words, \n\np(r;15) = II P('~lsi) \n\n1'! Ef~ \n\n(5) \n\nIn  order to  proceed  we  need  to  specify  a  probability  distribution  for  the  different \nmatching  possibilities.  There  are  three  cases.  Firstly,  the  match  Ii  may  be  to  a \ndummy-node  d  inserted into q to  raise it to the same size  as  5  so  as  to facilitate \ncomparison.  This  process  effectively  models  structural  errors  in  the  data-graph. \nThe  second  and  third  cases,  relate  to  whether  the  match  is  correct  or  in  error. \nAssuming that dummy  node insertions may  be  made with probability Ps  and that \nmatching  errors occur  with  probability  Pe ,  then  we  can  write  down  the following \ndistribution rule \n\nif Ii  =  d  or Si  =  d \n'f  1 \n1  Ii  =  Si \notherwise \n\n(6) \n\nThe second term in Equation (5)  is  more subtle; it represents the conditional prob(cid:173)\nability  of the  SPM  5  given  a  previously  determined  labelling  at  the  level  below. \nHowever, the mapping contains labels only from  the current levell, not labels from \nlevel  I - 1.  We  can reconcile  this  difference  by noting that selection of a  particular \nmapping at level I  limits  the number of consistent mappings  allowed  topologically \nat  the  level  below.  In  other  words  if  one  node  is  mapped  to  another  at  level  I, \n\n\fGraph Matching with Hierarchical Discrete Relaxation \n\n693 \n\nthe  consistent  interpretation  is  that the  children of the  nodes  must  match to each \nother.  Provided that  a  set  of mappings  is  available  for  the  child-nodes,  then  this \nconstraint can be used to model P(SljI-1).  The required child-node mappings are \nreferred  to as  \"Hierarchy Preserving Mappings\"  or HPM's .  It is  these  hierarchical \nmappings that lift  the requirements for  moralization in our matching scheme, since \nthey effectively encode potentially incestuous vertical relations.  We  will  denote the \nset of HPM's for  the descendants of the SPM S  as  Qs  and a  member of this set as \nQ = {qi, 'Vi  E Vj}.  Using this model the conditional probability P(SIfI-l) is  given \nby \n\np(SIfI-1) =  L  P(SIQ,/I-1)P(Qll- 1 ) \n\nQEQs \n\n(7) \n\nFollowing our modelling  of the  intra-level  probabilities,  in  this  inter-level  case  as(cid:173)\nsume  that  S  is  conditionally  independent  of 11- 1 given  Q,  i.e.  P(SIQ, / 1- 1)  = \nP(SIQ)\u00b7 \n\nTraditionally,  dictionary  based  hierarchical  schemes  have  operated  by  using  a  la(cid:173)\nbelling  determined  at  a  preceding level  to  prune  the dictionary  set  by  elimination \nof vertically inconsistent  items  [4].  This  approach can  easily  be  incorporated into \nour scheme by setting P(QI/I-l) equal to unity for  consistent items and to zero for \nthose  which  are inconsistent.  However  we  propose a  different  approach;  by  adopt(cid:173)\ning the same kind of label distribution used in  Equation 6 we  can grade the SPM's \naccording to their  consistency with the match at level 1 - 1,  i.e.  jI-l.  The model \nis  developed by factorising over the child  nodes qi  E Q  in the following  manner \n\nP(Qll- 1 )  =  II P(qih,!-1) \n\nqiEQ \n\n(8) \n\nThe  conditional  probabilities  are  assigned  by  a  re-application  of the  distribution \nrule given in  Equation  (6),  i.e. \n\nif dummy  node match \n'f \n1  qi  =  Ii \notherwise \n\n1-1 \n\n(9) \n\nFor  the conditional  probability of the  SPM given  the  HPM  Q,  we  adopt  a  simple \nuniform  model  under  the  assumption  that all  legitimate  mappings  are equivalent, \ni.e.  P(SIQ) = P(S) = I~I' \nThe  various  simplifications  can  be  assembled  along  the  lines  outlined  in  [10]  to \ndevelop  a  discrete  update  rule  for  matching  the  two  hierarchical  structures.  The \nMAP  update  decision  depends  only on  the  label  configurations  residing  on  levels \n1 - 1,  1 and 1 + 1 together with  the measurements residing on  levell.  Specifically, \nthe level  1 matching configuration satisfies the condition \n\nII = argm!F{ II p(~~ljt(j)) }P(fI-llil )P(PI11+1 ) \n\nf \n\njEV! \n\n(10) \n\nHere consistency of match  between  levels  land 1 - 1 of the hierarchy is  gauged  by \n\n\f694 \n\nR.  C.  Wilson and E.  R. Hancock \n\nthe quantity \n\nPUl-llfl) = :1  L  L  K(rD \n\n1  iEVI SEP  Qs \n\n1 \n\nexp \n\n[-(keH(rL S) + ks~(rL S))] \n\nL  K(r!-l)  exp \n\n[ - (keH (r~-l ,Q) + ks ~(r~-l, Q)) }11) \n\nQEQs \n\nIn the above expression H (r j, S)  is the \"Hamming distance\"  which counts the num(cid:173)\nber of label conflicts  between  the assigned match rj and the structure preserving \nmapping  S.  This  quantity  measures  the  consistency  of the  matched  labels.  The \nnumber of dummy nodes inserted into r j  by the mapping S is  denoted by ~ (r j, S). \nThis second  quantity measures the structural compatibility of the two hierarchical \ngraphs.  The  exponential  constants  ke  = In  (l-PeMI-P,)  and  ks  = In  1Ft,  are  re(cid:173)\nlated  to  the  probabilities  of structural  errors  and  mis-assignment  errors.  Finally, \nK(rj) = (1- Pe ){1- Pe)lrjl  is  a  normalisation constant.  Finally, it is  worth point(cid:173)\ning out that the discrete relaxation scheme of Equation  (10)  can be applied at any \nlevel  in  the  hierarchy.  In other words  the  process  can be operated in  top-down  or \nbottom-up modes  if required. \n3  Matching SAR Data \nIn  our experimental  evaluation  of the  discrete  relaxation  scheme  we  will  focus  on \nthe  matching of perceptual  groupings  of line-segments  in  radar images.  Here  the \nmodel graph is elicited from a digital map for the same area as the radar image.  The \nline  tokens  extracted from  the  radar data correspond  to  hedges  in  the  landscape. \nThese are  mapped as  quadrilateral field  boundaries in  the cartographic model.  To \nsupport this application, we  develop a  hierarchical matching scheme based on line(cid:173)\nsegments  and  corner  groupings.  The  method  used  to  extract  these  features  from \nthe  radar  images  is  explained  in  detail  in  [10].  Straight  line  segments  extracted \nfrom  intensity  ridges  are  organised  into  corner  groupings.  The  intra-level  graph \nis  a  constrained  Delaunay  triangulation  of the  line-segments.  Inter-level  relations \nrepresent  the subsumption of the bottom-level line  segments into corners. \n\nThe raw image data used in  this study is  shown in  Figure 2a.  The extracted line(cid:173)\nsegments  are shown  in  Figure  2c.  The map  used  for  matching is  shown  in  Figure \n2b.  The experimental  matching  study is  based  on  95  linear  segments  in  the  SAR \ndata and 30 segments contained in  the map.  However only 23 of the SAR segments \nhave feasible  matches within the map representation.  Figure 2c  shows  the matches \nobtained by non-hierarchical means.  The lines  are coded as  follows;  the black lines \nare correct matches while the grey lines are matching errors.  With the same coding \nscheme Figure 2d.  shows the result obtained using the hierarchical method outlined \nin this paper.  Comparing Figures  2c  and 2d it is  clear that the hierarchical method \nhas  been  effective  at grouping  significant  line  structure  and excluding clutter.  To \ngive  some  idea of relative  performance  merit,  in  the  case  of the  non-hierarchical \nmethod,  20  of the 23  matchable segments are correctly identified  with 75  incorrect \nmatches.  Application of the hierarchical method gives  19  correct matches, only  17 \nresidual clutter segments with 59  nodes correctly labelled as  clutter. \n4  Concl usions \nWe  have developed graph matching technique which is tailored to hierarchical rela(cid:173)\ntional descriptions.  The key  element  is  this development  is  to quantify  the match-\n\n\fGraph Matching with Hierarchical Discrete Relaxation \n\n695 \n\na) \n\nc) \n\n, \n\\ \n\" \n\n~\\ ' \\  \n\" \n\nb) \n\nd) \n\n-\n\n' \n\n\\ \n\"\\ \n\n-\n/\\{-\\'\\(\" \n\nFigure  2:  Graph  editing:  a)  Original  image,  b)  Digital  map,  c)  Non  hierarchical \nmatch, d)  Hierarchical match. \n\ning  consistency  using  the  concept  of hierarchy  preserving  mappings  between  two \ngraphs.  Central to the development of this novel technique is the idea of computing \nthe probability of a  particular node match by drawing on the topologically allowed \nmappings of the child nodes in the hierarchy.  Results on image data with lines and \ncorners as graph nodes reveal that the technique is capable of matching perceptual \ngroupings under moderate levels of corruption. \n\nReferences \n\n[1]  F.  Cohen and D.  Cooper.  Simple Parallel  Hierarchical  and Relaxation  Algorithms for \nSegmenting  Non-Causal  Markovian  Random  Fields.  IEEE  PAMI,  9,  1987,  pp.195-\n219. \n\n[2]  L.  Davis  and  T.  Henderson.  Hierarchical  Constraint  Processes  for  Shape  Analysis. \n\nIEEE PAMI,  3,  1981,  pp.265-277. \n\n[3]  B.  Gidas.  A  Renormalization  Group  Approach  to  Image  Processing  Problems.  IEEE \n\nPAMI,  11, 1989,  pp.164-180. \n\n[4]  T . Henderson.  Discrete  Relaxation  Techniques . Oxford University Press,  1990. \n[5]  D.J.  Spiegelhalter and S.L.  Lauritzen,  Sequential updating of conditional probabilities \n\non  directed  Graphical  structures,  Networks,  1990,  20, pp.579-605. \n\n[6]  D.  Marr, Vision. W.H.  Freeman and Co., San Francisco. \n[7]  J.  Pearl,  Probabilistic  Reasoning  in  Intelligent  Systems,  Morgan Kaufmann, 1988. \n[8]  M.  Meila  and M.  Jordan,  Optimal triangulation  with  continuous  cost  functions,  Ad(cid:173)\n\nvances in  Neural Information Processing  Systems 9,  to appear 1997. \n\n[9]  P.Smyth, D. Heckerman, M.1.  Jordan,  Probabilistic  independence networks for hidden \n\nMarkov probability  models,  Neural Computation, 9,  1997,  pp. 227-269. \n\n[10]  R.C . Wilson  and E .  R.  Hancock,  Structural  Matching  by  Discrete  Relaxation.  IEEE \n\nPAMI, 19, 1997,  pp.634- 648. \nIEEE PAMI,  June 1997. \n\n\f", "award": [], "sourceid": 1454, "authors": [{"given_name": "Richard", "family_name": "Wilson", "institution": null}, {"given_name": "Edwin", "family_name": "Hancock", "institution": null}]}