{"title": "Combining Neural and Symbolic Learning to Revise Probabilistic Rule Bases", "book": "Advances in Neural Information Processing Systems", "page_first": 107, "page_last": 114, "abstract": null, "full_text": "Combining Neural and  Symbolic \n\nLearning to Revise Probabilistic Rule \n\nBases \n\nJ. Jeffrey Mahoney and Raymond J. Mooney \n\nmahoney@cs.utexas.edu,  mooney@cs.utexas.edu \n\nDept.  of Computer Sciences \n\nUniversity of Texas \nAustin, TX 78712 \n\nAbstract \n\na system for  revising probabilis(cid:173)\n\nThis paper describes  RAPTURE  -\ntic  knowledge  bases  that  combines  neural  and  symbolic  learning \nmethods.  RAPTURE  uses  a  modified  version  of backpropagation \nto refine  the  certainty factors  of a  MYCIN-style  rule base  and  uses \nID3's  information gain  heuristic  to  add  new  rules.  Results on  re(cid:173)\nfining  two  actual  expert  knowledge  bases  demonstrate  that  this \ncombined approach performs better  than previous methods. \n\n1 \n\nIntroduction \n\nIn complex domains,  learning needs  to be biased  with  prior  knowledge in  order  to \nproduce satisfactory results from limited training data.  Recently, both connectionist \nand  symbolic methods have  been  developed  for  biasing learning with  prior knowl(cid:173)\nedge lFu, 1989; Towell et  a/.,  1990; Ourston and Mooney,  1990].  Most ofthese meth(cid:173)\nods revise  an imperfect knowledge base (usually obtained from a domain expert)  to \nfit  a set of empirical data.  Some of these methods have been successfully  applied to \nreal-world tasks, such as recognizing promoter sequences in DNA [Towell et ai., 1990; \nOurston and Mooney,  1990].  The results demonstrate that revising an expert-given \nknowledge  base  produces  more  accurate  results  than  learning  from  training  data \nalone. \n\nIn \n\nthis  paper,  we  describe \n\nthe  RAPTURE  system  (Revising  Approximate \n\n107 \n\n\f108 \n\nMahoney and Mooney \n\nProbabilistic Theories  Using  Repositories of Examples),  which  combines connec(cid:173)\ntionist  and  symbolic  methods  to  revise  both  the  parameters  and  structure  of a \ncertainty-factor rule base. \n\n2  The Rapture  Algorithm \n\nThe  RAPTURE  algorithm  breaks  down  into  three  main  phases.  First,  an  initial \nrule-base  (created by a  human expert)  is  converted  into a  RAPTURE  network.  The \nresult  is  then  trained  using  :ertainty-factor backpropagation (CFBP).  The  theory \nis  further  revised  through  network  architecture  modification.  Once  the  network  is \nfully  trained,  the  solution is  at  hand-there is  no  need  for  retranslation.  Each  of \nthese steps  is outlined in full  below. \n\n2.1  The Initial Rule-Base \n\nRAPTURE  uses  propositional certainty factor  rules  to represent  its theories.  These \nrules  have  the form  A  ~ D,  which  expresses  the idea that  belief in proposition A \ngives  a  0.8  measure of belief in  proposition  D  [Shafer  and  Pearl,  1990].  Certainty \nfactors  can range in value from  -1 to + 1,  and  indicate  a  degree  of confidence  in  a \nparticular proposition.  Certainty factor  rules  allow  updating of these  beliefs  based \nupon new  observed  evidence. \nRules combine evidence via probabilistic sum, which is defined  as  a EB b - a + b - abo \nIn general, all positive evidence is combined to determine the measure  of belief(MB) \nfor  a  given  proposition,  and  all negative evidence  is  combined  to obtain  a  measure \nof disbelief (MD).  The certainty factor  is  then  calculated using  C F  =  M B + MD. \nRAPTURE  uses  this  formalism  to represent  its  rule  base  for  a  variety  of reasons. \nFirst, it is perhaps the simplest method that retains  the desired  evidence-summing \naspect  of uncertain reasoning.  As each rule fires,  additional evidence is  contributed \ntowards belief in the  rule's consequent.  The use of probabilistic sum enables many \nsmall pieces  of evidence  to  add  up  to  significant  evidence.  This is  lacking  in  for(cid:173)\nmalisms that use  only  MIN  or MAX for  combining evidence  [Valtorta,  1988].  Sec(cid:173)\nond,  probabilistic sum is  a simple,  differentiable,  non-linear function.  This is  cru(cid:173)\ncial for implementing gradient descent  using backpropagation.  Finally, and perhaps \nmost significantly, is  the widespread  use of certainty factors.  Numerous knowledge(cid:173)\nbases  have  been  implemented  using  this  formalism,  which  immediately gives  our \napproach  a  large base of applicability. \n\n2.2  Converting the Rule Base into a  Network \n\nOnce the initial theory is obtained, it is converted into a  RAPTURE -network.  Build(cid:173)\ning the network begins by mapping all identical propositions in the rule-base  to the \nsame node in the network.  Input features (those only appearing as rule-antecedents) \nbecome input nodes, and output symbols (those only appearing as rule-consequents) \nbecome output nodes.  The certainty factors of the rules become the weights on the \nlinks  that  connect  nodes.  Networks  for  classification problems  contain one output \nfor  each  category.  When  an example is  presented,  the  certainty factor  for  each  of \nthe  categories  is  computed  and  the  example  is  assigned  to  the  category  with  the \n\n\fCombining Neural  and  Symbolic  Learning to  Revise  Probabilistic Rule Bases \n\n109 \n\n1.3 \n\nI \nI \n\n.2t' , , \nI , \n\nFigure  1:  A  RAPTURE NETWORK \n\nhighest  value.  Figure  1 illustrates the following set of rules. \n\nABC~D  E~D  C~G  EF~G  HI~C \n\nAs  shown  in  the  network,  conjuncts  must  first  pass  through  a  MIN  node  before \nany  activation  reaches  the  consequent  node.  Note  that  each  of the  conjuncts  is \nconnected  to  the  corresponding  MIN  mode with  a  solid  line.  This  represents  the \nfact  that the link is non-adjustable, and simply passes its full  activation value onto \nthe  MIN  node.  The  standard  (certainty-factor)  links  are  drawn  as  dotted  lines \nindicating that their values  are  adjustable. \nThis construction shows  how  easily a  RAPTURE-network  can model a  MYCIN  rule(cid:173)\nbase.  Each  representation  can  be  converted  into  the  other,  without  loss  or  cor(cid:173)\nruption of information.  They  are  two equivalent representations  of the same set  of \nrules. \n\n2.3  Certainty Factor Backpropagation \n\nUsing the constructed  RAPTURE-network,  we  desire  to maximize its predictive ac(cid:173)\ncuracy over a set of training examples.  Cycling through the examples one at a time, \nand slightly adjusting all relevant network weights in a direction that will minimize \nthe output error,  results  in  hill-climbing to a  local  minimum.  This is  the idea be(cid:173)\nhind gradient descent  [Rumelhart  et  al.,  1986),  which  RAPTURE  accomplishes with \nCertainty Factor Backpropagation (CFBP),  using  the following equations. \n\nApWji =  7Jopj(1 \u00b1  LWjkOpk) \n\n(1) \n\nk#-i \n\nIf Uj  is an output unit \n\n\f110 \n\nMahoney and Mooney \n\nIf Uj  is  not  an output unit \n\nhpj  =  L: hpk wkj(1 \u00b1 EWjkOpk) \n\nkmin \n\ni;tk \n\n(2) \n\n(3) \n\nThe  \"Sigma with  circle\"  notation is  meant to represent  probabilistic sum over  the \nindex,  and  the \u00b1 notation is shorthand for  two separate  cases.  If WjiOpi  ~ 0,  then \nis  used,  otherwise + is  used.  The  kmin  subscript  refers  to the  fact  that  we  do \n-\nnot perform this summation for  every unit k  (as in standard backpropagation), but \nonly those  units  that  received  some contribution from  unit  j.  Since  a  unit  j  may \nbe  required  to pass through  a min or max-node before  reaching the next  layer (k), \nit is possible  that its value may not reach  k. \n\nRAPTURE  deems a classification correct  when the output value for  the correct cate(cid:173)\ngory is greater than that of any other category.  No error propagation takes place in \nthis case  (hpj  = 0).  CFBP terminates when  overall error reaches  a  minimal value. \n\n2.4  Changing the Network Architecture \n\nWhenever  training accuracy  fails  to reach  100% through CFBP, it may be  an indi(cid:173)\ncation  that  the  network  architecture  is  inappropriate for  the  current  classification \ntask.  To date,  RAPTURE has been given two ways of changing network architecture. \nFirst,  whenever  the weight  of a  link  in the  network  approaches  zero,  it is  removed \nfrom  the  network  along with  all  of the nodes  and  links  that become  detached  due \nto this removal.  Further,  whenever  an intermediate node loses  all of its input links \ndue to link deletion,  it too is removed from the network,  along with its output link. \nThis link/node deletion is performed immediately after CFBP, and before  anything \nnew  is introduced  into the network. \nRAPTURE also has  a method for  adding new nodes into the network.  Specific nodes \nare added in an attempt to maximize the number of training examples that are clas(cid:173)\nsified  correctly.  The simple solution employed by  RAPTURE  is  to create new  input \nnodes  that  connect  directly,  either  positively or  negatively,  to one or  more output \nnodes.  These new  nodes are created in a way that will best help the network distin(cid:173)\nguish  among training examples that are being misclassified.  Specifically,  RAPTURE \nattempts to  distinguish for  each  output  category,  those  examples of that  category \nthat  are  being misclassified  (Le.  being classified  into a  different  output  category), \nfrom  those examples that  do  belong in these  different  output categories.  Quinlan's \nID3 information gain metric [Quinlan,  1986]  has been  adopted  by RAPTURE  to se(cid:173)\nlect  this  new  node,  which  becomes  positive evidence  for  the  correct  category,  and \nnegative evidence for  mistaken categories. \n\nWith these  new  nodes in place, we  can  now return  to CFBP, where  hopefully more \ntraining examples will be successfully classified.  This entire process (CFBP followed \nby  deleting  links  and  adding  new  nodes)  repeats  until  all  training  examples  are \ncorrectly  classified.  Once  this has occurred,  the network is  considered  trained,  and \ntesting  may begin. \n\n\fCombining Neural  and Symbolic Learning to Revise  Probabilistic  Rule Bases \n\n111 \n\nSoybean Test Ac:eurac:y \n\n..-\n,-'  ,  :-:...-\n\n,.' \n\n,,' \n\nI,' \n\nI\"~ \n\nn.oo-+-+---Lf--,.:---:-f--':........-::-t~=-\"'--+--\n\n70.00--+--+-_,-\"-\u2022 ...,..::. f---~ ''--+-----+----1----\n\ni \n1 \n\nI ,  \n\n,,' \n\n/ \nI \n(1 ,. ':' \n\n. ' ,  \n\n. ~ I,' \n\n015.00-++-+---+.--+----+---+---+--\n6O.00'-++-+_..;...<f-__  --+ __  -+ __ -+ __ \n\n.\n.'  I \n\n,' \n\nn,oo'---H-J(.<-\",'-;,-.' -+---+----+----+--\n\n##  \n\n\" \n\nI~ ~ . \ni,' \n\nSO.OO \n\n41.OO--:O'*'00:-----:20.00:I-::----:00'*00=-----,6O.IIO\u00b1:-----:1O.::I-:00,-:----.\"\"..... \n\nO'CII--,O~.oo.------:!~'=----7.lco.:i-;CII;-------;I:-.\\50.=OO 1hinil:p \n\nFigure 2:  RAPTURE Testing Accuracy \n\n3  Experimental Results \n\nTo  date,  RAPTURE  has  been  tested  on  two  real-world  domains.  The first  of these \nis  a  domain for  recognizing promoter sequences  in strings of DNA-nucleotides.  The \nsecond  uses  a  theory for  diagnosing soybean diseases.  These datasets are discussed \nin  detail in the following sections. \n\n3.1  Promoter Recognition Results \n\nA  prokaryotic  promoter is  a  short  DNA sequence  that  precedes  the  beginnings  of \ngenes,  and  are  locations  where  the  protein  RNA  polymerase  binds  to  the  DNA \nstructure [Towell  et  al.,  1990].  A set of propositional Horn-clause rules for  recogniz(cid:173)\ning promoters, along with 106  labelled examples (53  promoters, 53  non-promoters) \nwas  provided  as  the initial theory. \n\nIn order for  this theory to used by  RAPTURE  it had to be modified into a  certainty \nfactor format.  This was  done  by  breaking up  rules  with multiple antecedents,  into \nseveral  rules.  In  this fashion,  each  antecedent  is  able  to  contribute some evidence \ntowards  belief in  the  consequent.  Initial  certainty  factors  were  assigned  in  such  a \nway  that if every antecedent  (from the original rule) were true,  a certainty factor of \n0.9  would  result  for  the  consequent. \n\nTo  test  RAPTURE  using  this  dataset,  standard  training  and  test  runs  were  per(cid:173)\nformed,  which  resulted  in  the learning curve of Figure  2a.  This graph  is  a  plot of \naverage  performance  in  accuracy  at  classifying  DNA  strings  over  25  independent \ntrials.  A  single  trial consists  of providing each  system  with  increasing  numbers of \nexamples  to use  for  training,  and  then  seeing  how  well  it  can  classify  unseen  test \nexamples.  This graph clearly demonstrates the advantages of an evidence summing \n\n\f112 \n\nMahoney and Mooney \n\nsystem like RAPTURE over a pure Horn-clause system such as EITHER,  a pure induc(cid:173)\ntive system such as ID3, or a pure connectionist system, like backprop.  Also plotted \nin the  graph,  is  KBANN  [Towell  et  a/.,  1990],  a  symbolic-connectionist system  that \nuses  standard  backpropagation,  and  RAPTURE-O,  which  is  simply  RAPTURE  given \nno  initial  theory,  emphasizing  the  importance of the  expert  knowledge.  For  this \ndataset,  CFBP  alone  was  all  that  was required  in order to train the  network.  The \nnode  addition module was  never  called. \n\n3.2  Soybean Disease Diagnosis Results \n\nThe  Soybean  Data comes  from  [Michalski  and  Chilausky,  1980]  and  is  a  dataset \nof 562  examples of diseased  soybean  plants.  Examples were  described  by  a  string \nof 35  features  including  the  condition of the  stem,  the  roots,  the seeds,  as  well  as \ninformation such  as  the  time  of year,  temperature,  and  features  of the  soil.  An \nexpert  classified  each  example  into  one  of 15  soybean  diseases.  This  dataset  has \nbeen used  as  a benchmark for  a number of learning systems.  Figure 2b is a learning \ncurve  on  this  data comparing  RAPTURE,  RAPTURE-O,  backpropagation,  ID3,  and \nEITHER. \nThe headstart given to RAPTURE  does  not last throughout  testing in this domain. \nRAPTURE  maintains a  statistically significant  lead over  the other  systems  (except \nRAPTURE-O)  through 80 examples, but by 150 examples, all systems are performing \nat  statistically  equivalent  levels.  A  likely  explanation  for  this  is  that  the  expert \nprovided theory is more helpful on the easier to diagnose diseases than on those that \nare  more  difficult.  But  these  easy  ones  are  also  easy  to learn  via pure  induction, \nand good rules can be created after seeing only a few examples.  Trials have actually \nbeen run out to 300 examples, though all systems are performing at equivalent levels \nof accuracy. \n\n4  Related Work \n\nThe SEEK  system [Ginsberg et  a/.,  1988) revises rule bases containing M-of-N rules, \nthough  can  not modify real-valued  weights  and  contains no means for  adding  new \nrules.  Valtorta [Valtorta, 1988)  has examined the computational complexity of var(cid:173)\nious  refinement  tasks  for  probabilistic  knowledge  bases,  and  shows  that  refining \nthe weights  to fit  a  set  of training data is  an  NP-Hard  problem.  Ma and Wilkins \n[Ma  and  Wilkins,  1991)  have  developed  methods for  improving the  accuracy  of a \ncertainty-factor knowledge base by deleting rules,  and they report modest improve(cid:173)\nments in the  accuracy  of a  MYCIN  rule  base.  Gallant [Gallant,  1988)  designed  and \nimplemented a  system that  combines expert domain knowledge  with connectionist \nlearning, though is not suitable for multi-layer networks or for combination functions \nlike probabilistic sum.  KBANN  [Towell  et  a/.,  1990]  uses standard  backpropagation \nto refine  a symbolic rule base,  though the mapping between  the symbolic rules  and \nthe network is only an approximation.  Fu [Fu,  1989) and Lacher [Lacher,  1992) have \nalso used  backpropagation techniques to revise  certainty factors on rules.  However, \nthe current  publications on these two projects do not  address the problem of alter(cid:173)\ning the  network  architecture  (i.e.  adding new  rules)  and do not  present  results  on \nrevising  actual expert  knowledge bases. \n\n\fCombining Neural  and Symbolic  Learning to Revise  Probabilistic Rule  Bases \n\n113 \n\n5  Future Work \n\nThe  current  method  for  changing  network  architecture  in  RAPTURE  is  restricted \nto adding new  input units that  directly  feed  the outputs.  We  hope  to incorporate \nnewer  techniques for  creating and linking to hidden nodes,  in order to improve the \nrange of architectural  changes  that it can make. \n\nAnother  area requiring further  research  concerns  the differences  between  certainty(cid:173)\nfactor  networks  and  traditional connectionist  networks.  Further comparison of the \nRAPTURE  and  KBANN  approaches to knowledge-base refinement are  also indicated. \n\nFinally, in recent  years,  certainty-factors have been  the subject of considerable crit(cid:173)\nicism  from  researchers  in  uncertain  reasoning  [Shafer  and  Pearl,  1990].  However, \nthe basic  revision  framework  in  RAPTURE  should  be  applicable to other uncertain \nreasoning formalisms such  as  Bayesian networks,  Dempster-Shafer theory,  or fuzzy \nlogic [Shafer and Pearl,  1990].  As long as the activation functions in the correspond(cid:173)\ning network  implementations of these  methods  are differentiable,  backpropagation \ntechniques should  be employable. \n\n6  Conclusions \n\nAutomatic refinement  of probabilistic rule  bases is  an  under-studied  problem with \nimportant applications  to the  development of intelligent systems.  This paper  has \ndescribed  and  evaluated  an  approach  to  refining  certainty-factor  rule  bases  that \nintegrates connectionist  and symbolic learning.  The approach is  implemented in  a \nsystem called  RAPTURE,  which uses a revised  backpropagation algorithm to modify \ncertainty factors and ID3's information gain criteria to determine new rules to add to \nthe  network.  In other words,  connectionist  methods are  used  to adjust parameters \nand symbolic methods are  used  to make structural changes  to the knowledge base. \n\nIn domains with limited training data or domains requiring meaningful explanations \nfor  conclusions,  refining  existing  expert  knowledge  has  clear  advantages.  Results \non  revising  three  real-world  knowledge  bases  indicates  that  RAPTURE  generally \nperforms better than purely inductive systems (ID3 and backpropagation), a purely \nsymbolic revision  system  (EITHER),  and  and  purely  connectionist  revision  system \n(KBANN). \n\nThe certainty-factor networks  used  in  RAPTURE  blur  the distinction  between  con(cid:173)\nnectionist and symbolic representations.  They can be viewed either as connectionist \nnetworks or symbolic rule bases.  RAPTURE demonstrates the utility of applying con(cid:173)\nnectionist learning methods to \"symbolic\" knowledge bases and employing symbolic \nmethods to modify  \"connectionist\"  networks.  Hopefully  these  results  will  encour(cid:173)\nage  others  to  explore  similar opportunities  for  cross-fertilization  of ideas  between \nconnectionist  and symbolic learning. \n\nAcknowledgements \n\nThis research  was supported  by the  National Science  Foundation under  grant  IRI-\n9102926, the  NASA  Ames Research  Center under grant NCC  2-629,  and the Texas \nAdvanced Research Program under grant 003658114.  We wish to thank R.S. Michal-\n\n\f114 \n\nMahoney and Mooney \n\nski for furnishing the soybean data,  M.  Noordewier,  G.G. Towell, and J.W. Shavlik \nfor  supplying the DNA  data,  and  the  KBANN  results. \n\nReferences \n\n[Fu,  1989]  Li-Min  Fu.  Integration of neural  heuristics  into knowledge-based  infer(cid:173)\n\nence.  Connection  Science,  1(3):325-339, 1989. \n\n[Gallant, 1988]  S.1.  Gallant.  Connectionist ex'\")ert  systems.  Communications  of the \n\nAssociation for  Computing  Machinery,  31:152-169, 1988. \n\n[Ginsberg  et  al.,  1988]  A.  Ginsberg,  S.  M.  Weiss,  and  P.  Politakis.  Automatic \nknowledge  based  refinement  for  classification  systems.  Artificial  Intelligence, \n35:197-226,  1988. \n\n[Lacher,  1992]  R.C. Lacher.  Expert  networks:  Paradigmatic conflict,  technological \n\nrapprochement.  Neuroprose  FTP Archive,  1992. \n\n[Ma and  Wilkins, 1991]  Y.  Ma and  D.  C.  Wilkins.  Improving  the  performance  of \n\ninconsistent  knowledge  bases  via  combined  optimization  method.  In  Proceed(cid:173)\nings  of the  Eighth  International  Workshop  on  Machine  Learning,  pages  23-27, \nEvanston, IL, June  1991. \n\n[Michalski and Chilausky,  1980]  R.  S.  Michalski and S.  Chilausky.  Learning by be(cid:173)\n\ning  told  and  learning  from  examples:  An  experimental  comparison  of the  two \nmethods  of knowledge  acquisition  in  the  context  of developing  an  expert  sys(cid:173)\ntem for  soybean  disease  diagnosis.  Journal  of Policy  Analysis  and  Information \nSystems,  4(2):126-161, 1980. \n\n[Ourston  and  Mooney,  1990]  D.  Ourston  and  R.  Mooney.  Changing  the  rules:  a \n\ncomprehensive  approach  to theory  refinement.  In  Proceedings  of the  Eighth  Na(cid:173)\ntional  Conference  on  Artificial  Intelligence,  pages  815-820,  Detroit,  MI,  July \n1990. \n\n[Quinlan,  1986]  J.  R.  Quinlan. \n\n1(1):81-106,1986. \n\nInduction  of  decision  trees.  Machine  Learning, \n\n[Rumelhart  et  al.,  1986]  D. E. Rumelhart, G.  E. Hinton, and J. R. Williams. Learn(cid:173)\ning internal  representations by error  propagation.  In D.  E.  Rumelhart and J.  L. \nMcClelland, editors,  Parallel Distributed Processing,  Vol.  I,  pages 318-362.  MIT \nPress,  Cambridge, MA,  1986. \n\n[Shafer  and Pearl,  1990]  G.  Shafer  and  J.  Pearl,  editors.  Readings  in  Uncertain \n\nReasoning.  Morgan Kaufmann, Inc.,  San  Mateo,CA,  1990. \n\n[Towell  et  al.,  1990]  G.  G. Towell, J.  W.  Shavlik, and  Michiel  O.  Noordewier.  Re(cid:173)\n\nfinement  of  approximate  domain  theories  by  knowledge-based  artificial  neural \nnetworks.  In  Proceedings  of the  Eighth  National  Conference  on  Artificial Intelli(cid:173)\ngence,  pages 861-866, Boston,  MA,  July  1990. \n\n[Valtorta,  1988]  M.  Valtorta.  Some results on the complexity of knowledge-base re(cid:173)\nfinement.  In  Proceedings  of the  Sixth  International  Workshop  on  Machine  Learn(cid:173)\ning,  pages  326-331, Ithaca,  NY,  June  1988. \n\n\f", "award": [], "sourceid": 612, "authors": [{"given_name": "J.", "family_name": "Mahoney", "institution": null}, {"given_name": "Raymond", "family_name": "Mooney", "institution": null}]}