{"title": "Support Vector Machines for Multiple-Instance Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 577, "page_last": 584, "abstract": null, "full_text": "Support  Vector  Machines for \nMulti ple-Instance  Learning \n\nStuart Andrews,  Ioannis  Tsochantaridis and Thomas Hofmann \nDepartment of Computer Science,  Brown University,  Providence, RI 02912 \n\n{stu,it,th}@cs.brown.edu \n\nAbstract \n\nThis  paper  presents  two  new  formulations  of  multiple-instance \nlearning as a  maximum margin problem.  The proposed extensions \nof the  Support  Vector  Machine  (SVM)  learning  approach lead  to \nmixed integer quadratic programs that can be solved  heuristically. \nOur generalization of SVMs  makes  a  state-of-the-art classification \ntechnique,  including  non-linear  classification via kernels,  available \nto  an  area that  up  to now  has  been  largely  dominated by  special \npurpose methods.  We  present  experimental  results  on  a  pharma(cid:173)\nceutical data set and on applications in automated image indexing \nand document categorization. \n\n1 \n\nIntroduction \n\nMultiple-instance learning  (MIL)  [4]  is  a  generalization of supervised classification \nin which training class labels are associated with sets of patterns, or bags, instead of \nindividual patterns.  While every pattern may possess an associated true label, it is \nassumed that pattern labels are only indirectly accessible through labels attached to \nbags.  The law of inheritance is  such that a set receives a particular label, if at least \none  of the patterns in the  set possesses  the label.  In the important case  of binary \nclassification,  this  implies  that  a  bag  is  \"positive\"  if  at  least  one  of  its  member \npatterns is  a positive example.  MIL differs from the general set-learning problem in \nthat the set-level classifier is  by design induced by a  pattern-level classifier.  Hence \nthe key challenge in MIL is  to cope with the ambiguity of not knowing which of the \npatterns in a  positive bag are the actual positive examples and which ones are not. \n\nThe  MIL  setting  has  numerous  interesting  applications.  One  prominent  applica(cid:173)\ntion  is  the  classification  of  molecules  in  the  context  of  drug  design  [4].  Here, \neach  molecule  is  represented  by  a  bag  of possible  conformations.  The  efficacy  of \na  molecule  can  be  tested  experimentally,  but  there  is  no  way  to  control  for  indi(cid:173)\nvidual  conformations.  A  second  application is  in image indexing for  content-based \nimage  retrieval.  Here,  an image can be viewed  as  a  bag of local  image  patches  [9] \nor  image  regions.  Since  annotating  whole  images  is  far  less  time  consuming  then \nmarking  relevant  image regions,  the ability  to deal  with  this  type of weakly  anno(cid:173)\ntated data is  very desirable.  Finally, consider the problem of text categorization for \nwhich we  are the first  to apply the MIL  setting.  Usually, documents which contain \na  relevant  passage are  considered  to be relevant  with  respect  to a  particular cate-\n\n\fgory or topic,  yet class labels are rarely available on the passage level and are most \ncommonly  associated  with  the  document  as  a  whole.  Formally,  all  of  the  above \napplications share the  same type of label  ambiguity  which  in  our opinion  makes  a \nstrong argument in favor  of the relevance of the MIL  setting. \n\nWe  present two approaches to modify and extend Support Vector Machines (SVMs) \nto  deal  with  MIL  problems.  The first  approach explicitly treats the pattern labels \nas  unobserved integer  variables, subjected  to  constraints  defined  by  the  (positive) \nbag labels.  The goal then is  to maximize the usual pattern margin, or soft-margin, \njointly  over  hidden  label  variables  and  a  linear  (or  kernelized)  discriminant  func(cid:173)\ntion.  The second  approach generalizes the notion of a  margin to bags and aims  at \nmaximizing  the  bag  margin  directly.  The  latter  seems  most  appropriate  in  cases \nwhere  we  mainly  care  about  classifying  new  test  bags,  while  the  first  approach \nseems  preferable whenever the goal is  to derive an accurate pattern-level classifier. \nIn the case of singleton bags, both methods are identical and reduce to the standard \nsoft-margin SVM formulation. \n\nAlgorithms for the MIL problem were first presented in [4,  1, 7].  These methods (and \nrelated analytical results)  are based on hypothesis classes consisting of axis-aligned \nrectangles.  Similarly,  methods  developed  subsequently  (e.g.,  [8,  12])  have  focused \non specially tailored machine learning algorithms that do not compare favorably in \nthe limiting case of the standard classification setting.  A notable exception is  [10]. \nMore recently, a kernel-based approach has been suggested which derives MI-kernels \non  bags from  a  given  kernel  defined  on  the  pattern-level  [5].  While  the  MI-kernel \napproach treats the MIL problem merely as a representational problem, we strongly \nbelieve that a  deeper  conceptual modification of SVMs  as  outlined in this  paper is \nnecessary.  However,  we  share the ultimate goal  with  [5],  which  is  to make state-of(cid:173)\nthe-art kernel-based classification methods available for  multiple-instance learning. \n\n2  Multiple-Instance Learning \n\nIn  statistical  pattern  recognition,  it  is  usually  assumed  that  a  training  set  of  la(cid:173)\nbeled  patterns  is  available  where  each  pair  (Xi, Yi)  E  ~d X  Y  has  been  generated \nindependently from  an unknown distribution.  The goal is  to induce a classifier, i.e., \na  function  from  patterns  to  labels  !  :  ~d --+  y.  In  this  paper,  we  will  focus  on \nthe  binary  case  of Y  =  {-I, I}.  Multiple-instance learning  (MIL)  generalizes this \nproblem  by  making  significantly  weaker  assumptions  about  the  labeling  informa(cid:173)\ntion.  Patterns  are  grouped  into  bags  and  a  label  is  attached to each  bag  and  not \nto every pattern.  More formally, given  is  a  set of input patterns Xl, ... , Xn  grouped \ninto bags B l , ... , B m ,  with BI =  {Xi: i  E I} for  given index sets I  ~ {I, ... , n} (typ(cid:173)\nically  non-overlapping).  With each  bag B I  is  associated  a  label  YI.  These  labels \nare interpreted in  the following  way:  if YI  = -1, then Yi  = -1 for  all  i  E I,  i.e.,  no \npattern in the bag is  a  positive example.  If on the other hand YI  =  1,  then at least \none  pattern Xi  E  BI is  a  positive  example of the  underlying  concept.  Notice  that \nthe  information  provided  by  the  label  is  asymmetric  in  the  sense  that  a  negative \nbag label  induces  a  unique  label  for  every  pattern in  a  bag,  while  a  positive  label \ndoes not.  In general, the relation between pattern labels Yi  and bag labels YI  can be \nexpressed compactly as YI  =  maxiEI Yi  or alternatively as a set of linear constraints \n\n'\"' Yi  + 1 \n~ -2- ;:::  1,  VI s.t.  YI  =  1,  and  Yi  =  -1,  VI  s.t.  YI  =  -1. \niEI \n\n(1) \n\nFinally, let us  call a discriminant function! : X  --+  ~ MI-separating with respect to \na  multiple-instance data set if sgn maxiEI !(Xi) =  YI  for  all  bags  BI holds. \n\n\f(a) \n\n3 \n\n1 -\n\n..... Q) \n\n..\u2022.. \n\n<j) \n\n2 \n\n2 \n\n3 \n\n2 \n\n.... \n\n@. \n\n2 \n\n3 \n\n2 \n\n(b) \n\n2 \n\n2 \n\n8 \\ \n3 \n\n2 \n\n3 \n\nFigure  1:  Large  margin  classifiers  for  MIL.  Negative  patterns  are  denoted  by  \"-\" \nsymbols,  positive bag patterns by  numbers  that encode the bag membership.  The \nfigure  to the left  sketches the mi-SVM  solution  while  the figure  to the right shows \nthe MI-SVM  solution. \n\n3  Maximum Pattern Margin  Formulation of MIL \n\nWe omit an introduction to SVMs and refer the reader to the excellent books on this \ntopic,  e.g.  [11].  The mixed integer formulation of MIL  as a  generalized soft-margin \nSVM  can be written as follows  in  primal form \n\nmi-SVM \n\nminmin  -llwI12+CL~i \n{v;}  w,b,\u20ac \n\n. \nt \n\n1 \n2 \n\n(2) \n\ns.t.  Vi:  Yi((w,xi)+b):::=:l-~i' ~i:::=:O,  Yi  E{-l,l},and (1)  hold. \n\nNotice that in  the standard classification setting,  the labels Yi  of training patterns \nXi  would  simply  be  given,  while  in  (2)  labels  Yi  of  patterns  Xi  not  belonging  to \nany  negative  bag  are  treated  as  unknown  integer  variables.  In  mi-SVM  one  thus \nmaximizes a  soft-margin criterion jointly over possible label assignments as well  as \nhyperplanes.  Figure 1 (a)  illustrates this idea for the separable case:  We are looking \nfor  an MI-separating linear discriminant such that there is  at least one pattern from \nevery positive bag in the positive halfspace, while all patterns belonging to negative \nbags are in the negative halfspace.  At the same time,  we  would  like  to achieve the \nmaximal  margin  with  respect  to  the  (completed)  data  set  obtained  by  imputing \nlabels for  patterns in positive  bags in  accordance with Eq.  (1). \nThis is  similar to the approach pursued in  [6]  and  [3]  for  transductive inference.  In \nthe latter case,  patterns are either labeled or unlabeled.  Unlabeled data points are \nutilized to refine the decision boundary by maximizing the margin on all data points. \nWhile  the labeling for  each unlabeled  pattern can be  carried out independently in \ntransductive inference, labels of patterns in positive bags are coupled in MIL through \nthe inequality constraints. \n\nThe mi-SVM formulation leads to a  mixed integer programming problem.  One has \nto find both the optimal labeling and the optimal hyperplane.  On a conceptual level \nthis  mixed  integer formulation  captures exactly  what  MIL  is  about,  i.e. to  recover \nthe unobserved pattern labels  and to simultaneously find  an optimal discriminant. \nYet,  this  poses  a  computational  challenge  since  the  resulting  mixed  integer  pro(cid:173)\ngramming problem cannot be solved efficiently  with state-of-the-art tools,  even for \nmoderate size  data sets.  We  will  present an optimization heuristic in  Section 5. \n\n\f4  Maximum Bag Margin Formulation of MIL \n\nAn  alternative  way  of applying  maximum  margin  ideas  to  the  MIL  setting  is  to \nextend  the  notion  of a  margin  from  individual  patterns  to  sets  of patterns.  It is \nnatural to define the functional  margin of a  bag with respect to a  hyperplane by \n\nII ==  YI  max( (w, Xi) + b). \n\niEI \n\n(3) \n\nThis  generalization  reflects  the  fact  that  predictions  for  bag  labels  take  the  form \nYI  =  sgn maxiEI( (w, Xi) +b).  Notice that for a positive bag the margin is defined by \nthe margin of the  \"most positive\"  pattern, while the margin of a  negative bag is  de(cid:173)\nfined by the \"least negative\" pattern.  The difference between the two formulations of \nmaximum-margin problems is  illustrated in  Figure 1.  For the pattern-centered mi(cid:173)\nSVM  formulation, the margin of every pattern in a  positive bag matters, although \none  has  the freedom  to set  their  label  variables  so  as  to maximize  the  margin.  In \nthe  bag-centered  formulation,  only  one  pattern  per  positive  bag  matters,  since  it \nwill  determine  the  margin  of the  bag.  Once  these  \"witness\"  patterns  have  been \nidentified,  the  relative  position  of other  patterns  in  positive  bags  with  respect  to \nthe  classification  boundary  becomes  irrelevant.  Using  the  above  notion  of a  bag \nmargin,  we  define  an MIL  version of the soft-margin classifier by \n\nMI-SVM \n\nmm -llwl1 2 + C ~~I \n.  1 \nw , b ,~ 2 \n\n' \"  \nI \n\n(4) \n\ns.t.  VI:  YI mal x ( (w, Xi)  + b)  :::::  1 - ~I,  ~I :::::  O . \n\n\u2022 E \n\nFor negative bags one  can unfold  the max operation by introducing one inequality \nconstraint per pattern, yet with a single slack variable ~I.  Hence the constraints on \nnegative bag patterns,  where YI  =  -1 , read as  -(W, Xi)  - b:::::  1- ~I'  Vi  E  I. \nFor  positive  bags,  we  introduce  a  selector  variable  s(I)  E  I  which  denotes  the \npattern  selected  as  the  positive  \"witness\"  in  BI.  This  will  result  in  constraints \n(w, xs(I)) + b :::::  1 - ~I.  Thus we  arrive at the following  equivalent formulation \n\nmin min -llwl1 2  + C 2: ~I \n1 \ns  w ,b,~ 2 \n\nI \n\n(5) \n\ns.t.  VI:  YI  =  -1  /\\  -(W,Xi) - b:::::  1- ~I,  Vi E I, \n\nor  YI=l \n\n/\\  (w,xs(I))+b:::::1-~I'  and6:::::0. \n\n(6) \nIn this formulation,  every positive bag BI is  thus effectively represented by a single \nmember  pattern  XI  ==  xs(I).  Notice  that  \"non-witness\"  patterns  (Xi,  i  E  I  with \ni  =I- s(I))  have no impact on the objective. \nFor given selector variables, it is straightforward to derive the dual objective function \nwhich  is  very  similar to the  standard SVM  Wolfe  dual.  The only  major difference \nis  that  the  box constraints for  the  Lagrange parameters  c\u00a5  are  modified  compared \nto the standard SVM  solution,  namely one gets \no ::;  C\u00a5I  ::;  C, \n\nfor  I  s.t.  YI  =  1  and  0::; 2: C\u00a5i  ::;  C, \n\nfor  I  s.t.  YI  =  -1. \n\n(7) \n\niEI \n\nHence,  the influence of each  bag  is  bounded by  C. \n\n5  Optimization Heuristics \n\nAs we  have shown,  both formulations, mi-SVM and MI-SVM, can be cast as mixed(cid:173)\ninteger  programs.  In  deriving  optimization  heuristics,  we  exploit  the fact  that  for \n\n\finitialize  Yi  = YI  for  i  E  I \nREPEAT \n\ncompute  SVM  solution  vv , b  for  data  set  with  imputed  labels \ncompute  outputs  Ii  =  (VV, Xi)  + b  for  all  xi  in  positive  bags \nset  Yi  =  sgn(fi)  for  every  i  E  I,  YI  =  1 \nFOR  (every  positive  bag  BI) \n\nIF  (L iEI( l  + Yi)/2 == 0) \n\ncompute  i*  =  arg maxiEI Ii \nset  Yi*  =  1 \n\nEND \n\nEND \n\nWHILE  (imputed  labels  have  changed) \nOUTPUT  (vv, b) \n\nFigure  2:  Pseudo-code for  mi-SVM  optimization heuristics  (synchronous update). \n\ninitialize  XI  = L iE I xillII  for  every  positive  bag  BI \nREPEAT \n\ncompute  QP  solution  vv,b  for  data  set  with \n\npositive  examples  {XI : YI  =  I} \n\ncompute  outputs  Ii  =  (VV,Xi)  + b  for  all  xi  in  positive  bags \nset  XI  =  Xs(I)  ,  8(I)  =  arg maxiEI Ii  for  every  I,  YI  =  1 \n\nWHILE  (selector  variables  8(1)  have  changed) \nOUTPUT  (vv, b) \n\nFigure 3:  Pseudo-code for  MI-SVM optimization heuristics  (synchronous update). \n\ngiven integer  variables,  i.e.  the hidden labels  in  mi-SVM  and the selector variables \nin MI-SVM,  the problem reduces to a  QP that can be solved exactly.  Of course, all \nthe derivations also hold for  general kernel functions  K . \n\nA  general scheme  for  a  simple  optimization heuristic  may be  described  as  follows. \nAlternate the following two steps:  (i)  for given integer variables, solve the associated \nQP and find the optimal discriminant function,  (ii)  for a given discriminant, update \none,  several, or all integer variables in a  way that  (locally)  minimizes the objective. \nThe latter step may involve the update of a label variable Yi  of a single pattern in mi(cid:173)\nSVM, the update of a  single selector variable 8(I)  in MI-SVM, or the simultaneous \nupdate of all integer variables.  Since the integer variables are essentially decoupled \ngiven the discriminant  (with the exception of the bag constraints in mi-SVM),  this \ncan  be  done  very  efficiently.  Also  notice  that  we  can  re-initialize  the  QP-solver \nat  every  iteration  with  the  previously  found  solution,  which  will  usually  result  in \na  significant  speed-up.  In  terms  of  initialization  of  the  optimization  procedure, \nwe  suggest  to  impute  positive  labels  for  patterns  in  positive  bags  as  the  initial \nconfiguration in  mi-SVM.  In  MI-SVM,  XI  is  initialized  as  the  centroid  of the  bag \npatterns.  Figure  2  and  3  summarize  pseudo-code  descriptions  for  the  algorithms \nutilized in the experiments. \n\nThere are many possibilities to refine the above heuristic strategy, for  example,  by \nstarting from  different  initial conditions, by using branch and bound techniques to \nexplore larger parts of the discrete part of the search space, by performing stochas(cid:173)\ntic  updates  (simulated  annealing)  or  by  maintaining  probabilities  on  the  integer \nvariables  in  the  spirit  of deterministic  annealing.  However,  we  have  been  able  to \nachieve competitive results even with the simpler optimization heuristics, which val-\n\n\fMUSK1 \nMUSK2 \n\n84.8 \n84.9 \n\n88.0 \n84.0 \n\n88.9 \n82.5 \n\nEMDDl12J  DD  19J  MI-NN  l10J \n\nIAPR l4J  mi-SVM  MI-SVM \n\n92.4 \n89.2 \n\n87.4 \n83.6 \n\n77.9 \n84.3 \n\nTable 1:  Accuracy results for  various methods on the MUSK  data sets. \n\nidate the maximum margin formulation of SVM. We will address further algorithmic \nimprovements in future  work. \n\n6  Experimental Results \n\nWe  have performed experiments on various data sets to evaluate the proposed tech(cid:173)\nniques  and  compare  them  to  other  methods  for  MIL.  As  a  reference  method  we \nhave  implemented  the  EM  Diverse  Density  (EM-DD)  method  [12],  for  which  very \ncompetitive results have been reported on the MUSK  benchmark!. \n\n6.1  MUSK  Data Set \n\nThe  MUSK  data sets  are  the  benchmark  data  sets  used  in  virtually  all  previous \napproaches  and  have  been  described  in  detail  in  the  landmark  paper  [4].  Both \ndata sets,  MUSK1  and MUSK2, consist of descriptions of molecules  using multiple \nlow-energy conformations.  Each conformation is  represented by a  166-dimensional \nfeature  vector  derived  from  surface  properties.  MUSK1  contains  on  average  ap(cid:173)\nproximately 6 conformation per molecule, while  MUSK2  has on average more than \n60  conformations in each bag.  The averaged results  of ten  10-fold cross-validation \nruns  are  summarized  in  Table  1.  The  SVM  results  are  based  on  an  RBF  kernel \nK(x, y)  =  exp( -')'llx - Y112)  with  coarsely  optimized  ')'.  For  both  MUSK1  and \nMUSK2  data sets,  mi-SVM  achieves  competitive accuracy  values.  While  MI-SVM \noutperforms  mi-SVM  on  MUSK2,  it  is  significantly  worse  on  MUSK1.  Although \nboth methods fail  to achieve the performance of the best method  (iterative APR)2, \nthey compare favorably with other approaches to MIL. \n\n6.2  Automatic Image Annotation \n\nWe  have generated new  MIL  data sets for  an image annotation task.  The original \ndata  are  color  images  from  the  Corel  data  set  that  have  been  preprocessed  and \nsegmented with the Blobworld system [2].  In this representation, an image consists \nof  a  set  of  segments  (or  blobs) ,  each  characterized  by  color,  texture  and  shape \ndescriptors.  We  have utilized three different  categories  (\"elephant\",  \"fox\",  \"tiger\") \nin  our experiments.  In each case,  the data sets  have  100 positive and  100 negative \nexample  images.  The  latter  have  been  randomly  drawn  from  a  pool  of  photos  of \nother animals.  Due to the limited accuracy of the image segmentation, the relative \nsmall number of region descriptors and the small training set size, this ends up being \nquite a hard classification problem.  We are currently investigating alternative image \n\n1 However,  the  description  of EM-DD  in  [12]  seems  to  indicate  that  the  authors  used \nthe test data to select  the optimal solution obtained from  multiple runs of the algorithm. \nIn  the pseudo-code  formulation  of EM-DD,  Di  is  used  to  compute the error  for  the i-th \ndata fold,  where  it  should  in fact  be  D t  = D  - Di  (using the notation  of [12]).  We  have \nused the corrected version of the algorithm in our experiments and have obtained accuracy \nnumbers using EM-DD  that are more in line with previously published results. \n\n2Since the IAPR (iterative axis parallel rectangle)  methods in [4]  have been specifically \ndesigned and optimized for  the  MUSK  classification  task,  the superiority of APR should \nnot  be interpreted as a  failure. \n\n\fData Set  Dims \nCategory \nElephant \nFox \nTiger \n\ninst/feat \n1391/230 \n1320/230 \n1220/230 \n\nEM-DD \n\n78.3 \n56.1 \n72.1 \n\nmi-SVM \nlinear  poly \n78.1 \n82.2 \n55.2 \n58.2 \n78.4 \n78.1 \n\nrbf \n80.0 \n57.9 \n78.9 \n\nMI-SVM \n\nlinear \n81.4 \n57.8 \n84.0 \n\nrbf \npoly \n79.0 \n73.1 \n59.4  58.8 \n81.6 \n66.6 \n\nTable  2:  Classification accuracy of different  methods on the Corel image data sets. \n\nData Set  Dims \nCategory \nTST1 \nTST2 \nTST3 \nTST4 \nTST7 \nTST9 \nTST10 \n\ninst/feat \n3224/6668 \n3344/6842 \n3246/6568 \n3391/6626 \n3367/7037 \n3300/6982 \n3453/7073 \n\nEM-DD \n\n85.8 \n84.0 \n69.0 \n80.5 \n75.4 \n65.5 \n78.5 \n\nmi-SVM \nlinear  poly \nrbf \n92.5 \n93.6 \n90.4 \n75.9 \n78.2 \n74.3 \n83.3 \n69.0 \n87.0 \n82.8 \n80.0 \n69.6 \n78.7  81.3 \n81.3 \n55.2 \n65.6 \n67.5 \n79.6 \n78.3 \n52.6 \n\nMI-SVM \n\nlinear \n93.9 \n84.5 \n82.2 \n82.4 \n78.0 \n60.2 \n79.5 \n\nrbf \npoly \n93.7 \n93.8 \n76.4 \n84.4 \n85.1 \n77.4 \n82.9  77.3 \n64.5 \n78.7 \n63.7 \n57.0 \n81.0  69.1 \n\nTable  3:  Classification  accuracy  of  different  methods  on  the  TREC9  document \ncategorization sets. \n\nrepresentations  in  the  context  of  applying  MIL  to  content-based  image  retrieval \nand  automated  image  indexing,  for  which  we  hope  to  achieve  better  (absolute) \nclassification accuracies.  However, these data sets seem legitimate for  a comparative \nperformance analysis.  The results are summarized in Table 2.  They show that both, \nmi-SVM  and MI-SVM  achieve a similar accuracy and outperform EM-DD by a few \npercent.  While MI-SVM performed marginally better than mi-SVM,  both heuristic \nmethods  were  susceptible  to  other  nearby  local  minima.  Evidence  of  this  effect \nwas  observed  through experimentation with  asynchronus  updates,  as  described  in \nSection 5, where we varied the number of integer variables updated at each iteration. \n\n6.3  Text  Categorization \n\nFinally,  we  have  generated  MIL  data  sets  for  text  categorization.  Starting  from \nthe  publicly  available  TREC9  data set,  also  known  as  OHSUMED,  we  have  split \ndocuments  into  passages  using  overlapping  windows  of  maximal  50  words  each. \nThe original  data set  consists  of several  years  of selected  MEDLINE  articles.  We \nhave  worked  with  the  1987  data set  used  as  training data in  the  TREC9 filtering \ntask which consists of approximately 54,000 documents.  MEDLINE documents are \nannotated  with  MeSH  terms  (Medical  Subject  Headings),  each  defining  a  binary \nconcept.  The  total  number  of  MeSH  terms  in  TREC9  was  4903.  While  we  are \ncurrently  performing  a  larger scale  evaluation  of MIL  techniques  on  the  full  data \nset,  we  report  preliminary  results  here  on  a  smaller,  randomly  subsampled  data \nset.  We  have  been  using  the  first  seven  categories of the  pre-test  portion  with  at \nleast 100  positive examples.  Compared to the other data sets the representation is \nextremely sparse and high-dimensional, which  makes this data an interesting addi(cid:173)\nt ional benchmark.  Again,  using linear and polynomial  kernel functions,  which  are \ngenerally known to work well for  text categorization, both methods show improved \nperformance over EM-DD in almost all  cases.  No  significant difference between the \ntwo methods is  clearly evident for  the text classification task. \n\n\f7  Conclusion and  Future Work \n\nWe  have  presented  a  novel  approach  to  multiple-instance  learning  based  on  two \nalternative generalizations of the maximum margin idea used in SVM classification. \nAlthough  these formulations  lead  to hard mixed  integer  problems,  even  simple  lo(cid:173)\ncal optimization heuristics  already yield  quite competitive results compared to the \nbaseline  approach.  We  conjecture  that  better  optimization  techniques,  that  can \nfor  example avoid unfavorable local minima,  may further  improve the classification \naccuracy.  Ongoing  work  will  also  extend  the  experimental  evaluation  to  include \nlarger scale problems. \n\nAs far  as the MIL research problem is  concerned, we  have considered a  wider range \nof  data sets  and  applications  than  is  usually  done  and  have  been  able  to  obtain \nvery  good  results  across  a  variety  of data  sets.  We  strongly  suspect  that  many \nMIL  methods have been  optimized to perform well  on the  MUSK  benchmark and \nwe  plan  to  make  the  data sets  used  in  the  experiments  available  to  the  public  to \nencourage further  empirical comparisons. \n\nAcknowledgments \nThis work was  sponsored by an NSF-ITR grant,  award number IIS-0085836. \n\nReferences \n[1]  P.  Auer.  On learning  from  multi-instance examples:  Empirical  evaluation  of a  the(cid:173)\n\noretical  approach.  In  Proc.  14th  International  Conf.  on  Machin e  Learning,  pages \n21- 29.  Morgan  Kaufmann,  San Francisco,  CA,  1997. \n\n[2]  C.  Carson,  M.  Thomas,  S.  Belongie,  J.  M.  Hellerstein,  and J.  Malik.  Blobworld:  A \n\nsystem for  region-based  image indexing and retrieval.  In  Proceedings  Third  Interna(cid:173)\ntional  Conference  on  Visual  Information  Systems.  Springer,  1999. \n\n[3]  A.  Demirez  and  K.  Bennett.  Optimization  approaches  to  semisupervised  learning. \nIn  M.  Ferris,  O.  Mangasarian,  and J.  Pang,  editors,  Applications  and  Algorithms  of \nComplementarity. Kluwer  Academic Publishers,  Boston, 2000. \n\n[4]  T .  G. Dietterich, R.  H.  Lathrop, and T . Lozano-Perez.  Solving the multiple instance \n\nproblem with axis-parallel rectangles.  Artificial Intellig ence, 89(1-2):31- 71 ,  1997. \n\n[5]  T.  Gartner,  P.  A.  Flach,  A.  Kowalczyk, and A. J.  Smola.  Multi-instance kernels.  In \n\nProc.  19th  International  Conf.  on  Machine  Learning.  Morgan  Kaufmann,  San Fran(cid:173)\ncisco,  CA,  2002. \n\n[6]  T.  Joachims.  Transductive inference  for  text  classification  using support  vector  ma(cid:173)\n\nchines.  In  Proceedings  16th  International  Conference  on  Machine  Learning,  pages \n200- 209.  Morgan  Kaufmann , San Francisco,  CA,  1999. \n\n[7]  P.M. Long and L. Tan.  PAC learning axis aligned rectangles with respect to product \ndistributions from multiple-instance examples. In Proc.  Compo  Learning  Theory,  1996. \nIn \n\n[8]  O.  Maron  and  T.  Lozano-Perez.  A  framework  for  multiple-instance  learning. \n\nAdvances  in  Neural  Information  Processing  Systems,  volume  10.  MIT  Press,  1998. \n[9]  O.  Maron  and  A.  L.  Ratan.  Multiple-instance  learning  for  natural  scene  classifica(cid:173)\n\ntion.  In Proc.  15th  International  Conf.  on  Machine Learning, pages 341- 349.  Morgan \nKaufmann,  San Francisco,  CA,  1998. \n\n[10]  J. Ramon and L.  De Raedt.  Multi instance neural networks.  In Proceedings  of ICML-\n\n2000,  Workshop  on  Attribute- Valu e  and  Relational Learning,  2000 . \n\n[11]  B.  SchOlkopf and A.  Smola.  Learning  with  Kernels.  Support  Vector  Machines,  Regu(cid:173)\n\nlarization,  Optimization  and  Beyond.  MIT  Press,  2002. \n\n[12]  Qi  Zhang and  Sally  A.  Goldman.  EM-DD:  An  improved multiple-instance learning \ntechnique.  In  Advances  in  Neural  Information  Processing  Systems,  volume  14.  MIT \nPress,  2002. \n\n\f", "award": [], "sourceid": 2232, "authors": [{"given_name": "Stuart", "family_name": "Andrews", "institution": null}, {"given_name": "Ioannis", "family_name": "Tsochantaridis", "institution": null}, {"given_name": "Thomas", "family_name": "Hofmann", "institution": null}]}