{"title": "Transductive Inference for Estimating Values of Functions", "book": "Advances in Neural Information Processing Systems", "page_first": 421, "page_last": 427, "abstract": null, "full_text": "Transductive Inference for  Estimating \n\nValues of Functions \n\nOlivier Chapelle*, Vladimir Vapnik*,t,  Jason Westontt.t,* \n\n*  AT&T Research Laboratories, Red  Bank,  USA. \n\nt  Royal Holloway,  University of London,  Egham, Surrey,  UK. \n\ntt  Barnhill BioInformatics.com, Savannah, Georgia,  USA. \n\n{ chapelle, vlad, weston} @research.att.com \n\nAbstract \n\nWe  introduce an algorithm for  estimating the values of a  function \nat a set of test points  Xe+!, ... , xl+m  given a set of training points \n(XI,YI), ... ,(xe,Ye)  without  estimating  (as  an  intermediate  step) \nthe regression function .  We demonstrate that this direct (transduc(cid:173)\nti ve)  way  for  estimating  values  of the  regression  (or  classification \nin  pattern  recognition)  can  be  more  accurate  than  the  tradition(cid:173)\nalone  based  on  two  steps,  first  estimating  the  function  and  then \ncalculating the values of this function  at the  points of interest. \n\n1 \n\nIntroduction \n\nFollowing [6]  we consider a general scheme of transductive inference.  Suppose there \nexists  a  function  y*  =  fo(x)  from  which  we  observe  the  measurements  corrupted \nwith noise \n\n(1) \nFind an algorithm A that using both the given set of training data (1)  and the given \nset of test data \n\n. (xe, Ye)),  Yi  = Y;  + ~i' \n\n((Xl, YI),\" \n\nselects from  a  set of functions  {x t--+  f (x)}  a  function \n\n(Xl+!,' .. , XHm) \n\nY = f(x)  = fA(xlxl,YI, ... ,Xl,Yl,XHI\"\",XHm) \n\nand minimizes  at the points of interest the functional \n\nR(A) =  E (~ (y; -\n\ni=l+l \n\nfA(Xilxl,Yl, ... ,Xl,Ye,Xl+l, . .. ,Xl+m))2) \n\n(2) \n\n(3) \n\n(4) \n\nwhere  expectation  is  taken  over  X  and~.  For  the training  data we  are  given  the \nvector  X  and the value Y,  for  the test data we  are only given  x. \n\nUsually,  the  problem  of  estimating  values  of  a  function  at  points  of  interest  is \nsol ved  in  two  steps:  first  in  a  given  set  of functions  f (x, a),  a  E  A one  estimates \nthe regression, i.e  the function  which minimizes the functional \n\nR(a) = J ((y -\n\nf(x, a))2dF(x, Y), \n\n(5) \n\n\f422 \n\n0.  Chapelle,  V. N.  Vapnik and J.  Weston \n\n(the inductive step) and then using the estimated function Y =  f(x,al) we calculate \nthe values  at points of interest \n\nyi  =  f(x;, ae) \n\n(6) \n\n(the deductive step). \n\nNote, however, that the estimation of a function is  equivalent to estimating its val(cid:173)\nues  in  the  continuum  points  of the  domain  of the  function.  Therefore,  by  solving \nthe  regression  problem  using  a  restricted  amount  of information,  we  are  looking \nfor  a  more  general  solution  than  is  required.  In  [6]  it  is  shown  that  using  a  di(cid:173)\nrect  estimation  method  one  can obtain  better  bounds  than  through  the  two  step \nprocedure. \n\nIn  this  article  we  develop  the  idea introduced in  [5]  for  estimating the  values  of a \nfunction  only at the given points. \n\nThe material is  organized as follows.  In Section 1 we  consider the classical  (induc(cid:173)\ntive)  Ridge Regression procedure, and the leave-one--out technique which is used to \nmeasure the quality of its  solutions.  Section 2 introduces the transductive method \nof inference  for  estimation of the  values  of a  function  based on  this  leave-one- out \ntechnique. \nIn  Section  3  experiments  which  demonstrate  the  improvement  given \nby  transductive inference  compared to  inductive  inference  (in  both regression and \npattern recognition)  are presented.  Finally, Section 4 summarizes the results. \n\n2  Ridge  Regression and  the Leave-One-Out  procedure \n\nIn order to describe our transductive method,  let  us first discuss the classical two(cid:173)\nstep  (inductive plus  deductive)  procedure of Ridge Regression.  Consider the set of \nfunctions  linear in their  parameters \n\nn \n\nf(x, a) = L aicPi(x). \n\ni=1 \n\n(7) \n\nTo  minimize  the  expected  loss  (5),  where  F(x, y)  is  unknown,  we  minimize  the \nfollowing  empirical functional  (the so-called Ridge Regression functional  [1]) \n\nRemp(a)  =  e L)Yi -\n1 ~ \n\nl \n\ni=1 \n\nf(Xi, a))  + 1'110.11 \n2 \n\n2 \n\n(8) \n\nwhere l' is  a fixed  positive constant, called the regularization parameter.  The min(cid:173)\nimum is  given  by the vector of coefficients \n\nae  =  a(xl, Yl, ... , Xl, Yl)  =  (KT K  + 1'1)-1 KTy \n\nwhere \n\nand  K  is  a  matrix with elements: \n\ny  =  (Y1,  ... ,Ylf, \n\nKij=cPj(Xi),  i=I, ... ,\u00a3,  j=I, ... ,n. \n\n(9) \n\n(10) \n\n(11) \n\nThe problem is to choose the value l' which provides small expected loss for  training \non a  sample  Sl =  {(Xl,Yl), .. . ,(Xl,Yl)}. \nFor this purpose,  we  would  like  to choose  l' such that  f\"f  minimizing  (8)  also mini(cid:173)\nmizes \n\nR  =  J (Y*  - f\"f(x* ISl))2dF(x*, y*)dF(Se). \n\n(12) \n\n\fTransductive Inference for Estimating Values  of Functions \n\n423 \n\nSince F(x, y)  is  unknown one cannot estimate this minimum directly.  To solve this \nproblem  we  instead use  the  leave-one-out procedure,  which  is  an almost  unbiased \nestimator of (12).  The leave-one-out error of an algorithm on the training sample \nSf.  is \n\n(13) \n\nThe  leave-one-out  procedure  consists  of removing  from  the  training  data one  el(cid:173)\nement  (say  (Xi, Yi)),  constructing  the  regression  function  only  on  the  basis  of the \nremaining training data and then testing the removed element.  In this fashion  one \ntests all f  elements  of the  training data using f  different  decision  rules.  The mini(cid:173)\nmum over, of (13)  we consider as the minimum over, of (12)  since the expectation \nof (13)  coincides  with  (12)  [2]. \n\nFor Ridge Regression, one can derive a dosed form expression for  the leave- one- out \nerror.  Denoting \n\nthe error incurred by the leave-one-out procedure is  [6] \n\nT. \n\nloo(r)  -\n\n1 \n-_ \n\nf  L \n\n~=1 \n\nf.  (Y'_kTA-1KTy)2 \n\n'Y \n\n~ \n\n~ \n\n1 _  kT A-1k. \n~ \n\n'Y \n\n~ \n\nwhere \n\nLet, = ,0  be  the minimum of (15).  Then the vector \n\nkt =  (i>I(xd\u00b7\u00b7\u00b7 ,l/>n(Xt)f\u00b7 \n\nyO  =  K*(KT K  +,0 I)-I KTy \n\nwhere \n\nK*-\n\n( \n\nI/>(XHI) \n\n. \n\n1/>1 (XHm) \n\n(14) \n\n(15) \n\n(16) \n\n(17) \n\n(18) \n\nis  the Ridge Regression estimate of the unknown  values  (Ye+l' ... ,Ye+m)' \n\n3  Leave-One-Out  Error  for  Transductive Inference \n\nIn  transductive  inference,  our goal  is  to find  an  algorithm  A  which  minimizes  the \nfunctional  (4)  using both the training data (1)  and the test data (2).  We suggest the \nfollowing  method:  predict  (Ye+l' ... 'Ye+m)  by finding  those values  which  minimize \nthe leave-one-out error of Ridge  Regression training on the joint set \n\n(Xl, yd,\u00b7\u00b7 . ,(Xl, Yl), (Xl+l, ye+l),\u00b7\u00b7 ., (XHm, Ye+m)' \n\n(19) \nThis  is  achieved  in  the  following  way.  Suppose  we  treat  the  unknown  values \n(Ye+l\"  .. ,Ye+m)  as  variables  and  for  some  fixed  value  of these  variables  we  min(cid:173)\nimize  the following  empirical functional \n\nRemp(aly;, .. \u00b7, y~) = f: m  ~(Yi - f(xi,a))2 + . L  (y;  -\n\nHm \n\nf. \n\n(\n\n) \n\nf(xi, a))2  +,llaI1 2 . \n\n~=l \n\n~=l+1 \n\n(20) \nThis  functional  differs  only  in  the  second  term  from  the  functional  (8)  and  corre(cid:173)\nsponds to performing Ridge  Regression  with  the extra pairs \n\n(21) \n\n\f424 \n\nO.  Chapel/e,  V.  N.  Vapnik and J.  Weston \n\nSuppose  that  vector  Y\"  =  (Yi, ... , y:n)  is  taken  from  some  set  Y\"  E Y  such  that \nthe pairs  (21)  can be considered  as  a  sample  drawn  from  the same distribution as \nthe  pairs  (Xl, yi), ... , (Xl, yi)\u00b7  In  this  case  the  leave-one-out error of minimizing \n(20)  over the set  (19)  approximates the functional  (4).  We  can measure this leave(cid:173)\none-out error using  the  same  technique  as  in  Ridge  Regression.  Using  the  closed \nform  (15)  one obtains \n\n7loo(rly~, .. \u00b7,y~) =  -f-- L \n1 \n+ m  i=l \n\nl+m (Y:' _ kT A-I kTY) 2 \n\n~ \nt~T ~-1~ \n1 - ki  A-y  ki \n\nwhere we  denote x = (Xl, ... , Xl+m),  Y = (YI, ... , Yl, Yi+1\" \n\n.. , Yi+m)T,  and \n\nKij=<pj(Xi),  i=I, ... ,i+m,  j=I, ... ,n. \n\nkt  =  (<PI(Xt}  ... ,<Pn(xt)f\u00b7 \n\n(22) \n\n(23) \n\n(24) \n\n(25) \n\nNow  let  us rewrite the expression  (22)  in  an equivalent form  to separate the terms \nwith Y from  the terms with  x.  Introducing \n\nand the matrix M  with elements \n\nl+m \n\nM .. _  \"  CikCkj \ntJ  - ~  2 \n\nk=l  Cu \n\nwe  obtain the equivalent expression of (22) \n\n(26) \n\n(27) \n\n(28) \n\nIn  order  for  the  Y\"  which  minimize  the  leave-one-out  procedure  to  be  valid  it \nis  required  that  the  pairs  (21)  are  drawn from  the  same  distribution  as  the  pairs \n(Xl, yi), ... , (Xl, yi)\u00b7  To satisfy this constraint we  choose vectors Y\"  from  the set \n\nY =  {Y\" : IIY\" - y011  -s:  R} \n\nwhere the vector yO  is  the solution obtained from  classical Ridge Regression. \nTo  minimize  (28)  under constraint  (29)  we  use the functional \n\n(29) \n\n(30) \n\nwhere 'Y\"  is  a  constant depending on R. \nNow,  to find  the values at the given points of interest  (2)  all that remains is to find \nthe  minimum  of  (30)  in  Y\".  Note  that  the  matrix  M  is  obtained  using  only  the \nvectors  X.  Therefore,  to find  the  minimum  of this  functional  we  rewrite  Equation \n(30)  as \n\nwhere \n\n(32) \n\n\fTransductive Inference for Estimating Values of Functions \n\n425 \n\nand Mo  is  a e x e matrix, Ml  is  a e x m  matrix and M2  is a m  x m  matrix.  Taking \nthe derivative of (31)  in  y*  we  obtain the condition for  the solution \n\n2M1Y + 2M2Y* - 2),*Yo + 2),*Y*  =  0 \n\nwhich  gives the predictions \n\nY* = ()'* 1+ M 2)-1 (-MIY + ),*yO) . \n\n(33) \n\n(34) \n\nIn this algorithm  (which  we  will  call Transductive Regression)  we  have two param(cid:173)\neters  to  control:  )'  and )'*.  The  choice  of )' can  be  found  using  the  leave-one-out \nestimator (15)  for  Ridge  Regression.  This leaves )'*  as  the only free  parameter. \n\n4  Experiments \n\nTo  compare  our  one- step  transductive  approach  with  the  classical  two- step  ap(cid:173)\nproach  we  performed  a  series of experiments on regression  problems.  We  also  de(cid:173)\nscribe experiments applying our technique to the problem of pattern recognition. \n\n4.1  Regression \n\nWe  conducted computer simulations for  the regression  problem using two datasets \nfrom  the DELVE repository:  boston and kin-32th. \nThe  boston dataset  is  a  well- known  problem  where  one  is  required  to  estimate \nhouse  prices  according to  various  statistics  based  on  13  locational,  economic  and \nstructural  features  from  data  collected  by  the  U.S  Census  Service  in  the  Boston \nMassachusetts area. \nThe kin-32th dataset is  a realistic simulation of the forward dynamics of an 8 link \nall-revolute robot arm.  The task is  to predict the distance of the end-effector from \na  target,  given  32  inputs  which  contain  information  on  the  joint  positions,  twist \nangles and so  forth. \n\nBoth  problems  are  nonlinear  and  contain  noisy  data.  Our  objective  is  to  com(cid:173)\npare  our  transductive  inference  method  directly  with  the  inductive  method  of \nRidge  Regression.  To  do  this  we  chose  the  set  of  basis  functions  \u00a2i(X)  = \nexp (-llx - xiI 12/2(2),  i  =  1, ... , e, and  found  the  values  of  )'  and  a  for  Ridge \nRegression which minimized the leave-one-out bound (15).  We then used the same \nvalues of these parameters in  our transductive approach, and using the basis  func(cid:173)\ntions \u00a2i(X)  =  exp (-llx - XiW /2(72)  , i  =  1, . . . , e + m,  we  then chose  a  fixed  value \nof)'* . \n\nFor  the  boston dataset  we  followed  the  same  experimental  setup  as  in  [4],  that \nis,  we  partitioned  the  training  set  of 506  observations  randomly  100  times  into  a \ntraining set of 481  observations and a  testing set of 25  observations.  We  chose  the \nvalues of)' and a by taking the minimum average leave- one-out error over five  more \nrandom  splits  of the  data stepping over  the  parameter  space.  The  minimum  was \nfound  at )'  =  0.005  and log  a  =  0.7.  For  our  transductive method,  we  also  chose \nthe parameter),*  =  10.  In Figure la we  plot mean squared error (MSE)  on the test \nset averaged over the 100 runs against log a  for Ridge Regression and Transductive \nRegression.  Transductive  Regression  outperforms  Ridge  Regression,  especially  at \nthe minimum. \n\nTo observe the influence of the number of test points m on the generalization ability \nof our  transductive  method,  we  ran  further  experiments,  setting  )'*  =  e/2m  for \n\n\f426 \n\n0.  Chapel/e,  V.  N.  Vapnik and J.  Weston \n\n9.2 \n\n, \n9  , \n\n1- Tranaductive  Regression 1 \n\n- _ .  Ridge  Regress.,n \n\n8r---~----~----~---------, \n\n\" \n\n7.95 \n\n7.9 \n\ng7.85 \nUJ \nW \n~ 7.8 \n\n7.75 \n\n7.7 \n\n7.65;L--~5:--=~10===-15==:\"2=0-----:!25 \n\nTest Set Size \n(b) \n\n0.135 \n\nO.13,r---~--=--- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -\n\n0.125 \n\n~  0.12 \nt: \n~0. 115 \n;l \nI- 0.11 \n\n0.105 \n\n0.1 \n\n0.095 \n\n8.8 \n\n8.6 \n\n8.4 \n\n8.2 \n\n8 \n\n7.8 \n\n7.6 \n\n0.4 \n\n0.6 \n\n0.8 \nLog sigma \n\n1.2 \n\n(a) \n\n, , , , \n, , \n1- Transductive Regression 1 :' \n, , \n, , , \n\n- - - RIdge RegreSSion \n\n/ \n\n, , \n\" \n\n, - --- , \n\n015 \n\n0.14 \n\n, \n, \n, , \n\n0.13 \n\ng \n~0. 12 \nQ) \nI-\n\n0.11 \n\n0.1 \n\n0.09 \n1 \n\n, \n, \n\n, , \n\n1.5 \n\n2.5 \n\n3 \n\n2 \nLog sigma \n(c) \n\n0.09''-------:'':50-----.,.1 0~0----1~50,------.,.200~---:2=-=-'50 \n\nTest Set Size \n(d) \n\nFigure  1:  A  comparison  of  Transductive  Regression  to  Ridge  Regression  on  the \nboston dataset:  (a)  error rates for  varying  (J',  (b)  varying the  test set size,  m,  and \non the kin-32fh dataset:  (c)  error rates for varying (J',  (d)  varying  the  test set size. \n\ndifferent  values  of m .  In  Figure  1b  we  plot  m  against MSE  on  the testing  set,  at \nlog  (J'  =  0.7.  The results  indicate  that  increasing the  test  set  size  gives  improved \nperformance in Transductive Regression.  For  Ridge  Regression,  of course,  the size \nof the testing set  has no  influence on the generalization ability. \nWe  then performed similar experiments on the kin-32fh dataset.  This time,  as we \nwere  interested in  large testing sets  giving  improved performance for  Transductive \nRegression  we  chose  100  splits  where  we  took a  subset  of only  64  observations for \ntraining and 256 for testing.  Again the leave-one-out estimator was used to find the \nwe  also  chose  the  parameter ,* =  0.1.  We  plotted MSE  on  the  testing set against \nvalues, = 0.1  and log  (J'  = 2 for  Ridge Regression, and for Transductive Regression \nlog  (J'  (Figure  1c)  and  the size  of the test set m  for  log  (J'  = 2.75  (also, ,* = 50/m) \n\n(Figure 1d) for  the two algorithms.  For large test set sizes our method outperforms \nRidge  Regression. \n\n4.2  Pattern Recognition \n\nThis technique can also be applied for pattern recognition problems by solving them \nbased  on  minimizing  functional  (8)  with  y  =  \u00b11.  Such  a  technique  is  known  as  a \nLinear Discriminant  (LD)  technique. \n\n\fTransductive Inference for Estimating Values  of Functions \n\n427 \n\nPostal \nBanana \nDiabetes \nTitanic \nBreast Cancer \nHeart \nThyroid \n\nAB  ABR  SVM  TLD \n4.7 \n-\n1l.4 \n23.3 \n22.4 \n25.7 \n15.7 \n4.0 \n\n5.5 \n11.5 \n23.5 \n22.4 \n26.0 \n16.0 \n4.8 \n\n12.3 \n26.5 \n22.6 \n30.4 \n20.3 \n4.4 \n\n-\n\n10.9 \n23.8 \n22.6 \n26.5 \n16.6 \n4.6 \n\nTable  1:  Comparison  of percentage  test  error  of AdaBoost  (A B) ,  Regularized  Ad(cid:173)\naBoost  (ABR),  Support  Vector Machines  (SVM)  and  Tmnsductive  Linear Discrim(cid:173)\nination  (TLD)  on  seven  datasets. \n\nTable  1 describes results of experiments on classification in the following  problems: \n2 class  digit  recognition  (0  - 4  versus 5 - 9)  splitting the training set into 23  runs \nof  317  observations  and  considering  a  testing  set  of  2000  observations,  and  six \nproblems  from  the  UCI  database.  We  followed  the  same  experimental  setup  as \nin  [3]: \nthe  performance  of a  classifier  is  measured  by  its  average  error  over  one \nhundred partitions of the datasets into training and testing sets.  Free parameter(s) \nare chosen via validation on the first five  training datasets.  The performance of the \ntransductive  LD  technique  was  compared  to Support  Vector  Machines,  AdaBoost \nand  Regularized  AdaBoost  [3]. \nIt is  interesting to note that in spite of the fact that LD  technique is  one of the sim(cid:173)\nplest pattern recognition techniques, transductive inference based upon this method \nperforms well  compared to state of the art methods of pattern recognition. \n\n5  Summary \n\nIn  this  article  we  performed  transductive  inference  in  the  problem  of estimating \nvalues  of functions  at  the  points  of interest.  We  demonstrate that  estimating  the \nunknown values via a one- step (transductive)  procedure can be more accurate than \nthe traditional two-step  (inductive plus deductive)  one. \n\nReferences \n\n[1]  A.  Hoerl and R. W. Kennard. Ridge regression:  Biased estimation for nonorthog(cid:173)\n\nonal problems.  Technometrics,  12(1):55-67, 1970. \n\n[2]  A.  Luntz and V.  Brailovsky.  On the estimation of characters obtained in statis(cid:173)\ntical procedure of recognition,.  Technicheskaya  Kibernetica,  1969.  [In  Russian]. \n[3]  G.  Witsch,  T.  Onoda, and  K.-R.  Muller.  Soft  margins for  adaboost.  Technical \n\nreport, Royal  Holloway,  University of London, 1998.  TR-98-2l. \n\n[4]  C.  Saunders,  A.  Gammermann,  and  V.  Vovk.  Ridge  regression  learning  algo(cid:173)\n\nrithm in dual variables.  In  Proccedings  of the  15th International  Conference  on \nMachine  Learning,  pages 515-52l. Morgan Kaufmann,  1998. \n\n[5]  V. Vapnik.  Estimating of values of regression at the point of interest.  In Method \n\nof Pattern Recognition.  Sovetskoe Radio,  1977.  [In  Russian]. \n\n[6]  V.  Vapnik.  Estimation  of Dependences  Based  on  Empirical  Data.  Springer(cid:173)\n\nVerlag, New  York,  1982. \n\n\f", "award": [], "sourceid": 1699, "authors": [{"given_name": "Olivier", "family_name": "Chapelle", "institution": null}, {"given_name": "Vladimir", "family_name": "Vapnik", "institution": null}, {"given_name": "Jason", "family_name": "Weston", "institution": null}]}