{"title": "Instance-Based Relevance Feedback for Image Retrieval", "book": "Advances in Neural Information Processing Systems", "page_first": 489, "page_last": 496, "abstract": null, "full_text": " \n\n Instance-Based Relevance Feedback for \n Image Retrieval \n\n\n \n Giorgio Giacinto and Fabio Roli \n Department of Electrical and Electronic Engineering \n University of Cagliari \n Piazza D'Armi, Cagliari Italy 09121 \n {giacinto,roli}@diee.unica.it \n\n\n Abstract \n\n High retrieval precision in content-based image retrieval can be \n attained by adopting relevance feedback mechanisms. These \n mechanisms require that the user judges the quality of the results of \n the query by marking all the retrieved images as being either \n relevant or not. Then, the search engine exploits this information to \n adapt the search to better meet user's needs. At present, the vast \n majority of proposed relevance feedback mechanisms are \n formulated in terms of search model that has to be optimized. Such \n an optimization involves the modification of some search \n parameters so that the nearest neighbor of the query vector contains \n the largest number of relevant images. In this paper, a different \n approach to relevance feedback is proposed. After the user \n provides the first feedback, following retrievals are not based on k-\n nn search, but on the computation of a relevance score for each \n image of the database. This score is computed as a function of two \n distances, namely the distance from the nearest non-relevant image \n and the distance from the nearest relevant one. Images are then \n ranked according to this score and the top k images are displayed. \n Reported results on three image data sets show that the proposed \n mechanism outperforms other state-of-the-art relevance feedback \n mechanisms. \n\n\n1 Introduction \n\nA large number of content-based image retrieval (CBIR) systems rely on the vector \nrepresentation of images in a multidimensional feature space representing low-level \nimage characteristics, e.g., color, texture, shape, etc. [1]. Content-based queries are \noften expressed by visual examples in order to retrieve from the database the images \nthat are \"similar\" to the examples. This kind of retrieval is often referred to as K \nnearest-neighbor retrieval. It is easy to see that the effectiveness of content-based \nimage retrieval systems (CBIR) strongly depends on the choice of the set of visual \nfeatures, on the choice of the \"metric\" used to model the user's perception of image \nsimilarity, and on the choice of the image used to query the database [1]. Typically, \nif we allow different users to mark the images retrieved with a given query as \n\n\f\n \n\n\nrelevant or non-relevant, different subsets of images will be marked as relevant. \nAccordingly, the need for mechanisms to adapt the CBIR system response based on \nsome feedback from the user is widely recognized. \n\nIt is interesting to note that while relevance feedback mechanisms have been first \nintroduced in the information retrieval field [2], they are receiving more attention in \nthe CBIR field (Huang). The vast majority of relevance feedback techniques \nproposed in the literature is based on modifying the values of the search parameters \nas to better represent the concept the user bears in mind. To this end, search \nparameters are computed as a function of the relevance values assigned by the user \nto all the images retrieved so far. As an example, relevance feedback is often \nformulated in terms of the modification of the query vector, and/or in terms of \nadaptive similarity metrics. [3]-[7]. Recently, pattern classification paradigms such \nas SVMs have been proposed [8]. Feedback is thus used to model the concept of \nrelevant images and adjust the search consequently. \n\nConcept modeling may be difficult on account of the distribution of relevant images \nin the selected feature space. \"Narrow domain\" image databases allows extracting \ngood features, so that images bearing similar concepts belong to compact clusters. \nOn the other hand, \"broad domain\" databases, such as image collection used by \ngraphic professionals, or those made up of images from the Internet, are more \ndifficult to subdivide in cluster because of the high variability of concepts [1]. In \nthese cases, it is worth extracting only low level, non-specialized features, and \nimage retrieval is better formulated in terms of a search problem rather then concept \nmodeling. \n\nThe present paper aims at offering an original contribution in this direction. Rather \nthen modeling the concept of \"relevance\" the user bears in mind, feedback is used to \nassign each image of the database a relevance score. Such a score depends only \nfrom two dissimilarities (distances) computed against the images already marked by \nthe user: the dissimilarity from the set of relevant images, and the dissimilarity from \nthe set of non-relevant images. Despite its computational simplicity, this mechanism \nallows outperforming state-of-the-art relevance feedback mechanisms both on \n\"narrow domain\" databases, and on \"broad domain\" databases. \n\nThis paper is organized as follows. Section 2 illustrates the idea behind the proposed \nmechanism and provides the basic assumptions. Section 3 details the proposed \nrelevance feedback mechanism. Results on three image data sets are presented in \nSection 4, where performances of other relevance feedback mechanisms are \ncompared. Conclusions are drawn in Section 5. \n\n2 Instance-based relevance estimation \n\nThe proposed mechanism has been inspired by classification techniques based on \nthe \"nearest case\" [9]-[10]. Nearest-case theory provided the mechanism to compute \nthe dissimilarity of each image from the sets of relevant and nonrelevant images. \nThe ratio between the nearest relevant image and the nearest non-relevant image has \nbeen used to compute the degree of relevance of each image of the database [11]. \nThe present section illustrates the rationale behind the use of the nearest-case \nparadigm. \n\nLet us assume that each image of the database has been represented by a number of \nlow-level features, and that a (dis)similarity measure has been defined so that the \nproximity between pairs of images represents some kind of \"conceptual\" similarity. \nIn other words, the chosen feature space and similarity metric is meaningful at least \nfor a restricted number of users. \n\n\f\n \n\n\nA search in image databases is usually performed by retrieving the k most similar \nimages with respect to a given query. The dimension of k is usually small, to avoid \ndisplaying a large number of images at a time. Typical values for k are between 10 \nand 20. However, as the \"relevant\" images that the user wishes to retrieve may not \nfit perfectly with the similarity metric designed for the search engine, the user may \nbe interested in exploring other regions of the feature space. To this end, the user \nmarks the subset of \"relevant\" images out of the k retrieved. Usually, such relevance \nfeedback is used to perform a new k-nn search by modifying some search \nparameters, i.e., the position of the query point, the similarity metric, and other \ntuning parameters [1]-[7]. Recent works proposed the use of support vector machine \nto learn the distribution of relevant images [8]. These techniques require some \nassumption about the general form of the distribution of relevant images in the \nfeature space. As it is difficult to make any assumption about such a distribution for \nbroad domain databases, we propose to exploit the information about the relevance \nof the images retrieved so far in a nearest-neighbor fashion. \n\nNearest-neighbor techniques, as used in statistical pattern recognition, case-based \nreasoning, or instance-based learning, are effective in all applications where it is \ndifficult to produce a high-level generalization of a \"class\" of objects [9]-[10],[12]-\n[13]. Relevance learning in content base image retrieval may well fit into this \ndefinition, as it is difficult to provide a general model that can be adapted to \nrepresent different concepts of similarity. In addition, the number of available cases \nmay be too small to estimate the optimal set of parameters for such a general model. \nOn the other hand, it can be more effective to use each \"relevant\" image as well as \neach \"non-relevant\" image, as \"cases\" or \"instances\" against which the images of \nthe database should be compared. Consequently, we assume that an image is as \nmuch as relevant as much as its dissimilarity from the nearest relevant image is \nsmall. Analogously, an image is as much as non-relevant as much as its dissimilarity \nfrom the nearest non-relevant image is small. \n\n3 Relevance Score Computation \n\nAccording to previous section, each image of the database can be thus characterized \nby a \"degree of relevance\" and a \"degree of non-relevance\" according to the \ndissimilarities from the nearest relevant image, and from the nearest non-relevant \nimage, respectively. However, it should be noted that these degrees should be \ntreated differently because only \"relevant\" images represent a \"concept\" in the \nuser's mind, while \"non-relevant\" images may represent a number of other concepts \ndifferent from user's interest. In other words, while it is meaningful to treat the \ndegree of relevance as a degree of membership to the class of relevant images, the \nsame does not apply to the degree of non-relevance. For this reason, we propose to \nuse the \"degree of non-relevance\" to weight the \"degree of relevance\". \n\nLet us denote with R the subset of indexes j {1,...,k} related to the set of relevant \nimages retrieved so far and the original query (that is relevant by default), and with \nNR the subset of indexes j (1,...,k} related to the set of non-relevant images \nretrieved so far. For each image I of the database, according to the nearest neighbor \nrule, let us compute the dissimilarity from the nearest image in R and the \ndissimilarity from the nearest image in NR. Let us denote these dissimilarities as \ndR(I) and dNR(I), respectively. The value of dR(I) can be clearly used to measure \nthe degree of relevance of image I, assuming that small values of dR(I) are related \nto very relevant images. On the other hand, the hypothesis that image I is relevant to \nthe user's query can be supported by a high value of dNR(I). Accordingly, we \ndefined the relevance score \n\n\f\n \n\n\n ! ( )$'1\nrelevance(I) = 1 + dR I (1) \n \"# dN (I)%&\n\nThis formulation of the score can be easily explained in terms of a distance-\nweighted 2-nn estimation of the posterior probability that image I is relevant. The 2 \nnearest neighbors are made up of the nearest relevant image, and the nearest non-\nrelevant image, while the weights are computed as the inverse of the distance from \nthe nearest neighbors. \n\nThe relevance score computed according to equation (1) is then used to rank the \nimages and the first k are presented to the user. \n\n4 Experimental results \n\nIn order to test the proposed method and compare it with other methods described in \nthe literature, three image databases have been used: the MIT database, a database \ncontained in the UCI repository, and a subset of the Corel database. These databases \nare currently used for assessing and comparing relevance feedback techniques \n[5],[7],[14]. \n\nThe MIT database was collected by the MIT Media Lab \n(ftp://whitechapel.media.mit.edu/pub/VisTex). This database contains 40 texture \nimages that have been manually classified into fifteen classes. Each of these images \nhas been subdivided into sixteen non-overlapping images, obtaining a data set with \n640 images. Sixteen Gabor filters were used to characterise these images, so that \neach image is represented by a 16-dimensional feature vector [14]. \n\nThe database extracted from the UCI repository \n(http://www.cs.uci.edu/mlearn/MLRepository.html) consists of 2,310 outdoor \nimages. The images are subdivided into seven data classes (brickface, sky, foliage, \ncement, window, path, and grass). Nineteen colour and spatial features characterise \neach image. (Details are reported in the UCI web site). \n\nThe database extracted from the Corel collection is available at the KDD-UCI \nrepository (http://kdd.ics.uci.edu/databases/CorelFeatures/CorelFeatures.data.html). \nWe used a subset made up of 19513 images, manually subdivided into 43 classes. \nFor each image, four sets of features were available at the web site. In this paper, we \nreport the results related to the Color Moments (9 features), and the Co-occurrence \nTexture (16 features) feature sets \n\nFor each dataset, the Euclidean distance metric has been used. A linear \nnormalisation procedure has been performed, so that each feature takes values in the \nrange between 0 and 1. \n\nFor the first two databases, each image is used as a query, while for the Corel \ndatabase, 500 images have been randomly extracted and used as query, so that all \nthe 43 classes are represented. At each retrieval iteration, twenty images are \nreturned. Relevance feedback is performed by marking images belonging to the \nsame class of the query as relevant, and all other images as non-relevant. The user's \nquery itself is included in the set of relevant images. This experimental set up \naffords an objective comparison among different methods, and is currently used by \nmany researchers [5],[7],[14]. Results are evaluated in term of the retrieval \nprecision averaged over all the considered queries. The precision is measured as the \nfraction of relevant images contained in the 20 top retrieved images. \n\n\f\n \n\n\nAs the first two databases are of the \"narrow domain\" type, while the third is of the \n\"broad domain\" type, this experimental set-up allowed a thorough testing of the \nproposed technique. \n\nFor the sake of comparison, retrieval performances obtained with two methods \nrecently described in the literature are also reported: MindReader [3] which \nmodifies the query vector and the similarity metric on account of features relevance, \nand Bayes QS (Bayesian Query Shifting) which is based on query reformulation [7]. \nThese two methods have been selected because they can be easily implemented, and \ntheir performances can be compared to those provided by a large number of \nrelevance feedback techniques proposed in the CBIR literature (see for example \nresults presented in [15]). It is worth noting that results presented in different papers \ncannot be directly compared to each other because they are not related to a common \nexperimental set-up. However, as they are related to the same data sets with similar \nexperimental set-up, a qualitative comparisons let us conclude that the performance \nof the two above techniques are quite close to other results in the literature. \n\n4 . 1 E x p e r i m e n t s w i t h t h e M I T d a t a b a s e \n\nThis database can be considered of the \"narrow domain\" type as it contains only \nimages of textures of 40 different types. In addition, the selected feature space is \nvery suited to measure texture similarity. \n\nFigure 1 show the performances of the proposed relevance feedback mechanism and \nthose of the two techniques used for comparison. \n\n 100\n\n\n\n\n 95\n\n\n\n 90 Relevance\n Score\n Bayes QS\n\n % Precision 85 MindReader\n\n\n\n 80\n\n\n\n\n 75\n 0 rf 1 rf 2 rf 3 rf 4 rf 5 rf 6 rf 7 rf 8 rf\n Iter. Rel. Feedback\n \n Figure 1: Retrieval Performances for the MIT database in terms of average \n percentage retrieval precision. \n\n \n\nAfter the first feedback iteration (1rf in the graph), each relevance feedback \nmechanism is able to improve the average precision attained in the first retrieval by \nmore than 10%, the proposed mechanism performing slightly better than \nMindReader. This is a desired behaviour as a user typically allows few iterations. \nHowever, if the user aims to better refine the search by additional feedback \niteration, MindReader and Bayes QS are not able to exploit the additional \ninformation, as they provide no improvements after the second feedback iteration. \nOn the other hand, the proposed mechanism provides further improvement in \nprecision by increasing the number of iteration. These improvements are very small \n\n\f\n \n\n\nbecause the first feedback already provides a high precision value, near to 95%. \n\n4 . 2 E x p e r i m e n t s w i t h t h e U C I d a t a b a s e \n\nThis database too can be considered of the \"narrow domain\" type as the images \nclearly belong to one of the seven data classes, and features have been extracted \naccordingly. \n\n 100\n\n\n\n\n 98\n\n\n\n\n 96 Relevance\n Score\n Bayes QS\n\n % Precision 94 MindReader\n\n\n\n 92\n\n\n\n\n 90\n 0 rf 1 rf 2 rf 3 rf 4 rf 5 rf 6 rf 7 rf 8 rf\n Iter. Rel. Feedback \n Figure 2: Retrieval Performances for the UCI data set in terms of average \n percentage retrieval precision. \n\nFigure 2 show the performances attained on the UCI database. Retrieval precision is \nvery high after the first extraction with no feedback. Nonetheless, each of the \nconsidered mechanism is able to exploit relevance feedback, Mindreader and Bayes \nQS providing a 6% improvement, while the proposed mechanism attains a 8% \nimprovement. This example clearly shows the superiority of the proposed technique, \nas it attains a precision of 99% after the second iteration. Further iterations allow \nattaining a 100% precision. On the other hand, Bayes QS also exploits further \nfeedback iteration attaining a precision of 98% after 7 iterations, while MindReader \ndoes not improve the precision attained after the first iteration. As the user typically \nallows very few feedback iterations, the proposed mechanism proved to be very \nsuited for narrow domain databases as it allows attaining a precision close to 100%. \n\n4 . 3 E x p e r i m e n t s w i t h t h e C o r e l d a t a b a s e \n\nFigures 3 and 4 show the performances attained on two feature sets extracted from \nthe Corel database. This database is of the \"broad domain\" type as images represent \na very large number of concepts, and the selected feature sets represent conceptual \nsimilarity between pairs of images only partly. \n\nReported results clearly show the superiority of the proposed mechanism. Let us \nnote that the retrieval precision after the first k-nn search (0rf in the graphs) is quite \nsmall. This is a consequence of the difficulty of selecting a good feature space to \nrepresent conceptual similarity between pairs of images in a broad domain database. \nThis difficulty is partially overcome by using MindReader or Bayes QS as they \nallow improving the retrieval precision by 10% to 15% according to the number of \niteration allowed, and according to the selected feature space. Let us recall that both \nMindReader and Bayes QS perform a query movement in order to perform a k-nn \n\n\f\n \n\n\nquery on a more promising region of the feature space. On the other hand, the \nproposed mechanism based on ranking all the images of the database according to a \nrelevance score, not only provided higher precision after the first feedback, but also \nallow to improve significantly the retrieval precision as the number of iteration is \nincreased. As the initial precision is quite small, a user may have more willingness \nto perform further iterations as the proposed mechanism allows retrieving new \nrelevant images. \n\n\n\n\n\n \n\nFigure 3: Retrieval Performances for the Corel data set (Color Moments feature set) \n in terms of average percentage retrieval precision \n\n\n\n\n\n \n Figure 4: Retrieval Performances for the Corel data set (Co-occurrence Texture \n feature set) in terms of average percentage retrieval precision. \n\n5 Conclusions \n\nIn this paper, we proposed a novel relevance feedback technique for content-based \nimage retrieval. While the vast majority of relevance feedback mechanisms aims at \nmodeling user's concept of relevance based on the available labeled samples, the \nproposed mechanism is based on ranking the images according to a relevance score \ndepending on the dissimilarity from the nearest relevant and non-relevant images. \n\n\f\n \n\n\nThe rationale behind our choice is the same of case-based reasoning, instance-based \nlearning, and nearest-neighbor pattern classification. These techniques provide good \nperformances when the number of available training samples is too small to use \nstatistical techniques. This is the case of relevance feedback in CBIR, where the use \nof classification models should require a suitable formulation in order to avoid so-\ncalled \"small sample\" problems. \n\nReported results clearly showed the superiority of the proposed mechanism \nespecially when large databases made up of images related to many different \nconcepts are searched. In addition, while many relevance feedback techniques \nrequire the tuning of some parameters, and exhibit high computational complexity, \nthe proposed mechanism does not require any parameter tuning, and exhibit a low \ncomputational complexity, as a number of techniques are available to speed-up \ndistance computations. \n\nR e f e r e n c e s \n\n[1] Smeulders A.W.M., Worring M., Santini S., Gupta A., Jain R.: Content-based image \nretrieval at the end of the early years. IEEE Trans. on Pattern Analysis and Machine \nIntelligence 22(12) (2000) 1349-1380 \n\n[2] G. Salton and M.J. McGill, Introduction to modern information retrieval, New York, \nMcGraw-Hill, 1988. \n\n[3] Ishikawa Y., Subramanys R., Faloutsos C.: MindReader: Querying databases through \nmultiple examples. In Proceedings. of the 24th VLDB Conference (1998) 433-438 \n\n[4] Santini S., Jain R.: Integrated browsing and querying for image databases. IEEE Multimedia \n7(3) (2000) 26-39 \n\n[5] Rui Y., Huang T.S.: Relevance Feedback Techniques in Image retrieval. In Lew M.S. (ed.): \nPrinciples of Visual Information Retrieval. Springer, London, (2001) 219-258 \n\n[6] Sclaroff S., La Cascia M., Sethi S., Taycher L.: Mix and Match Features in the ImageRover \nsearch engine. In Lew M.S. (ed.): Principles of Visual Information Retrieval. Springer-Verlag, \nLondon (2001) 219-258 \n\n[7] Giacinto G., Roli F.: Bayesian relevance feedback for content-based image retrieval. Pattern \nRecognition 37(7) (2004) 1499-1508 \n\n[8] Zhou X.S. and Huang T.S.: Relevance feedback in image retrieval: a comprehensive review, \nMultimedia Systems 8(6) (2003) 536-544 \n\n[9] Aha D.W., Kibler D., Albert M.K. Instance Based learning Algorithms. Machine Learning, 6, \n(1991) 37-66 \n\n[10] Althoff K-D. Case-Based Reasoning. In Chang S.K. (ed.) Handbook on Software Engineering \nand Knowledge Engineering, World Scientific (2001), 549-588. \n\n[11] Bloch I. Information Combination Operators for Data Fusion: A Comparative Review with \nClassification. IEEE Trans. on System, Man and Cybernetics - Part A, 26(1) (1996) 52-67 \n\n[12] Duda R.O., Hart P.E., and Stork D.G.: Pattern Classification. John Wiley and Sons, Inc., New \nYork, 2001 \n\n[13] Hastie T., Tibshrirani R., and Friedman J.: The Elements of Statistical Learning. Springer, \nNew York, 2001 \n\n[14] Peng J., Bhanu B., Qing S., Probabilistic feature relevance learning for content-based \nimage retrieval, Computer Vision and Image Understanding 75 (1999) 150-164. \n\n[15] He J., Li M., Zhang H-J, Tong H., Zhang C, Mean Version Space: a New Active Learning \nMethod for Content-Based Image Retrieval, Proc. of MIR 2004, New York, USA. (2004) 15-22. \n\n\f\n", "award": [], "sourceid": 2555, "authors": [{"given_name": "Giorgio", "family_name": "Gia\\-cin\\-to", "institution": null}, {"given_name": "Fabio", "family_name": "Roli", "institution": null}]}