{"title": "A NEURAL NETWORK CLASSIFIER BASED ON CODING THEORY", "book": "Neural Information Processing Systems", "page_first": 174, "page_last": 183, "abstract": "", "full_text": "174 \n\nA Neural Network Classifier Based on Coding Theory \n\nTzt-Dar Chlueh and Rodney Goodman \n\neanrornla Instltute of Technology. Pasadena. eanromla 91125 \n\nABSTRACT \n\nThe new neural network classifier we propose transforms the \nclassification problem into the coding theory problem of decoding a noisy \ncodeword. An input vector in the feature space is transformed into an internal \nrepresentation which is a codeword in the code space, and then error correction \ndecoded in this space to classify the input feature vector to its class. Two classes \nof codes which give high performance are the Hadamard matrix code and the \nmaximal length sequence code. We show that the number of classes stored in an \nN-neuron system is linear in N and significantly more than that obtainable by \nusing the Hopfield type memory as a classifier. \n\nI. INTRODUCTION \n\nAssociative recall using neural networks has recently received a great deal \nof attention. Hopfield in his papers [1,2) deSCribes a mechanism which iterates \nthrough a feedback loop and stabilizes at the memory element that is nearest the \ninput, provided that not many memory vectors are stored in the machine. He has \nalso shown that the number of memories that can be stored in an N-neuron \nsystem is about O.15N for N between 30 and 100. McEliece et al. in their work (3) \nshowed that for synchronous operation of the Hopfield memory about N /(2IogN) \ndata vectors can be stored reliably when N is large. Abu-Mostafa (4) has predicted \nthat the upper bound for the number of data vectors in an N-neuron Hopfield \nmachine is N. We believe that one should be able to devise a machine with M, the \nnumber of data vectors, linear in N and larger than the O.15N achieved by the \nHopfield method. \n\nFeature Space = B = {-1 , 1 } \n\nN \n\nN \n\nL \nCode Space = B = {-1 \u2022 1 } \n\nL \n\nFigure 1 (a) Classification problems versus (b) Error control decoding problems \n\nIn this paper we are specifically concerned with the problem of \nclassification as in pattern recognition. We propose a new method of building a \nneural network classifier, based on the well established techniques of error \ncontrol coding. ConSider a typical classification problem (Fig. l(a)), in which one \nis given a priori a set of classes, C( a), a = 1, .... M. Associated with each class is a \nfeature vector which labels the class ( the exemplar of the class), I.e. it is the \n\n\u00a9 American Institute of Physics 1988 \n\n\f175 \n\nmost representative point in the class region. The input is classified into the \nclass with the nearest exemplar to the input. Hence for each class there is a \nregion in the N-dimensional binary feature space BN == (I. -I}N. in which every \nvector will be classified to the corresponding class. \n\nA similar problem is that of decoding a codeword in an error correcting \ncode as shown in Fig. I(b). In this case codewords are constructed by design and \nare usually at least dmtn apart. The received corrupted codeword is the input to \nthe decoder. which then finds the nearest codeword to the input. In principle \nthen. if the distance between codewords is greater than 2t +1. it is possible to \ndecode (or classify) a noisy codeword (feature vector) into the correct codeword \n(exemplar) provided that the Hamming distance between the noisy codeword and \nthe correct codeword is no more than t. Note that there is no guarantee that the \nexemplars are uniformly distributed in BN. consequently the attraction radius \n(the maximum number of errors that can occur in any given feature vector such \nthat the vector can st111 be correctly classified) will depend on the minimum \ndistance between exemplars. \n\nMany solutions to the minimum Hamming distance classification have \nbeen proposed. the one commonly used is derived from the idea of matched filters \nin communication theory. Lippmann [5) proposed a two-stage neural network \nthat solves this classification problem by first correlating the input with all \nexemplars and then picking the maximum by a \"winner-take-all\" circuit or a \nnetwork composed of two-input comparators. In Figure 2. fI.f2 .... .fN are the N \ninput bits. and SI.S2 .... SM are the matching score s(Similartty) of f with the M \nexemplars. The second block picks the maximum of sI.S2 ..... SM and produces the \nindex of the exemplar with the largest score. The main disadvantage of such a \nclassifier is the complexity of the maximum-picking circuit. for example a \n''winner-take-all'' net needs connection weights of large dynamic range and \ngraded-response neurons. whilst the comparator maximum net demands M-I \ncomparators organized in log2M stages. \n\nM \nA \nX \nI \nM \nU \nM \n\n(0:) \n\nf = d + e \n\n\u2022 \ng = c + e \n\n(0:) \n\n,,-_;_~ .... ~ DECODER~SS(f) \n\ncloSS(f) \n\nFeature \nSpace \n\nCode \nSpace \n\nFig. 2 A matched filter type classifier Fig. 3 Structure of the proposed classifier \n\nOur main idea is thus to transform every vector in the feature space to a \nvector in some code space in such a way that every exemplar corresponds to a \ncodeword in that code. The code should preferably (but not necessarily) have the \nproperty that codewords are uniformly distributed in the code space. that is, the \nHamming distance between every pair of codewords is the same. With this \ntransformation. we turn the problem of classification into the coding problem of \ndecoding a noisy codeword. We then do error correction decoding on the vector in \nthe code space to obtain the index of the noisy codeword and hence classify the \noriginal feature vector. as shown in Figure 3. \n\nThis paper develops the construction of such a classification machine as \nfollows. First we conSider the problem of transforming the input vectors from the \nfeature space to the code space. We describe two hetero-associative memories for \ndOing this. the first method uses an outer product matrix technique Similar to \n\n\f176 \n\nthat of Hopfield's. and \nthe second method generates its matrix by the \npseudoinverse techruque[S.7J. Given that we have transformed the problem of \nassociative recall. or classification. into the problem of decoding a noisy \ncodeword. we next consider suitable codes for our machine. We require the \ncodewords in this code to have the property of orthogonality or \npseudo-orthogonality. that is. the ratio of the cross-correlation to the \nauto-correlation of the codewords is small. We show two classes of such good \ncodes for this particular decoding problem l.e. the Hadamard matrix codes. and \nthe maximal length sequence codes[8J. We next formulate the complete decoding \nalgorithm. and describe the overall structure of the classifier in terms of a two \nlayer neural network. The first layer performs the mapping operation on the \ninput. and the second one decodes its output to produce the index of the class to \nwhich the input belongs. \n\nThe second part of the paper is concerned with the performance of the \nclassifier. We first analyze the performance of this new classifier by finding the \nrelation between the maximum number of classes that can be stored and the \nclassification error rate. We show (when using a transform based on the outer \nproduct method) that for negligible misclassification rate and large N. a not very \ntight lower bound on M. the number of stored classes. is 0.22N. We then present \ncomprehensive simulation results that confirm and exceed our theoretical \nexpectations. The Simulation results compare our method with the Hopfield \nmodel for both the outer product and pseudo-inverse method. and for both the \nanalog and hard limited connection matrices. In all cases our classifier exceeds \nthe performance of the Hopfield memory in terms of the number of classes that \ncan be reliably recovered. \n\nD. TRANSFORM TECHNIQUES \n\nOur objective is to build a machine that can discriminate among input \nSuppose \nvectors and classify each one of them into the appropriate class. \nd(a) E BN is the exemplar ofthe corresponding class e(a.). a. = 1.2 ..... M . Given the \ninput f . we want the machine to be able to identify the class whose exemplar is \nclosest to f. that is. we want to calculate the follOWing function. \n\nclass ( f) = a. \n\nif f \n\nI f - d( a) I < I f - dH3) I \n\nwhere I I denotes Hamming distance in BN. \n\nWe approach the problem by seeking a transform ~ that maps each \nexemplar d(a) in BN to the corresponding codeword w(a) in BL. And an input \nfeature vector f = dey) + e is thus mapped to a noisy codeword g = wlY) + e' where e \nis the error added to the exemplar, and e' is the corresponding error pattern in the \ncode space. We then do error correction decoding on g to get the index of the \ncorresponding codeword. Note that e' may not have the same Hamming weight as \ne, that is, the transformation ~ may either generate more errors or eliminate \nerrors that are present in the original input feature vector. We require ~ to \nsatisfy the following equation, \n\nand ~ will be implemented uSing a Single-layer feedfoIWard network. \n\n0.=0,1 ..... M-l \n\n\f177 \n\nThus we first construct a matrix according to the sets of d(a)'s and w(a)'s, call it T, \nand define r:, as \n\nwhere sgn is the threshold operator that maps a vector in RL to BL and R is the \nfield of real numbers. \n\nLet D be an N x M matrix whose