Part of Advances in Neural Information Processing Systems 8 (NIPS 1995)
David J. Miller, Ajit Rao, Kenneth Rose, Allen Gersho
A new learning algorithm is developed for the design of statistical classifiers minimizing the rate of misclassification. The method, which is based on ideas from information theory and analogies to statistical physics, assigns data to classes in probability. The dis(cid:173) tributions are chosen to minimize the expected classification error while simultaneously enforcing the classifier's structure and a level of "randomness" measured by Shannon's entropy. Achievement of the classifier structure is quantified by an associated cost. The con(cid:173) strained optimization problem is equivalent to the minimization of a Helmholtz free energy, and the resulting optimization method is a basic extension of the deterministic annealing algorithm that explicitly enforces structural constraints on assignments while re(cid:173) ducing the entropy and expected cost with temperature. In the limit of low temperature, the error rate is minimized directly and a hard classifier with the requisite structure is obtained. This learn(cid:173) ing algorithm can be used to design a variety of classifier structures. The approach is compared with standard methods for radial basis function design and is demonstrated to substantially outperform other design methods on several benchmark examples, while of(cid:173) ten retaining design complexity comparable to, or only moderately greater than that of strict descent-based methods.
592
D. NnLLER.A.RAO.K. ROSE.A. GERSHO