Comparing Biases for Minimal Network Construction with Back-Propagation

Part of Advances in Neural Information Processing Systems 1 (NIPS 1988)

Bibtex Metadata Paper

Authors

Stephen Hanson, Lorien Pratt

Abstract

learning

representations during

Rumelhart (1987). has proposed a method for choosing minimal or "simple" in Back-propagation networks. This approach can be used to (a) dynamically select the number of hidden units. (b) construct a representation that is appropriate for the problem and (c) thus improve the generalization ability of Back-propagation networks. The method Rumelhart suggests involves adding penalty terms to the usual error function. In this paper we introduce RumelhartĀ·s minimal networks idea and compare two possible biases on the weight search space. These biases are compared in both simple counting problems and a speech recognition problem. In general. the constrained search does seem to minimize the number of hidden units required with an expected increase in local minima.