Part of Advances in Neural Information Processing Systems 7 (NIPS 1994)
Anthony Bell, Terrence J. Sejnowski
A new learning algorithm is derived which performs online stochas(cid:173) tic gradient ascent in the mutual information between outputs and inputs of a network. In the absence of a priori knowledge about the 'signal' and 'noise' components of the input, propagation of information depends on calibrating network non-linearities to the detailed higher-order moments of the input density functions. By incidentally minimising mutual information between outputs, as well as maximising their individual entropies, the network 'fac(cid:173) torises' the input into independent components. As an example application, we have achieved near-perfect separation of ten digi(cid:173) tally mixed speech signals. Our simulations lead us to believe that our network performs better at blind separation than the Herault(cid:173) J utten network, reflecting the fact that it is derived rigorously from the mutual information objective.
468
Anthony J. Bell, Terrence J. Sejnowski