Part of Advances in Neural Information Processing Systems 17 (NIPS 2004)
Yoshitatsu Matsuda, Kazunori Yamaguchi
In this paper, linear multilayer ICA (LMICA) is proposed for extracting independent components from quite high-dimensional observed signals such as large-size natural scenes. There are two phases in each layer of LMICA. One is the mapping phase, where a one-dimensional mapping is formed by a stochastic gradient algorithm which makes more highly- correlated (non-independent) signals be nearer incrementally. Another is the local-ICA phase, where each neighbor (namely, highly-correlated) pair of signals in the mapping is separated by the MaxKurt algorithm. Because LMICA separates only the highly-correlated pairs instead of all ones, it can extract independent components quite efficiently from ap- propriate observed signals. In addition, it is proved that LMICA always converges. Some numerical experiments verify that LMICA is quite ef- ficient and effective in large-size natural image processing.
1 Introduction
Independent component analysis (ICA) is a recently-developed method in the fields of signal processing and artificial neural networks, and has been shown to be quite useful for the blind separation problem [1][2][3] [4]. The linear ICA is formalized as follows. Let s and A are N -dimensional source signals and N N mixing matrix. Then, the observed signals x are defined as x = As. (1)
The purpose is to find out A (or the inverse W ) when the observed (mixed) signals only are given. In other words, ICA blindly extracts the source signals from M samples of the observed signals as follows: ^ S = W X, (2)
http://www.graco.c.u-tokyo.ac.jp/~matsuda
where X is an N M matrix of the observed signals and ^ S is the estimate of the source signals. This is a typical ill-conditioned problem, but ICA can solve it by assuming that the source signals are generated according to independent and non-gaussian probability dis- tributions. In general, the ICA algorithms find out W by maximizing a criterion (called the contrast function) such as the higher-order statistics (e.g. the kurtosis) of every com- ponent of ^ S. That is, the ICA algorithms can be regarded as an optimization method of such criteria. Some efficient algorithms for this optimization problem have been proposed, for example, the fast ICA algorithm [5][6], the relative gradient algorithm [4], and JADE [7][8].
Now, suppose that quite high-dimensional observed signals (namely, N is quite large) are given such as large-size natural scenes. In this case, even the efficient algorithms are not much useful because they have to find out all the N 2 components of W . Recently, we pro- posed a new algorithm for this problem, which can find out global independent components by integrating the local ICA modules. Developing this approach in this paper, we propose a new efficient ICA algorithm named " the linear multilayer ICA algorithm (LMICA)." It will be shown in this paper that LMICA is quite efficient than other standard ICA algo- rithms in the processing of natural scenes. This paper is an extension of our previous works [9][10].
This paper is organized as follows. In Section 2, the algorithm is described. In Section 3, numerical experiments will verify that LMICA is quite efficient in image processing and can extract some interesting edge detectors from large natural scenes. Lastly, this paper is concluded in Section 4.