{"title": "New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit", "book": "Advances in Neural Information Processing Systems", "page_first": 273, "page_last": 279, "abstract": null, "full_text": "New Approximations of Differential \nEntropy for Independent Component \n\nAnalysis and Projection Pursuit \n\nAapo Hyvarinen \n\nHelsinki University of Technology \n\nLaboratory of Computer and Information Science \n\nP.O. Box 2200, FIN-02015 HUT, Finland \n\nEmail: aapo.hyvarinen 2, a density of \nnegative kurtosis. Thus the densities in this family can be used as examples of dif(cid:173)\nferent symmetric non-Gaussian densities. In Figure 2, the different approximations \nare plotted for this family, using parameter values .5 ::; a ~ 3. Since the densi(cid:173)\nties used are all symmetric, the first terms in the approximations were neglected. \nAgain, it is clear that both of the approximations Ha and Hb introduced in Section 7 \nwere much more accurate than the cumulant-based approximation in [2, 9]. (In the \ncase of symmetric densities, these two cumulant-based approximations are identi(cid:173)\ncal). Especially in the case of sparse densities (or densities of positive kurtosis), the \ncumulant-based approximations performed very poorly; this is probably because it \ngives too much weight to the tails of the distribution. \n\nReferences \n[1] S. Amari, A. Cichocki, and H.H. Yang. A new learning algorithm for blind source sep(cid:173)\n\naration. In D . S. Touretzky, M. C. Mozer, and M. E . Hasselmo, editors, Advances in \nNeural Information Processing 8 (Proc. NIPS '95), pages 757- 763. MIT Press, Cam(cid:173)\nbridge, MA, 1996. \n\n12] P. Comon. Independent component analysis - a new concept? Signal Processing, \n\n36:287- 314, 1994. \n\n\fNew Approximationr of Differential Entropy \n\n279 \n\nFigure 1: Comparison of different ap(cid:173)\nproximations of negentropy, for the family \nof mixture densities in (10) parametrized \nby JL ranging from 0 to 1. Solid curve: \ntrue negentropy. Dotted curve: cumulant(cid:173)\nbased approximation. Dashed curve: ap(cid:173)\nproximation Ha in (8). Dot-dashed Cu:Lve: \napproximation Hb in (9). Our two ap(cid:173)\nproximations were clearly better than the \ncumulant-based one. \n\n. . ' \n\n\"-\n\n\"', ... \n\n\" , , \n\\' \n\" \" \n\n... :- - .... \n\n-: \"\" \n\n0025 \n\n0 01 $ \n\no. , \n\n0 6\n\n. \n\n, ' \n\n'. \n\n, \" -. , '. , \n, ' \n\"':\" \n\n,' ...... \n\n~~ .. --\n\n\"\"-.... \" -:--~ ~--~-.---\n\n, . \n\nFigure 2: Comparison of different approximations of negentropy, for the family of densities \n(11) parametrized by Q. On the left, approximations for densities of positive kurtosis (.5 ~ \nQ < 2) are depicted, and on the right, approximations for densities of negative kurtosis \n(2 < Q ~ 3). Solid curve: true negentropy. Dotted curve: cumulant-based approximation. \nDashed curve: approximation Ha in (8). Dot-dashed curve: approximation Hb in (9) . \nClearly, our two approximations were much better than the cumulant-based one, especially \nin the case of densities of positive kurtosis. \n\n[3) D. Cook, A. Buja, and J. Cabrera. Projection pursuit indexes based on orthonormal \nfunction expansions. J. of Computational and Graphical Statistics, 2(3) :225-250, 1993. \n[4) T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley & \n\nSons, 1991. \n\n[5) J.H. Friedman. Exploratory projection pursuit. J. of the American Statistical Asso(cid:173)\n\nciation, 82(397):249-266, 1987. \n\n[6) P.J. Huber. Projection pursuit. The Annals of Statistics, 13(2):435-475, 1985. \n[7) A. Hyvarinen. Independent component analysis by minimization of mutual infor(cid:173)\n\nmation. Technical Report A46, Helsinki University of Technology, Laboratory of \nComputer and Information Science, 1997. \n\n[8) A. Hyviirinen. New approximations of differential entropy for independent compo(cid:173)\n\nnent analysis and projection pursuit. Technical Report A47, Helsinki University of \nTechnology, Laboratory of Computer and Information Science, 1997. Available at \nhttp://www.cis.hut.fi;-aapo. \n\n[9) M.C. Jones and R. Sibson. What is projection pursuit ? J. of the Royal Statistical \n\nSociety, ser. A , 150:1-36, 1987. \n\n[10) C. Jutten and J. Herault. Blind separation of sources, part I: An adaptive algorithm \n\nbased on neuromimetic architecture. Signal Processing, 24:1-10, 1991. \n\n[11) M. Kendall and A. Stuart. The Advanced Theory of Statistics. Charles Griffin & \n\nCompany, 1958. \n\n\f", "award": [], "sourceid": 1408, "authors": [{"given_name": "Aapo", "family_name": "Hyv\u00e4rinen", "institution": null}]}