{"title": "Rate Distortion Function in the Spin Glass State: A Toy Model", "book": "Advances in Neural Information Processing Systems", "page_first": 423, "page_last": 430, "abstract": null, "full_text": "Rate Distortion Function in the Spin Glass State:\n\na Toy Model\n\nTatsuto Murayama and Masato Okada\nLaboratory for Mathematical Neuroscience\n\nRIKEN Brain Science Institute\n\nSaitama, 351-0198, JAPAN\n\nfmurayama,okadag@brain.riken.go.jp\n\nAbstract\n\nWe applied statistical mechanics to an inverse problem of linear mapping\nto investigate the physics of optimal lossy compressions. We used the\nreplica symmetry breaking technique with a toy model to demonstrate\nShannon\u2019s result. The rate distortion function, which is widely known\nas the theoretical limit of the compression with a \ufb01delity criterion, is\nderived. Numerical study shows that sparse constructions of the model\nprovide suboptimal compressions.\n\n1\n\nIntroduction\n\nMany information-science studies are very similar to those of statistical physics. Statistical\nphysics and information science may have been expected to be directed towards common\nobjectives since Shannon formulated an information theory based on the concept of en-\ntropy. However, envisaging how this actually happened would have been dif\ufb01cult; that the\nphysics of disordered systems, and spin glass theory in particular, at its maturity naturally\nincludes some important aspects of information sciences, thus reuniting the two disciplines.\nThis cross-disciplinary \ufb01eld can thus be expected to develop much further beyond current\nperspectives in the future [1].\n\nThe areas where these relations are particularly strong are Shannon\u2019s coding theory [2] and\nclassical spin systems with quenched disorder, which is the replica theory of disordered\nstatistical systems [3]. Triggered by the work of Sourlas [4], these links have recently\nbeen examined in the area of matrix-based error corrections [5, 6], network-based com-\npressions [7], and turbo decoding [8]. Recent results of these topics are mostly based on\nthe replica technique. Without exception, their basic characteristics (such as channel ca-\npacity, entropy rate, or achievable rate region) are only captured by the concept of a phase\ntransition with a \ufb01rst-order jump between the optimal and the other solutions arising in the\nscheme.\n\nHowever, the research in the cross-disciplinary \ufb01eld so far can be categorized as a so-called\n\u2018zero-distortion\u2019 decoding scheme in terms of information theory: the system requires per-\nfect reproduction of the input alphabets [2]. Here, the same spin glass techniques should\nbe useful to describe the physics of systems with a \ufb01delity criterion; i.e., a certain degree\nof information distortion is assumed when reproducing the alphabets. This framework is\n\n\fcalled the rate distortion theory [9, 10]. Though processing information requires regard-\ning the concept of distortions practically, where input alphabets are mostly represented by\ncontinuous variables, statistical physics only employs a few approaches [11, 12].\n\nIn this paper, we introduce a prototype that is suitable for cross-disciplinary study. We\nanalyze how information distortion can be described by the concepts of statistical physics.\nMore speci\ufb01cally, we study the inverse problem of a Sourlas-type decoding problem by us-\ning the framework of replica symmetry breaking (RSB) of diluted disordered systems [13].\nAccording to our analysis, this simple model provides an optimal compression scheme\nfor an arbitrary \ufb01delity-criterion degree, though the encoding procedure remains an NP-\ncomplete problem without any practical encoders.\n\nThe paper is organized as follows.\nIn Section 2, we brie\ufb02y review the concept of the\nrate distortion theory as well as the main results related to our purpose. In Section 3, we\nintroduce a toy model. In Section 4, we obtain consistent results with information theory.\nConclusions are given in the last section. Detailed derivations will be reported elsewhere.\n\n2 Review: Rate Distortion Theory\n\nWe brie\ufb02y recall the de\ufb01nitions of the concepts of the rate distortion theory and state the\nsimplest version of the main result at the end of this section. Let J be a discrete ran-\ndom variable with alphabet J . Assume that we have a source that produces a sequence\nJ1; J2; (cid:1) (cid:1) (cid:1) ; JM , where each symbol is randomly drawn from a distribution. We will as-\nsume that the alphabet is \ufb01nit. Throughout this paper we use vector notation to represent\nsequences for convenience of explanation: J = (J1; J2; (cid:1) (cid:1) (cid:1) ; JM )T 2 J M . Here, the\nencoder describes the source sequence J 2 J M by a codeword (cid:24) = f (J ) 2 X N . The\ndecoder represents J by an estimate ^J = g((cid:24)) 2 ^J M , as illustrated in Figure 1. Note that\nM represents the length of a source sequence, while N represents the length of a codeword.\nHere, the rate is de\ufb01ned by R = N=M. Note that the relation N < M always holds when\na compression is considered; therefore, R < 1 also holds.\n\nDe\ufb01nition 2.1 A distortion function is a mapping\n\nd : J (cid:2) ^J ! R+\n\n(1)\n\nfrom the set of source alphabet-reproduction alphabet pairs into the set of non-negative\nreal numbers.\n\nIntuitively, the distortion d(J; ^J) is a measure of the cost of representing the symbol J by\nthe symbol ^J. This de\ufb01nition is quite general. In most cases, however, the reproduction\nalphabet ^J is the same as the source alphabet J . Hereafter, we set ^J = J and the\nfollowing distortion measure is adopted as the \ufb01delity criterion:\n\nDe\ufb01nition 2.2 The Hamming distortion is given by\n\nd(J; ^J) =(0\n\n1\n\nif J = ^J\nif J 6= ^J\n\n;\n\n(2)\n\n,\nwhich results in a probable error distortion, since the relation E[d(J; ^J)] = P[J 6= ^J]\nholds, where E[(cid:1)] represents the expectation and P[(cid:1)] the probability of its argument. The\ndistortion measure is so far de\ufb01ned on a symbol-by-symbol basis. We extend the de\ufb01nition\nto sequences:\n\n\fDe\ufb01nition 2.3 The distortion between sequences J ; ^J 2 J M is de\ufb01ned by\n\nd(J ; ^J ) =\n\n1\nM\n\nM\n\nXj=1\n\nd(Jj; ^Jj) :\n\n(3)\n\nTherefore, the distortion for a sequence is the average distortion per symbol of the elements\nof the sequence.\n\nDe\ufb01nition 2.4 The distortion associated with the code is de\ufb01ned as\n\nwhere the expectation is with respect to the probability distribution on J .\n\nD = E[d(J ; ^J )] ;\n\n(4)\n\nA rate distortion pair (R; D) should be achiebable if a sequence of rate distortion codes\n(f; g) exist with E[d(J ; ^J )] (cid:20) D in the limit M ! 1. Moreover, the closure of the set\nof achievable rate distortion pairs is called the rate distortion region for a source. Finally,\nwe can de\ufb01ne a function to describe the boundary:\n\nDe\ufb01nition 2.5 The rate distortion function R(D) is the in\ufb01mum of rates R, so that (R; D)\nis in the rate distortion region of the source for a given distortion D.\n\nAs in [7], we restrict ourselves to a binary source J with a Hamming distortion measure\nfor simplicity. We assume that binary alphabets are drawn randomly, i.e., the source is\nnot biased to rule out the possiblity of compression due to redundancy. We now \ufb01nd the\ndescription rate R(D) required to describe the source with an expected proportion of errors\nless than or equal to D. In this simpli\ufb01ed case, according to Shannon, the boundary can be\nwritten as follows.\n\nTheorem 2.1 The rate distortion function for a binary source with Hamming distortion is\ngiven by\n\nR(D) =(cid:26)1 (cid:0) H(D)\n\n0\n\n0 (cid:20) D (cid:20) 1\n2\n1\n2 < D\n\n;\n\n(5)\n\nwhere H((cid:1)) represents the binary entropy function.\n\nencoder\n\ndecoder\n\nJ (cid:0)! f\n\n(cid:0)! (cid:24) (cid:0)! g\n\n(cid:0)! ^J\n\nFigure 1: Rate distortion encoder and decoder\n\n3 General Scenario\n\nIn this section, we introduce a toy model for lossy compression. We use the inverse problem\nof Sourlas-type decoding to realize the optimal encoding scheme [4]. As in the previous\nsection, we assume that binary alphabets are drawn randomly from a non-biased source\nand that the Hamming distortion measure is selected for the \ufb01delity criterion.\n\nWe take the Boolean representation of the binary alphabet J , i.e., we set J = f0; 1g.\nWe also set X = f0; 1g to represent the codewords throughout the rest of this paper.\n\n\fLet J be an M-bit source sequence, (cid:24) an N-bit codeword, and ^J an M-bit reproduction\nsequence. Here, the encoding problem can be written as follows. Given a distortion D and\na randomly-constructed Boolean matrix A of dimensionality M (cid:2) N, we \ufb01nd the N-bit\ncodeword sequence (cid:24), which satis\ufb01es\n\nwhere the \ufb01delity criterion\n\n^J = A(cid:24)\n\n(mod 2) ;\n\nD = E[d(J ; ^J )]\n\n(6)\n\n(7)\n\nholds, according to every M-bit source sequence J. Note that we applied modulo 2 arith-\nmetics for the additive operations in (6). In our framework, decoding will just be a linear\nmapping ^J = A(cid:24), while encoding remains a NP-complete problem.\nKabashima and Saad recently expanded on the work of Sourlas, which focused on the zero-\nrate limit, to an arbitrary-rate case [5]. We follow their construction of the matrix A, so we\ncan treat practical cases. Let the Boolean matrix A be characterized by K ones per row\nand C per column. The \ufb01nite, and usually small, numbers K and C de\ufb01ne a particular\ncode. The rate of our codes can be set to an arbitrary value by selecting the combination\nof K and C. We also use K and C as control parameters to de\ufb01ne the rate R = K=C. If\nthe value of K is small, i.e., the relation K (cid:28) N holds, the Boolean matrix A results in\na very sparse matrix. By contrast, when we consider densely constructed cases, K must\nbe extensively big and have a value of O(N ). We can also assume that K is not O(1) but\nK (cid:28) N holds. The codes within any parameter region, including the sparsely-constructed\ncases, will result in optimal codes as we will conclude in the following section. This is one\nnew \ufb01nding of our analysis using statistical physics.\n\n4 Physics of the model: One-step RSB Scheme\n\nThe similarity between codes of this type and Ising spin systems was \ufb01rst pointed out by\nSourlas, who formulated the mapping of a code onto an Ising spin system Hamiltonian\nin the context of error correction [4]. To facilitate the current investigation, we \ufb01rst map\nthe problem to that of an Ising model with \ufb01nite connectivity following Sourlasfmethod.\nWe use the Ising representation f1; (cid:0)1g of the alphabet J and X rather than the Boolean\none f0; 1g; the elements of the source J and the codeword sequences (cid:24) are rewritten in\nIsing values by mapping only, and the reproduction sequence ^J is generated by taking\nproducts of the relevant binary codeword sequence elements in the Ising representation\n^Jhi1;i2;(cid:1)(cid:1)(cid:1) ;iK i = (cid:24)i1(cid:24)i2 (cid:1) (cid:1) (cid:1) (cid:24)iK , where the indices i1; i2; (cid:1) (cid:1) (cid:1) ; iK correspond to the ones per\nrow A, producing a Ising version of ^J. Note that the additive operation in the Boolean\nrepresentation is translated into the multiplication in the Ising one. Hereafter, we set\nJj; ^Jj; (cid:24)i = (cid:6)1 while we do not change the notations for simplicity. As we use statistical-\nmechanics techniques, we consider the source and codeword-sequence dimensionality (M\nand N, respectively) to be in\ufb01nite, keeping the rate R = N=M \ufb01nite. To explore the\nsystem\u2019s capabilities, we examine the Hamiltonian:\n\nH(S) = (cid:0) Xhi1;(cid:1)(cid:1)(cid:1) ;iK i\n\nAhi1;(cid:1)(cid:1)(cid:1) ;iK iJhi1;(cid:1)(cid:1)(cid:1) ;iK iSi1 (cid:1) (cid:1) (cid:1) SiK ;\n\n(8)\n\nwhere we have introduced the dynamical variable Si to \ufb01nd the most suitable Ising code-\nword sequence (cid:24) to provide the reproduction sequence ^J in the decoding stage. Elements\nof the sparse connectivity tensor Ahi1;(cid:1)(cid:1)(cid:1) ;iK i take the value one if the corresponding indices\nof codeword bits are chosen (i.e., if all corresponding indices of the matrix A are one) and\nzero otherwise; C ones per i index represent the system\u2019s degree of connectivity.\n\n\fFor calculating the partition function Z(A; J ) = TrfSg exp[(cid:0)(cid:12)H(S)], we apply the\nreplica method following the calculation of Kabashima and Saad [5]. To calculate replica-\nfree energy, we have to calculate the annealed average of the n-th power of the partition\nfunction by preparing n replicas. Here we introduce the inverse temperature (cid:12), which can\nbe interpreted as a measure of the system\u2019s sensitivity to distortions. As we see in the fol-\nlowing calculation, the optimal value of (cid:12) is naturally determined when the consistency of\nthe replica symmetry breaking scheme is considered [13, 3]. We use integral representa-\ntions of the Dirac (cid:14) function to enforce the restriction, C bonds per index, on A [14]:\n\n(cid:14)0\n@ Xhi2;i3;(cid:1)(cid:1)(cid:1) ;iK i\n\nAhi;i2;(cid:1)(cid:1)(cid:1) ;iK i (cid:0) C1\n\nA =I 2(cid:25)\n\n0\n\ngiving rise to a set of order parameters\n\ndZ\n2(cid:25)\n\nZ (cid:0)(C+1)ZPhi2 ;i3 ;(cid:1)(cid:1)(cid:1) ;iK i Ahi;i2 ;(cid:1)(cid:1)(cid:1) ;iK i ; (9)\n\nq(cid:11);(cid:12);(cid:1)(cid:1)(cid:1) ;(cid:13) =\n\n1\nN\n\nN\n\nXi=1\n\nZiS(cid:11)\n\ni S(cid:12)\n\ni (cid:1) (cid:1) (cid:1) S(cid:13)\ni ;\n\n(10)\n\nwhere (cid:11); (cid:12); (cid:1) (cid:1) (cid:1) ; (cid:13) represent replica indices, and the average over J is taken with respect\nto the probability distribution:\n\nP[Jhi1;i2;(cid:1)(cid:1)(cid:1) ;iK i] =\n\n1\n2\n\n(cid:14)(Jhi1;i2;(cid:1)(cid:1)(cid:1) ;iK i (cid:0) 1) +\n\n1\n2\n\n(cid:14)(Jhi1;i2;(cid:1)(cid:1)(cid:1) ;iK i + 1)\n\n(11)\n\nas we consider the non-biased source sequences for simplicity. Assuming the replica sym-\nmetry, we use a different representation for the order parameters and the related conjugate\nvariables [14]:\n\nq(cid:11);(cid:12);(cid:1)(cid:1)(cid:1) ;(cid:13) = qZ dx (cid:25)(x) tanhl((cid:12)x) ;\n^q(cid:11);(cid:12);(cid:1)(cid:1)(cid:1) ;(cid:13) = ^qZ dx ^(cid:25)(^x) tanhl((cid:12) ^x) ;\n\n(12)\n\n(13)\n\nwhere q = [(K (cid:0) 1)!N C]1=K and ^q = [(K (cid:0) 1)!](cid:0)1=K [N C](K(cid:0)1)=K are normalization\nconstants, and (cid:25)(x) and ^(cid:25)(^x) represent probability distributions related to the integration\nvariables. Here l denotes the number of related replica indices. Throughout this paper,\nintegrals with unspeci\ufb01ed limits denote integrals over the range of ((cid:0)1; +1). We then\nobtain an expression for the free energy per source bit expressed in terms of the probability\ndistributions (cid:25)(x) and ^(cid:25)(^x):\n\n(cid:0)(cid:12)f =\n\n1\nM\n\nhhln Z(A; J )ii\n\n= ln cosh (cid:12)\n\nK\n\nK\n\nYl=1\n\ndxl (cid:25)(xl)*ln 1 + tanh (cid:12)J\n\ntanh (cid:12)xl!+J\n+Z\n(cid:0) KZ dx (cid:25)(x)Z d^x ^(cid:25)(^x) ln(1 + tanh (cid:12)x tanh (cid:12) ^x)\n(1 + S tanh (cid:12) ^xl)# ;\nK Z\n\nd^xl ^(cid:25)(^xl) ln\"Tr\n\nYl=1\n\n+\n\nC\n\nC\n\nYl=1\n\nS\n\nC\n\nYl=1\n\n(14)\n\nwhere hh(cid:1) (cid:1) (cid:1) ii denotes the average over quenched randomness of A and J. The saddle\npoint equations with respect to probability distributions provide a set of relations between\n\n\f(cid:25)(x) and ^(cid:25)(^x):\n\n(cid:25)(x) =Z \"C(cid:0)1\nYl=1\n^(cid:25)(^x) =Z \"C(cid:0)1\nYl=1\n\nC(cid:0)1\n\nd^xl ^(cid:25)(^xl)# (cid:14) x (cid:0)\nXl=1\ndxl (cid:25)(xl)#*(cid:14)\"^x (cid:0)\n\n1\n(cid:12)\n\n^xl! ;\ntanh(cid:0)1 tanh (cid:12)J\n\n(15)\n\ntanh (cid:12)xl!#+J\n\n:\n\nK(cid:0)1\n\nYl=1\n\nBy using the result obtained for the free energy, we can easily perform further straight-\nforward calculations to \ufb01nd all the other observable thermodynamical quantities, including\ninternal energy:\n\n1\n\ne =\n\nM DDTrSH(S)e(cid:0)(cid:12)H(S)EE = (cid:0)\n\n1\nM\n\n@\n@(cid:12)\n\nhhln Z(A; J )ii ;\n\n(16)\n\nwhich records reproduction errors. Therefore, in terms of the considered replica symmetric\nansatz, a complete solution of the problem seems to be easily obtainable; unfortunately, it\nis not.\n\nThis set of equations (15) may be solved numerically for general (cid:12), K, and C. How-\never, there exists an analytical solution of this equations. We \ufb01rst consider this case. Two\ndominant solutions emerge that correspond to the paramagnetic and the spin glass phases.\nThe paramagnetic solution, which is also valid for general (cid:12), K, and C, is in the form of\n(cid:25)(x) = (cid:14)(x) and ^(cid:25) = (cid:14)(^x); it has the lowest possible free energy per bit fPARA = (cid:0)1,\nalthough its entropy sPARA = (R(cid:0)1) ln 2 is positive only for R (cid:21) 1. It means that the true\nsolution must be somewhere beyond the replica symmetric ansatz. As a \ufb01rst step, which is\ncalled the one-step replica symmetry breaking (RSB), n replicas are usually divided into\nn=m groups, each containing m replicas. Pathological aspects due to the replica symmetry\nmay be avoided making use of the newly-de\ufb01ned freedom m. Actually, this one-step RSB\nscheme is considered to provide the exact solutions when the random energy model limit\nis considered [15], while our analysis is not restricted to this case.\n\nThe spin glass solution can be calculated for both the replica symmetric and the one-step\nRSB ansatz. The former reduces to the paramagnetic solution (fRS = fPARA), which is\nunphysical for R < 1, while the latter yields (cid:25)1RSB(x) = (cid:14)(x), ^(cid:25)1RSB(^x) = (cid:14)(^x) with\nm = (cid:12)g(R)=(cid:12) and (cid:12)g obtained from the root of the equation enforcing the non-negative\nreplica symmetric entropy\n\nsRS = ln cosh (cid:12)g (cid:0) (cid:12)g tanh (cid:12)g + R ln 2 = 0 ;\n\nwith a free energy\n\nf1RSB = (cid:0)\n\n1\n(cid:12)g\n\nln cosh (cid:12)g (cid:0)\n\nR\n(cid:12)g\n\nln 2 :\n\n(17)\n\n(18)\n\nSince the target bit of the estimation in this model is Jhi1;(cid:1)(cid:1)(cid:1) ;iK i and its estimator the product\nSi1 (cid:1) (cid:1) (cid:1) SiK , a performance measure for the information corruption could be the per-bond\nenergy e. According to the one-step RSB framework, the lowest free energy can be calcu-\nlated from the probability distributions (cid:25)1RSB(x) and ^(cid:25)1RSB(^x) satisfying the saddle point\nequation (15) at the characteristic inverse temperature (cid:12)g, when the replica symmetric en-\ntropy sRS disappears. Therefore, f1RSB equals e1RSB. Let the Hamming distortion be our\n\ufb01delity criterion. The distortion D associated with this code is given by the fraction of the\nfree energies that arise in the spin glass phase:\n\nD =\n\nf1RSB (cid:0) fRS\n\n2jfRSj\n\n=\n\n1 (cid:0) tanh (cid:12)g\n\n2\n\n:\n\n(19)\n\n\fHere, we substitute the spin glass solutions into the expression, making use of the fact that\nthe replica symmetric entropy sRS disappears at a consistent (cid:12)g, which is determined by\n(17). Using (17) and (19), simple algebra gives the relation between the rate R = N=M\nand the distortion D in the form\n\nR = 1 (cid:0) H(D) ;\n\nwhich coincides with the rate distortion function retrieving Theorem 2.1. Surprisingly, we\ndo not observe any \ufb01rst-order jumps between analytical solutions. Recently, we have seen\nthat many approaches to the family of codes, characterized by the linear encoding opera-\ntions, result in a quite different picture: the optimal boundary is constructed in the random\nenergy model limit and is well captured by the concept of a \ufb01rst-order jump. Our analysis\nof this model, viewed as a kind of inverse problem, provides an exception. Many optimal\nconditions in textbook information theory may be well described without the concept of a\n\ufb01rst-order phase transitions from a view point of statistical physics.\n\nWe will now investigate the possiblity of the other solutions satisfying (15) in the case\nof \ufb01nite K and C. Since the saddle point equations (15) appear dif\ufb01cult for analytical\narguments, we resort to numerical evaluations representing the probability distributions\n(cid:25)1RSB(x) and ^(cid:25)1RSB(^x) by up to 105 bin models and carrying out the integrations by\nusing Monte Carlo methods. Note that the characteristic inverse temperature (cid:12)g is also\nevaluated numerically by using (17). We set K = 2 and selected various values of C to\ndemonstrate the performance of stable solutions. The numerical results obtained by the\none-step RSB senario show suboptimal properties [Figure 2]. This strongly implies that\nthe analytical solution is not the only stable solution. This conjecture might be veri\ufb01ed\nelsewhere, carrying out large scale simulations.\n\n5 Conclusions\n\nTwo points should be noted. Firstly, we found that the consistency between the rate dis-\ntortion theory and the Parisi one-step RSB scheme. Secondly, we conjectured that the\nanalytical solution, which is consistent with the Shannon\u2019s result, is not the only stable\nsolution for some situations. We are currently working on the veri\ufb01cation.\n\nAcknowledgments\n\nWe thank Yoshiyuki Kabashima and Shun-ichi Amari for their comments on the\nmanuscript. We also thank Hiroshi Nagaoka and Te Sun Han for giving us valuable ref-\nerences. This research is supported by the Special Postdoctoral Researchers Program at\nRIKEN.\n\nReferences\n\n[1] H. Nishimori. Statistical Physics of Spin Glasses and Information Processing. Oxford\n\nUniversity Press, 2001.\n\n[2] T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, 1991.\n[3] V. Dotsenko. Introduction to the Replica Theory of Disordered Statistical Systems.\n\nCambridge University Press, 2001.\n\n[4] N. Sourlas. Spin-glass models as error-correcting codes. Nature, 339:693\u2013695, 1989.\n[5] Y. Kabashima and D. Saad. Statistical mechanics of error-correcting codes. Europhys.\n\nLett., 45:97\u2013103, 1999.\n\n[6] Y. Kabashima, T. Murayama, and D. Saad. Typical performance of Gallager-type\n\nerror-correcting codes. Phys. Rev. Lett., 84:1355\u20131358, 2000.\n\n\fR\n\n1\n\n0.8\n\n0.6\n\n0.4\n\n0.2\n\n0\n\n0\n\nK=2\nR(D)\n\np ( x ) \n^ ^\n\np ( x ) \n\n-1\n\n0\n\n1\n\n2\n\n2\n\n1\n\n0\n-2\n\n0.1\n\n0.2\n\n0.3\n\n0.4\n\n0.5\n\nD\n\nFigure 2: Numerically-constructed stable solutions: Stable solutions of (15) for the \ufb01nite\nvalues of K and L are calculated by using Monte Carlo methods. We use 105 bin models\nto approximate the probability distributions (cid:25)1RSB(x) and ^(cid:25)1RSB(^x), starting from various\ninitial conditions. The distributions converge to the continuous ones, giving suboptimal\nperformance. ((cid:14)) K = 2 and L = 3; 4; (cid:1) (cid:1) (cid:1) ; 12 ; Solid line indicates the rate distortion\nfunction R(D). Inset: Snapshots of the distributions, where L = 3 and (cid:12)g = 2:35.\n\n[7] T. Murayama. Statistical mechanics of the data compression theorem. J. Phys. A,\n\n35:L95\u2013L100, 2002.\n\n[8] A. Montanari and N. Sourlas. The statistical mechanics of turbo codes. Eur. Phys. J.\n\nB, 18:107\u2013119, 2000.\n\n[9] C. E. Shannon. Coding theorems for a discrete source with a \ufb01delity criterion. IRE\n\nNational Convention Record, Part 4, pages 142\u2013163, 1959.\n\n[10] T. Berger. Rate Distortion Theory: A Mathematical Basis for Data Compression.\n\nPrentice-Hall, 1971.\n\n[11] T. Hosaka, Y. Kabashima, and H. Nishimori. Statistical mechanics of lossy data\n\ncompression using a non-monotonic perceptron. cond-mat/0207356.\n\n[12] Y. Matsunaga and H. Yamamoto. A coding theorem for lossy data compression by\nLDPC codes. In Proceedings 2002 IEEE International Symposium on Information\nTheory, page 461, 2002.\n\n[13] M. Mezard, G. Parisi, and M. Virasoro. Spin-Glass Theory and Beyound. World\n\nScienti\ufb01c, 1987.\n\n[14] K. Y. M. Wong and D. Sherrington. Graph bipartitioning and spin glasses on a random\n\nnetwork of \ufb01xed \ufb01nite valence. J. Phys. A, 20:L793\u2013L799, 1987.\n\n[15] B. Derrida. The random energy model, an exactly solvable model of disordered sys-\n\ntems. Phys. Rev. B, 24:2613\u20132626, 1981.\n\n\f", "award": [], "sourceid": 2168, "authors": [{"given_name": "Tatsuto", "family_name": "Murayama", "institution": null}, {"given_name": "Masato", "family_name": "Okada", "institution": null}]}