{"title": "Message passing for task redistribution on sparse graphs", "book": "Advances in Neural Information Processing Systems", "page_first": 1529, "page_last": 1536, "abstract": "", "full_text": "Message passing for task redistribution on\n\nsparse graphs\n\nK. Y. Michael Wong\n\nHong Kong U. of Science & Technology\n\nClear Water Bay, Hong Kong, China\n\nphkywong@ust.hk\n\nDavid Saad\n\nNCRG, Aston University\nBirmingham B4 7ET, UK\nD.Saad@aston.ac.uk\n\nHong Kong U. of Science & Technology, Clear Water Bay, Hong Kong, China\n\nPermanent address: Dept. of Physics, Beijing Normal Univ., Beijing 100875, China\n\nZhuo Gao\n\nzhuogao@bnu.edu.cn\n\nAbstract\n\nThe problem of resource allocation in sparse graphs with real variables\nis studied using methods of statistical physics. An ef\ufb01cient distributed\nalgorithm is devised on the basis of insight gained from the analysis and\nis examined using numerical simulations, showing excellent performance\nand full agreement with the theoretical results.\n\n1 Introduction\n\nOptimal resource allocation is a well known problem in the area of distributed comput-\ning [1, 2] to which signi\ufb01cant effort has been dedicated within the computer science com-\nmunity. The problem itself is quite general and is applicable to other areas as well where a\nlarge number of nodes are required to balance loads/resources and redistribute tasks, such\nas reducing internet traf\ufb01c congestion [3]. The problem has many \ufb02avors and usually refers,\nin the computer science literature, to \ufb01nding practical heuristic solutions to the distribution\nof computational load between computers connected in a predetermined manner.\n\nThe problem we are addressing here is more generic and is represented by nodes of some\ncomputational power that should carry out tasks. Both computational powers and tasks will\nbe chosen at random from some arbitrary distribution. The nodes are located on a randomly\nchosen sparse graph of some given connectivity. The goal is to migrate tasks on the graph\nsuch that demands will be satis\ufb01ed while minimizing the migration of (sub-)tasks. An\nimportant aspect of the desired algorithmic solution is that decisions on messages to be\npassed are carried out locally; this enables an ef\ufb01cient implementation of the algorithm in\nlarge non-centralized distributed networks. We focus here on the satis\ufb01able case where the\ntotal computing power is greater than the demand, and where the number of nodes involved\nis very large. The unsatis\ufb01able case can be addressed using similar techniques.\n\nWe analyze the problem using the Bethe approximation of statistical mechanics in Sec-\ntion 2, and alternatively a new variant of the replica method [4, 5] in Section 3. We then\npresent numerical results in Section 4, and derive a new message passing distributed algo-\n\n\frepresenting the \u2019revised\u2019 assignment\n\napproach to resource allocation, we consider the load balancing task of minimizing the en-\n\ngraph is low, and the Bethe approximation well describes the local environment of a node.\n\n2 The statistical physics framework: Bethe approximation\n\nrithm on the basis of the analysis (in Section 5). We conclude the paper with a summary\nand a brief discussion on future work.\n\nWe consider a typical resource allocation task on a sparse graph of\u0006 nodes, labelled\ni=1;::;\u0006. Each nodei is randomly connected to\r other nodes1, and has a capacity\u0003i\nrandomly drawn from a distribution(cid:26)\u0004\u0003i\u0005. The objective is to migrate tasks between nodes\nsuch that each node will be capable of carrying out its tasks. The currentyij(cid:17)yji drawn\nfrom nodej toi is aimed at satisfying the constraint\nXjAijyij\u0007\u0003i(cid:21)0;\nfor nodei, whereAij=1=0 for con-\nnected/unconnected node pairsi andj, respectively. To illustrate the statistical mechanics\nergy function (cost)E=\b\u0004ij\u0005Aij(cid:30)\u0004yij\u0005, where the summation\u0004ij\u0005 runs over all pairs\nof nodes, subject to the constraints (1);(cid:30)\u0004y\u0005 is a general function of the currenty. For\nload balancing tasks,(cid:30)\u0004y\u0005 is typically a convex function, which will be assumed in our\nstudy. The analysis of the graph is done by introducing the free energyF=T\u0002Zy for\na temperatureT(cid:17)(cid:12)1\n, whereZy is the partition function\nZy=Y\u0004ij\u0005ZdyijYi\u00020\bXjAijyij\u0007\u0003i1Aex\u000424(cid:12)X\u0004ij\u0005Aij(cid:30)\u0004yij\u000535:\nThe\u0002 function returns 1 for a non-negative argument and 0 otherwise.\nWhen the connectivity\r is low, the probability of \ufb01nding a loop of \ufb01nite length on the\nIn the approximation, a node is connected to\r branches in a tree structure, and the corre-\nanother\r1 descendent nodes of the next generation.\nConsider a vertexV\u0004T\u0005 of capacity\u0003V\u0004T\u0005, and a currenty is drawn from the vertex.\nOne can write an expression for the free energyF\u0004yjT\u0005 as a function of the free energies\nF\u0004ykjTk\u0005 of its descendants, that branch out from this vertex\nF\u0004yjT\u0005=T\u0002\u0004\r1Yk=1(cid:18)Zdyk(cid:19)\u0002 \r1Xk=1yky\u0007\u0003V!\n\u0002ex\u0004\"(cid:12)\r1Xk=1\u0004F\u0004ykjTk\u0005\u0007(cid:30)\u0004yk\u0005\u0005#\u0005;\nwhereTk represents the tree terminated at thek\bh\nenergy can be considered as the sum of two parts,F\u0004yjT\u0005=\u0006TFav\u0007FV\u0004yjT\u0005, where\u0006T\nis the number of nodes in the treeT,Fav is the average free energy per node, andFV\u0004yjT\u0005\npotentialy as its sole argument, hence the terminology used.\n\nlations among the branches of the tree are neglected. In each branch, nodes are arranged\nin generations. A node is connected to an ancestor node of the previous generation, and\n\nis referred to as the vertex free energy2. Note that when a vertex is added to a tree, there is a\n\n1Although we focus here on graphs of \ufb01xed connectivity, one can easily accommodate any con-\n\nnectivity pro\ufb01le within the same framework; the algorithms presented later are completely general.\n\n2This term is marginalized over all inputs to the current vertex, leaving the difference in chemical\n\n(1)\n\n(2)\n\n(3)\n\ndescendent of the vertex. The free\n\n\fchange in the free energy due to the added vertex. Since the number of nodes increases by\n1, the vertex free energy is obtained by subtracting the free energy change by the average\nfree energy. This allows us to obtain the recursion relation\n\nFV\u0004yjT\u0005=T\u0002\u0004\r1Yk=1(cid:18)Zdyk(cid:19)\u0002 \r1Xk=1yky\u0007\u0003V\u0004T\u0005!\n\u0002ex\u0004\"(cid:12)\r1Xk=1\u0004FV\u0004ykjTk\u0005\u0007(cid:30)\u0004yk\u0005\u0005#\u0005Fav;\nFav=T\u0006\u0002\u0004\rYk=1(cid:18)Zdyk(cid:19)\u0002 \rXk=1yk\u0007\u0003V!\n\u0002ex\u0004\"(cid:12)\rXk=1\u0004FV\u0004ykjTk\u0005\u0007(cid:30)\u0004yk\u0005\u0005#\u0005\u0007\u0003;\nwhere\u0003V is the capacity of the vertexV fed by\r treesT1;:::;T\r, andh:::i\u0003 represents\nthe average over the distribution(cid:26)\u0004\u0003\u0005. In the zero temperature limit, Eq. (4) reduces to\nfykj\b\r1k=1yky\u0007\u0003V\u0004T\u0005(cid:21)0g\"\r1Xk=1\u0004FV\u0004ykjTk\u0005\u0007(cid:30)\u0004yk\u0005\u0005#Fav:\nFV\u0004yjT\u0005=\n\u0001i\u0002\nthe currenty0\nin a link from one vertex to another, fed by the treesT1 andT2, respectively;\nthe obtained expressions are\b\u0004y\u0005=h\u00c6\u0004yy0\u0005i? andhEi=h(cid:30)\u0004y0\u0005i? where\nh(cid:15)i?=(cid:28)Rdy0ex\u0004[(cid:12)\u0004FV\u0004y0jT1\u0005\u0007FV\u0004y0jT2\u0005\u0007(cid:30)\u0004y0\u0005\u0005\u2104\u0004(cid:15)\u0005\nRdy0ex\u0004[(cid:12)\u0004FV\u0004y0jT1\u0005\u0007FV\u0004y0jT2\u0005\u0007(cid:30)\u0004y0\u0005\u0005\u2104(cid:29)\u0003:\nelsewhere. To facilitate derivations, we focus on the quadratic cost function(cid:30)\u0004y\u0005=y2=2.\nIntroducing Lagrange multipliers, the function to be minimized becomes\u0004=\n\b\u0004ij\u0005Aijy2ij=2\u0007\bi(cid:22)i\u0004\bjAijyij\u0007\u0003i\u0005. Optimizing\u0004 with respect toyij, one ob-\ntainsyij=(cid:22)j(cid:22)i, where(cid:22)i is referred to as the chemical potential of nodei, and the\nall nodes, we introduce an extra regularization term(cid:15)\bi(cid:22)2i=2 to break the translational\nsymmetry, where(cid:15)!0. To study the characteristics of the problem one calculates the\naveraged free energy per nodeFav=Th\u0002Z(cid:22)iA;\u0003=\u0006, whereZ(cid:22) is the partition function\nYi24Zd(cid:22)i\u00020\bXjAij\u0004(cid:22)j(cid:22)i\u0005\u0007\u0003i1A35ex\u000424(cid:12)20\bX\u0004ij\u0005Aij\u0004(cid:22)j(cid:22)i\u00052\u0007(cid:15)Xi(cid:22)2i1A35:\n\nAlthough the analysis has also been carried out in the space of currents, we focus here on\nthe optimization problem in the space of the chemical potentials. Since the energy function\nis invariant under the addition of an arbitrary global constant to the chemical potentials of\n\nIn this section, we sketch the analysis of the problem using the replica method, as an alter-\nnative to the Bethe approximation. The derivation is rathe involved, details will be provided\n\nAn alternative formulation of the original optimization problem is to consider its\ndual.\n\nThe current distribution and the average free energy per link can be derived by integrating\n\n(4)\n\n(5)\n\n(6)\n\n(7)\n\nand the average free energy per node is given by\n\n3 The statistical physics framework: replica method\n\nThe results con\ufb01rm the validity of the Bethe approximation on sparse graphs.\n\ncurrent is driven by the potential difference.\n\n\fand averaging over all connectivity matrices, one \ufb01nds\n\nrelated to the order parameters via the generating function\n\nindependent of the replica indices, and is given by\n\nThe calculation follows the main steps of a replica based calculation in diluted systems [6],\n\na set of saddle point equations w.r.t the order parameters. Assuming replica symmetry, the\n\nis a result of the speci\ufb01c interaction considered which entangles nodes of different indices.\n\nusing the identity\u0002Z=i\u0001\u0002!0[Z\u00021\u2104=\u0002. The replicated partition function [5] is av-\neraged over all network con\ufb01gurations with connectivity and capacity distributions(cid:26)\u0004\u0003i\u0005.\nWe consider the case of intensive connectivity\r(cid:24)\u0007\u00041\u0005(cid:28)\u0006. Extending the analysis of [6]\nex\u0004\u0006\u0004\r2\rX\u0006;\u0007^\t\u0006;\u0007\t\u0006;\u0007\u0007\u0002Zd\u0003(cid:26)\u0004\u0003\u0005Y(cid:11) Zd(cid:22)(cid:11)Z1\u0003d(cid:21)(cid:11)Zd^(cid:21)(cid:11)2(cid:25)!\nhZ\u0002(cid:22)i=\n\u0002ex\u0004\"X(cid:11)(cid:18)i^(cid:21)(cid:11)\u0004(cid:21)(cid:11)\u0007\r(cid:22)(cid:11)\u0005(cid:12)2\u0004\r\u0007(cid:15)\u0005(cid:22)2(cid:11)(cid:19)#X\r\u0005;\nwhereX=\b\u0006;\u0007^\t\u0006;\u0007\t(cid:11)\u0004i^(cid:21)(cid:11)\u0005\u0006(cid:11)(cid:22)\u0007(cid:11)(cid:11)\u0007\b\u0006;\u0007\n2\t(cid:11)\u0006(cid:11)!\u0007(cid:11)!\t(cid:11)(cid:22)\u0006(cid:11)(cid:11)\u0004(cid:12)(cid:22)(cid:11)i^(cid:21)(cid:11)\u0005\u0007(cid:11). The\n\t\u0006;\u0007\norder parameters\t\u0006;\u0007 and^\t\u0006;\u0007, are labelled by the somewhat unusual indices\u0006 and\u0007,\nrepresenting the\u0002-component integer vectors\u0004\u00061;::;\u0006\u0002\u0005 and\u0004\u00071;::;\u0007\u0002\u0005 respectively. This\nThe order parameters\t\u0006;\u0007 and^\t\u0006;\u0007 are given by the extremum condition of Eq. (8), i.e., via\nsaddle point equations yield a recursion relation for a two-component functionR, which is\n\u0006(cid:11)!=\u0006Y(cid:11)(cid:18)Zd(cid:22)R\u0004z(cid:11);(cid:22)jT\u0005e(cid:12)(cid:22)2=2(cid:22)\u0007(cid:11)(cid:19)\u0007\u0003:\n\b\u0007\u0004z\u0005=X\u0006\t\u0006;\u0007Y(cid:11)\u0004z(cid:11)\u0005\u0006(cid:11)\nIn Eq. (9),T represents the tree terminated at the vertex node with chemical potential\n(cid:22), providing input to the ancestor node with chemical potentialz, andh:::i\u0003 represents\nthe average over the distribution(cid:26)\u0004\u0003\u0005. The resultant recursion relation forR\u0004z;(cid:22)jT\u0005 is\n1D\r1Yk=1(cid:18)Zd(cid:22)kR\u0004(cid:22);(cid:22)kjTk\u0005(cid:19)\u0002 \r1Xk=1(cid:22)k\r(cid:22)\u0007z\u0007\u0003V\u0004T\u0005!\nR\u0004z;(cid:22)jT\u0005=\n\u0002ex\u0004\"(cid:12)2 \r1Xk=1\u0004(cid:22)(cid:22)k\u00052\u0007(cid:15)(cid:22)2!#;\nwhere the vertex node has a capacity\u0003V\u0004T\u0005;D is a constant.R\u0004z;(cid:22)jT\u0005 is expressed in\nterms of\r1 functionsR\u0004(cid:22);(cid:22)kjTk\u0005 (k=1;::;\r1), integrated over(cid:22)k. This algebraic\n\r, where a node obtains input from its\r1 descendent nodes of the next generation, and\nTk represents the tree terminated at thek\bh\nExcept for the regularization factorex\u0004\u0004(cid:12)(cid:15)(cid:22)2=2\u0005,R turns out to be a function of\ny(cid:17)(cid:22)z, which is interpreted as the current drawn from a node with chemical po-\ntential(cid:22) by its ancestor with chemical potentialz. One can then express the functionR\nas the product of a vertex partition functionZV and a normalization factorW , that is,\nR\u0004z;(cid:22)jT\u0005=W\u0004(cid:22)\u0005ZV\u0004yjT\u0005. In the limit\u0002!0, the dependence on(cid:22) andy are separa-\nble, providing a recursion relation forZV\u0004yjT\u0005. This gives rise to the vertex free energy\nFV\u0004yjT\u0005=T\u0002ZV\u0004yjT\u0005 when a currenty is drawn from the vertex of a treeT. The re-\n\n(8)\n\n(9)\n\n(10)\n\nstructure is typical of the Bethe lattice tree-like representation of networks of connectivity\n\ndescendent.\n\ncursive equation and the average free energy expression agrees with the results in the Bethe\napproximation. These iterative equations can be directly linked to those obtained from a\nprincipled Bayesian approximation, where the logarithms of the messages passed between\nnodes are proportional to the vertex free energies.\n\n\f4 Numerical solution\n\nThe solution of Eq. (6) is obtained numerically. Since the vertex free energy of a node de-\npends on its own capacity and the disordered con\ufb01guration of its descendants, we generate\n1000 nodes at each iteration of Eq. (6), with capacities randomly drawn from the distribu-\n\noptimization search at each node, we \ufb01rst \ufb01nd the vertex saturation current drawn from\na node such that: (a) the capacity of the node is just used up; (b) the current drawn by\neach of its descendant nodes is just enough to saturate its own capacity constraint. When\nthese conditions are satis\ufb01ed, we can separately optimize the current drawn by each de-\nscendant node, and the vertex saturation current is equal to the node capacity subtracted by\nthe current drawn by its descendants. The optimal solution can be found using an exhaus-\ntive search, by varying the component currents in small discrete steps. This approach is\n\nTo compute the average energy, we randomly draw 2 nodes, compute the optimal current\n\ufb02owing between them, and repeat the sampling to obtain the average. Figure 1(a) shows\n\nthe tree structure, such that the iterative process corresponds to approximating the network\nby an increasingly extensive tree. We observe that after an initial rise with iteration steps,\nthe average energies converges to steady-state values, at a rate which increases with the\naverage capacity.\n\nTo study the convergence rate of the iterations, we \ufb01t the average energy at iteration step\n\ntion(cid:26)\u0004\u0003\u0005, each being fed by\r1 nodes randomly drawn from the previous iteration.\nWe have discretized the vertex free energiesFV\u0004yjT\u0005 function into a vector, whosei\bh\ncomponent is the value of the function corresponding to the currentyi. To speed up the\nparticularly convenient for\r=3, where the search is con\ufb01ned to a single parameter.\nthe results as a function of iteration step\b, for a Gaussian capacity distribution(cid:26)\u0004\u0003\u0005 with\nvariance 1 and averageh\u0003i. Each iteration corresponds to adding one extra generation to\n\b usinghE\u0004\b\u0005E\u00041\u0005i(cid:24)ex\u0004\u0004(cid:13)\b\u0005 in the asymptotic regime. As shown in the inset of\nFig. 1(a), the relaxation rate(cid:13) increases with the average capacity. It is interesting to note\ndistribution\b\u0004y\u0005 consists of a delta function component aty=0 and a continuous com-\nas\u00063\nvariables, message passing scales as\u0006. An even more important advantage, relevant to\n\nthat a cusp exists at the average capacity of about 0.45. Below that value, convergence\nof the iteration is slow, since the average energy curve starts to develop a plateau before\nthe \ufb01nal convergence. On the other hand, the plateau disappears and the convergence is\nfast above the cusp. The slowdown of convergence below the cusp is probably due to the\nappearance of increasingly large clusters of nonzero currents on the network, since clus-\nters of nodes with negative capacities become increasingly extensive, and need to draw\ncurrents from increasingly extensive regions of nodes with excess capacities to satisfy the\ndemand. Figure 1(b) illustrates the current distribution for various average capacities. The\n\nThe local nature of the recursion relation Eq. (6) points to the possibility that the network\noptimization can be solved by message passing approaches, which have been successful\nin problems such as error-correcting codes [8] and probabilistic inference [9]. The major\nadvantage of message passing is its potential to solve a global optimization problem via\nlocal updates, thereby reducing the computational complexity. For example, the compu-\ntational complexity of quadratic programming for the load balancing task typically scales\n, whereas capitalizing on the network topology underlying the connectivity of the\n\nponent whose breadth decreases with average capacity. The fraction of links with zero\ncurrents increases with the average capacity. Hence at a low average capacity, links with\nnonzero currents form a percolating cluster, whereas at a high average capacity, it breaks\ninto isolated clusters.\n\n5 Distributed algorithms\n\n\f(a)\n\n101\n\n100\n\n>\nE\n<\n\n10\u22121\n\n10\u22122\n\n0\n\n10\n\n0.15\n\n(c)\n\n\u03b3\n\n100\n\n10\u22121\n\n10\u22122\n\n0\n\n0.2 0.4 0.6 0.8\n\n<\u039b>\n\n2\n\n(b)\n\n1.5\n\n)\ny\n(\nP\n\n1\n\n0.5\n\n30\n\n40\n\n20\nt\n\n0.1\n\n0.8\n\n0.5\n\n0\n\n0\n\n2\n\n(d)\n\nc=3\n\n1.5\n\n)\n0\n=\n\u00b5\n(\nP\n\n0.5\n\nc=3\n\n>\nE\n<\n)\n2\n\u2212\nc\n(\n\n)\n0\n=\ny\n(\nP\n\n0.4\n\n0\n\n0\n\n0.1\n\n1\ny\n\n0.5\n<\u039b>\n\n1.5\n\n2\n\n\u22121\n\u00b5\n\n<\u039b>\n\n0.8\n\n0\n\n0\n\n1\n\n0.5\n\n0.1\n\n0\n\n0\n\n>\nE\n<\n\n0.1\n\n0.05\n\n0\n\n0\n\nc=5\n\n1\n\n0\n\n\u22122\n\n\u22121.5\n\nc=4\n\n\u22120.5\n\n0\n\n0.2\n\n0.4\n\n0.6\n\n0.8\n\nc=5\n\n)\n\u00b5\n(\nP\n\n1\n\n0.5\n<\u039b>\n\n0.5\n<\u039b>\n\nFigure 1: Results for system size\u0006=1000 and(cid:30)\u0004y\u0005=y2=2. (a)hEi obtained by iterating\nEq. (6) as a function of\b forh\u0003i=0.1, 0.2, 0.4, 0.6, 0.8 (top to bottom) and\r=3. Dashed\nline: The asymptotichEi forh\u0003i=0:1. Inset:(cid:13) as a function ofh\u0003i. (b) The distribution\n\b\u0004y\u0005 obtained by iterating Eq. (6) to steady states for the same parameters and average\ncapacities as in (a), from right to left. Inset:\b\u0004y=0\u0005 as a function ofh\u0003i. Symbols:\r=3\n((cid:13)) and (\u0003),\r=4 (\u0006) and (4),\r=5 (C) and (\u0006); each pair obtained from Eqs. (11) and\n(14) respectively. Line:e\u0006f\u0004h\u0003i=\u00042\u0005. (c)hEi as a function ofh\u0003i for\r=3;4;5. Symbols:\nresults of Eq. (6) ((cid:13)), Eq.(11) (\u0003), and Eq. (14) (\u0006). Inset:hEi multiplied by\u0004\r2\u0005 as\na function ofh\u0003i for the same conditions. (d) The distribution\b\u0004(cid:22)\u0005 obtained by iterating\nto right. Inset:\b\u0004(cid:22)=0\u0005 as a function ofh\u0003i. Symbols: same as (b).\nare more complex, since they are functionsFV\u0004yjT\u0005 of the currenty. We simplify the mes-\nLet\u0004Aij;Bij\u0005(cid:17)\u0004\bFV\u0004yijjTj\u0005=\byij;\b2FV\u0004yijjTj\u0005=\by2ij\u0005 be the message passed from\n\nsage to 2 parameters, namely, the \ufb01rst and second derivatives of the vertex free energies.\nFor the quadratic load balancing task, it can be shown that a self-consistent solution of the\nrecursion relation, Eq. (6), consists of vertex free energies which are piecewise quadratic\nwith continuous slopes. This makes the 2-parameter message a very precise approximation.\n\npractical implementation, is its distributive nature; it does not require a global optimizer,\nand is particularly suitable for distributive control in evolving networks.\n\nHowever, in contrast to other message passing algorithms which pass conditional probabil-\nity estimates of discrete variables to neighboring nodes, the messages in the present context\n\nEq. (14) to steady states for the same parameters and average capacities as in (b), from left\n\n\fnetwork con\ufb01gurations are generated randomly, with loops of lengths 3 or less excluded.\nUpdates are performed with random sequential choices of the nodes. As shown in Fig. 1(c),\nthe simulation results of the message passing algorithm have an excellent agreement with\nthose obtained by the recursion relation Eq.(6).\n\nnodej toi; using Eq.(6), the recursion relation of the messages become\nAij (cid:22)ij;Bij \u0002\u0004(cid:22)ij\u000524Xk6=iAjk\u0004(cid:30)00jk\u0007Bjk\u00051351; where\n(cid:22)ij=\u0001i\u0002\"\bk6=iAjk[yjk\u0004(cid:30)0jk\u0007Ajk\u0005\u0004(cid:30)00jk\u0007Bjk\u00051\u2104\u0007\u0003jyij\n;0#;\n\bk6=iAjk\u0004(cid:30)00jk\u0007Bjk\u00051\nwith(cid:30)0jk and(cid:30)00jk representing the \ufb01rst and second derivatives of(cid:30)\u0004y\u0005 aty=yjk respec-\ntively. The forward passing of the message from nodej toi is then followed by a backward\nmessage from nodej tok for updating the currentsyjk according to\nyjk yjk(cid:30)0jk\u0007Ajk\u0007(cid:22)ij\n:\n(cid:30)00jk\u0007Bjk\nWe simulate networks with\r=3,(cid:30)\u0004y\u0005=y2=2 and compute their average energies. The\n(cid:22)i=\u0001i\u0002241\r0\bXjAij(cid:22)j\u0007\u0003i1A;035:\nBoth Eqs. (11) and (14) allow us to study the distribution\b\u0004(cid:22)\u0005 of the chemical potentials\n(cid:22). As shown in Fig. 1(d),\b\u0004(cid:22)\u0005 consists of a delta function and a continuous component.\n0.53, close to the percolation threshold of 0.5 for\r=3.\nBesides the case of\r=3, Fig. 1(c) also shows the simulation results of the average energy\nfor\r=4;5, using both Eqs. (11) and (14). We see that the average energy decreases\nan exponential \ufb01thEi(cid:24)ex\u0004\u0004kh\u0003i\u0005 is applicable, wherek lies in the range 2.5 to 2.7.\nRemarkably, multiplying by a factor of\u0004\r2\u0005, we \ufb01nd that the 3 curves collapse in this\nregime of average capacity, showing that the average energy scales as\u0004\r2\u00051\n\nNodes with zero chemical potentials correspond to those with unsaturated capacity con-\nstraints. The fraction of unsaturated nodes increases with the average capacity, as shown in\nthe inset of Fig. 1(d). Hence at a low average capacity, saturated nodes form a percolating\ncluster, whereas at a high average capacity, it breaks into isolated clusters. It is interesting\nto note that at the average capacity of 0.45, below which a plateau starts to develop in the\nrelaxation rate of the recursion relation Eq. (6), the fraction of unsaturated nodes is about\n\nIt also provides a local iterative method for the optimization problem. As shown in\nFig. 1(c), both the recursion relation Eq.(6) and the message passing algorithm Eq.(11)\nyield excellent agreement with the iteration of chemical potentials Eq.(14).\n\nwhen the connectivity increases. This is because the increase in links connecting a node\nprovides more freedom to allocate resources. When the average capacity is 0.2 or above,\n\nFor the quadratic load balancing task considered here, an independent exact optimization\nis available for comparison. The K\u00a8uhn-Tucker conditions for the optimal solution yields\n\n(11)\n\n(12)\n\n(13)\n\n(14)\n\nregime, as shown in the inset of Fig. 1(c).\n\nFurther properties of the optimized networks have been studied by simulations, and will\nbe presented elsewhere. Here we merely summarize the main results. (a) When the av-\nerage capacity drops below 0.1, the energy rises above the exponential \ufb01t applicable to\nthe average capacity above 0.2. (b) The fraction of links with zero currents increases with\nthe average capacity, and is rather insensitive to the connectivity. Remarkably, except for\n\nin this\n\n\fvery small average capacities, the functione\u0006f\u0004h\u0003i=\u00042\u0005 has a very good \ufb01t with the data.\nIndeed, in the limit of largeh\u0003i, this function approaches the fraction of links with both\nvertices unsaturated, that is,[R10d\u0003(cid:26)\u0004\u0003\u0005\u21042\nR10d\u0003(cid:26)\u0004\u0003\u0005, which is the probability\non the average capacity, the exponent ranging from1 for\r=3 to0:8 for\r=5 for\nEq. (14), and being about -0.5 for\r=3;4;5 for Eq. (11). When the average capacity\n\n. (c) The fraction of unsaturated nodes increases\nwith the average capacity, and is rather insensitive to the connectivity. In the limit of large\naverage capacities, it approaches the upper bound of\nthat the capacity of a node is non-negative. (d) The convergence time of Eq. (11) can be\nmeasured by the time for the r.m.s. of the changes in the chemical potentials to fall below\na threshold. Similarly, the convergence time of Eq. (14) can be measured by the time for\nthe r.m.s. of the sums of the currents in both message directions of a link to fall below a\nthreshold. When the average capacity is 0.2 or above, we \ufb01nd the power-law dependence\n\ndecreases further, the convergence time deviates above the power laws.\n\n6 Summary\n\nWe have studied a prototype problem of resource allocation on sparsely connected networks\nusing the replica method, resulting in recursion relations interpretable using the Bethe ap-\nproximation. The resultant recursion relation leads to a message passing algorithm for\noptimizing the average energy, which signi\ufb01cantly reduces the computational complexity\nof the global optimization task and is suitable for online distributive control. The suggested\n2-parameter approximation produces results with excellent agreement with the original re-\ncursion relation. For the simple but illustrative example in this letter, we have considered a\nquadratic cost function, resulting in an exact algorithm based on local iterations of chem-\nical potentials, and the message passing algorithm shows remarkable agreement with the\nexact result. The suggested simple message passing algorithm can be generalized to more\nrealistic cases of nonlinear cost functions and additional constraints on the capacities of\nnodes and links. This constitutes a rich area for further investigations with many potential\napplications.\n\nAcknowledgments\n\nThis work is partially supported by research grants HKUST6062/02P and DAG04/05.SC25\nof the Research Grant Council of Hong Kong and by EVERGROW, IP No. 1935 in the FET,\nEU FP6 and STIPCO EU FP5 contract HPRN-CT-2002-00319.\n\nReferences\n\n[1] Peterson L. and Davie B.S., Computer Networks: A Systems Approach, Academic Press, San\n\nDiego CA (2000)\n\n[2] Ho Y.C., Servi L. and Suri R. Large Scale Systems 1 (1980) 51\n[3] Shenker S., Clark D., Estrin D. and Herzog S. ACM Computer Comm. Review 26 (1996) 19\n[4] Nishimori H. Statistical Physics of Spin Glasses and Information Processing, OUP UK (2001)\n[5] M\u00b4ezard M., Parisi P. and Virasoro M., Spin Glass Theory and Beyond, World Scienti\ufb01c, Singa-\n\npore (1987)\n\n[6] Wong K.Y.M. and Sherrington D. J. Phys. A20(1987) L793\n[7] Sherrington D. and Kirkpatrick S. Phys. Rev. Lett.35 (1975) 1792\n[8] Opper M. and Saad D. Advanced Mean Field Methods, MIT press (2001)\n[9] MacKay D.J.C., Information Theory, Inference and Learning Algorithms, CUP UK(2003)\n\n\f", "award": [], "sourceid": 2954, "authors": [{"given_name": "K. Y. Michael", "family_name": "Wong", "institution": null}, {"given_name": "David", "family_name": "Saad", "institution": null}, {"given_name": "Zhuo", "family_name": "Gao", "institution": null}]}