{"title": "Managing Power Consumption and Performance of Computing Systems Using Reinforcement Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 1497, "page_last": 1504, "abstract": null, "full_text": "Managing Power Consumption and Performance of\nComputing Systems Using Reinforcement Learning\n\nGerald Tesauro, Rajarshi Das, Hoi Chan, Jeffrey O. Kephart,\nCharles Lefurgy\u2217, David W. Levine and Freeman Rawson\u2217\n\nIBM Watson and Austin\u2217 Research Laboratories\n\n{gtesauro,rajarshi,hychan,kephart,lefurgy,dwl,frawson}@us.ibm.com\n\nAbstract\n\nElectrical power management in large-scale IT systems such as commercial data-\ncenters is an application area of rapidly growing interest from both an economic\nand ecological perspective, with billions of dollars and millions of metric tons of\nCO2 emissions at stake annually. Businesses want to save power without sac-\nri\ufb01cing performance. This paper presents a reinforcement learning approach to\nsimultaneous online management of both performance and power consumption.\nWe apply RL in a realistic laboratory testbed using a Blade cluster and dynam-\nically varying HTTP workload running on a commercial web applications mid-\ndleware platform. We embed a CPU frequency controller in the Blade servers\u2019\n\ufb01rmware, and we train policies for this controller using a multi-criteria reward\nsignal depending on both application performance and CPU power consumption.\nOur testbed scenario posed a number of challenges to successful use of RL, in-\ncluding multiple disparate reward functions, limited decision sampling rates, and\npathologies arising when using multiple sensor readings as state variables. We\ndescribe innovative practical solutions to these challenges, and demonstrate clear\nperformance improvements over both hand-designed policies as well as obvious\n\u201ccookbook\u201d RL implementations.\n\n1 Introduction\n\nEnergy consumption is a major and growing concern throughout the IT industry as well as for\ncustomers and for government regulators concerned with energy and environmental matters. To cite\na prominent example, the US Congress recently mandated a study of the power ef\ufb01ciency of servers,\nincluding a feasibility study of an Energy Star standard for servers and data centers [16]. Growing\ninterest in power management is also apparent in the formation of the Green Grid, a consortium of\nsystems and other vendors dedicated to improving data center power ef\ufb01ciency [7]. Recent trade\npress articles also make it clear that computer purchasers and data center operators are eager to\nreduce power consumption and the heat densities being experienced with current systems.\n\nIn response to these concerns, researchers are tackling intelligent power control of processors, mem-\nory chips and whole systems, using technologies such as processor throttling, frequency and voltage\nmanipulation, low-power DRAM states, feedback control using measured power values, and packing\nand virtualization to reduce the number of machines that need to be powered on to run a workload.\n\nThis paper presents a reinforcement learning (RL) approach to developing effective control poli-\ncies for real-time management of power consumption in application servers. Such power manage-\nment policies must make intelligent tradeoffs between power and performance, as running servers\nin low-power modes inevitably degrades the application performance. Our approach to this entails\ndesigning a multi-criteria objective function Upp taking both power and performance into account,\nand using it to give reward signals in reinforcement learning. We let Upp be a function of mean\n\n1\n\n\fapplication response time RT , and total power Pwr consumed by the servers in a decision interval.\nSpeci\ufb01cally, Upp subtracts a linear power cost from a performance-based utility U (RT ):\n\nUpp(RT, Pwr) = U (RT ) \u2212 \u01eb \u2217 Pwr\n\n(1)\n\nwhere \u01eb is a tunable coef\ufb01cient expressing the relative value of power and performance objec-\ntives. This approach admits other objective functions such as \u201cperformance value per watt\u201d\nUpp = U (RT )/Pwr, or a simple performance-based utility Upp = U (RT ) coupled with a con-\nstraint on total power.\n\nThe problem of jointly managing performance and power in IT-systems was only recently studied in\nthe literature [5, 6, 17]. Existing approaches use knowledge-intensive and labor-intensive modeling,\nsuch as developing queuing-theoretic or control-theoretic performance models. RL methods can\npotentially avoid such knowledge bottlenecks, by automatically learning high-quality management\npolicies using little or no built-in system speci\ufb01c knowledge. Moreover, as we discuss later, RL may\nhave the merit of properly handling complex dynamic and delayed consequences of decisions.\n\nIn Section 2 we give details of our laboratory testbed, while Section 3 describes our RL approach.\nResults are presented in Section 4, and the \ufb01nal section discusses next steps in our ongoing research\nand ties to related work.\n\n2 Experimental Testbed\n\nFigure 1 provides a high-level overview of our experimental testbed. In brief, a Workload Gener-\nator produces an HTTP-based workload of dynamically varying intensity that is routed to a blade\ncluster, i.e., a collection of blade servers contained in a single chassis. (Speci\ufb01cally, we use an IBM\nBladeCenter containing xSeries HS20 blade servers.) A commercial performance manager and our\nRL-based power manager strive to optimize a joint power-performance objective cooperatively as\nload varies, each adjusting its control parameters individually while sharing certain information\nwith the other manager. RL techniques (described subsequently) are used to train a state-action\nvalue function which de\ufb01nes the power manager\u2019s control policy. The \u201cstate\u201d is characterized by\na set of observable performance, power and load intensity metrics collected in our data collection\nmodule as detailed below. The \u201caction\u201d is a throttling of CPU frequency1 that is achieved by setting\na \u201cpowercap\u201d on each blade that provides an upper limit on the power that the blade may consume.\nGiven this limit, a feedback controller embedded in the server\u2019s \ufb01rmware [11] continuously monitors\nthe power consumption, and continuously regulates the CPU clock speed so as to keep the power\nconsumption close to, but not over, the powercap limit. The CPU throttling affects both application\nperformance as well as power consumption, and the goal of learning is to achieve the optimal level\nof throttling in any given state that maximizes cumulative discounted values of joint reward Upp.\nWe control workload intensity by varying the number of clients nc sending HTTP requests. We\nvaried nc in a range from 1 to 50 using a statistical time-series model of web traf\ufb01c derived from\nobservations of a highly accessed Olympics web site [14]. Clients behave according to a closed-loop\nmodel [12] with exponentially distributed think times of mean 125 msec.\n\nThe commercial performance manager is WebSphere Extended Deployment (WXD)[18], a multi-\nnode webserver environment providing extensive data collection and performance management\nfunctionality. WXD manages the routing policy of the Workload Distributer as well as control\nparameters on individual blades, such as the maximum workload concurrency.\n\nOur data collector receives several streams of data and provides a synchronized report to the power\npolicy evaluator on a time scale \u03c4l (typically set to 5 seconds). Data generated on much faster time\nscales than \u03c4l are time-averaged over the interval, otherwise the most recent values are reported.\nAmong the aggregated data are several dozen performance metrics collected by a daemon running\non the WXD data server, such as mean response time, queue length and number of CPU cycles per\ntransaction; CPU utilization and effective frequency collected by local daemons on each blade; and\ncurrent power and temperature measurements collected by the \ufb01rmware on each blade, which are\npolled using IPMI commands sent from the BladeCenter management module.\n\n1An alternative technique with different power/performance trade-offs is Dynamic Voltage and Frequency\n\nScaling (DVFS).\n\n2\n\n\fManager-to-manager \n\nInteractions\n\n   Performance\n\n Manager\n\n (WebSphere XD)\n\nPower\n\nManager\n\nControl\nPolicy\n\nPerformance\nData\n\nPower\nData\n\nControl\nPolicy\n\nr\no\nt\nu\nb\ni\nr\nt\ns\ni\nD\nd\na\no\nl\nk\nr\no\nW\n\n \n\ns\nt\ns\ne\nu\nq\ne\nR\n\nWorkload\nGenerator\n\n \n \n \n\nP\nT\nT\nH\n\nBlade Chassis\n\nBlade System\n\nBlade System\n\nBlade System\n\nt\nn\ne\nm\nn\ng\ni\ns\ns\nA\n\n \nr\ne\nw\no\nP\n\nPower\n\nFigure 1: Overview of testbed environment.\n\n2.1 Utility function de\ufb01nition\n\nOur speci\ufb01c performance-based utility U (RT ) in Eq. 1 is a piecewise linear function of response\ntime RT which returns a maximum value of 1.0 when RT is less than a speci\ufb01ed threshold RT0,\nand which drops linearly when RT exceeds RT0, i.e.,\n\nU (RT /RT0) = \u00bd 1.0\n\n2.0 \u2212 RT /RT0\n\nif RT \u2264 RT0\notherwise\n\n(2)\n\nSuch a utility function re\ufb02ects the common assumptions in customer service level agreements that\nthere is no incentive to improve the performance once it reaches the target threshold, and that there\nis always a constant incentive to improve performance if it violates the threshold.\nIn all of our\nexperiments, we set RT0 = 1000 msec, and we also set the power scale factor \u01eb = 0.01 in Eq. 1.\nAt this value of \u01eb the power-performance tradeoff is strongly biased in favor of performance, as is\ncommonly desired in today\u2019s data centers. However, larger values of \u01eb could be appropriate in future\nscenarios where power is much more costly, in which case the optimal policies would tolerate more\nfrequent performance threshold violations in order to save more aggressively on power consumption.\n\n2.2 Baseline Powercap Policies\n\nTo assess the effectiveness of our RL-based power management policies, we compare with two\ndifferent benchmark policies: \u201cUN\u201d (unmanaged) and \u201cHC\u201d (hand-crafted). The unmanaged policy\nalways sets the powercap to a maximal value of 120W; we veri\ufb01ed that the CPU runs at the highest\nfrequency under all load conditions with this setting.\n\nThe hand-crafted policy was created as follows. We measured power consumption on a blade server\nat extremely low (nc = 1) and high (nc = 50) loads, \ufb01nding that in all cases the power consumption\nranged between 75 and 120 watts. Given this range, we established a grid of sample points, with p\u03ba\nrunning from 75 watts to 120 watts in increments of 5 watts, and the number of clients running from\n0 to 50 in increments of 5. For each of the 10 possible settings of p\u03ba, we held nc \ufb01xed at 50 for 45\nminutes to permit WXD to adapt to the workload, and then decremented nc by 5 every 5 minutes.\nFinally, the models RT (p\u03ba, nc) and P wr(p\u03ba, nc), were derived by linearly interpolating for the RT\nand P wr between the sampled grid points.\nWe substitute these models into our utility function Upp(RT, P wr) to obtain an equivalent util-\nity function U \u2032 depending on p\u03ba and nc, i.e., U \u2032(p\u03ba, nc) = Upp(RT (p\u03ba, nc), P wr(p\u03ba, nc)). We\ncan then choose the optimal powercap for any workload intensity nc by optimizing U \u2032: p\u2217\n\u03ba(nc) =\narg maxp\u03ba U \u2032(p\u03ba, nc).\n\n3\n\n\f3 Reinforcement Learning Approach\n\nOne may naturally question whether RL could be capable of learning effective control policies for\nsystems as complex as a population of human users interacting with a commercial web application.\nSuch systems are surely far from full observability in the MDP sense. Without even considering\nwhether the behavior of users is \u201cMarkovian,\u201d we note that the state of a web application may\ndepend, for example, on the states of the underlying middleware and Java Virtual Machines (JVMs),\nand these states are not only unobservable, they also have complex historical dependencies on prior\nload and performance levels over multiple time scales. Despite such complexities, we have found in\nour earlier work [15, 9] that RL can in fact learn decent policies when using severely limited state\ndescriptions, such as a single state variable representing current load intensity. The focus of our\nwork in this paper is to examine empirically whether RL may obtain better policies by including\nmore observable metrics in the state description.\n\nAnother important question is whether current decisions have long-range effects, or if it suf\ufb01ces to\nsimply learn policies that optimize immediate reward. The answer appears to vary in an interest-\ning way: under low load conditions, the system response to a decision is fairly immediate, whereas\nunder conditions of high queue length (which may result from poor throttling decisions), the respon-\nsiveness to decisions may become sluggish and considerably delayed.\n\nOur reinforcement learning approach leverages our recent \u201cHybrid RL\u201d approach [15], which orig-\ninally was applied to autonomic server allocation. Hybrid RL is a form of of\ufb02ine (batch) RL that\nentails devising an initial control policy, running the initial policy in the live system and logging a set\nof (state, action, reward) tuples, and then using a standard RL/function approximator combination\nto learn a value function V (s, a) estimating cumulative expected reward of taking action a in state\ns. (The term \u201cHybrid\u201d refers to the fact that expert domain knowledge can be engineered into the\ninitial policy without needing explicit engineering or interfacing into the RL module.) The learned\nvalue function V then implies a policy of selecting the action a\u2217 in state s with highest expected\nvalue, i.e., a\u2217 = arg maxa V (s, a).\nFor technical reasons detailed below, we use the Sarsa(0) update rule rather than Q-Learning (note\nthat unlike textbook Sarsa, decisions are made by an external \ufb01xed policy). Following [15], we\nset the discount parameter \u03b3 = 0.5; we found some preliminary evidence that this is superior to\nsetting \u03b3 = 0.0 but haven\u2019t been able to systematically study the effect of varying \u03b3. We also\nperform standard direct gradient training of neural net weights: we train a multilayer perceptron\nwith 12 sigmoidal hidden units, using backprop to compute the weight changes. Such an approach\nis appealing, as it is simple to implement and has a proven track record of success in many practical\napplications. There is a theoretical risk that the approach could produce value function divergence.\nHowever, we have not seen such divergence in our application. Were it to occur, it would not\nentail any live performance costs, since we train of\ufb02ine. Additionally, we note that instead of direct\ngradient training, we can use Baird\u2019s residual gradient method [4], which guarantees convergence to\nlocal Bellman error minima. In practice we \ufb01nd that direct gradient training yields good convergence\nto Bellman error minima in \u223c5-10K training epochs, requiring only a few CPU minutes on a 3GHz\nworkstation.\n\nIn implementing an initial policy to be used with Hybrid RL, one would generally want to exploit\nthe best available human-designed policy, combined with suf\ufb01cient randomized exploration needed\nby RL, in order to achieve the best possible learned policy. However, in view of the dif\ufb01culty\nexpected in designing such initial policies, it would be advantageous to be able to learn effective\npolicies starting from simplistic initial policies. We have therefore trained our RL policies using an\nextremely simple performance-biased random walk policy for setting the powercap, which operates\nas follows: At every decision point, p\u03ba either is increased by 1 watt with probability p+, or decreased\nby 1 watt with probability p\u2212 = (1 \u2212 p+). The upward bias p+ depends on the ratio r = RT/RT0\nof current mean response time to response time threshold according to: p+ = r/(1 + r). Note that\nthis rule implies an unbiased random walk when r = 1 and that p+ \u2192 1 for r \u226b 1, while p+ \u2192 0\nwhen r \u226a 1. This simple rule seems to strike a good balance between keeping the performance\nnear the desired threshold, while providing plenty of exploration needed by RL, as can been seen in\nFigure 2.\n\nHaving collected training data during the execution of an initial policy, the next step of Hybrid RL is\nto design an (input, output) representation and functional form of the value function approximator.\n\n4\n\n\fcap\n\ngoal\n\n)\ns\nt\nt\na\nw\n\n(\n \nr\ne\nw\no\nP\ng\nv\nA\n\n \n\n)\nc\ne\ns\nm\n\n \n\n(\n \nT\nR\ng\nv\nA\n\ns\nt\nn\ne\n\ni\nl\n\nC\n\n 120\n 110\n 100\n 90\n 80\n 70\n\n 3000\n 2500\n 2000\n 1500\n 1000\n 500\n 0\n\n 50\n 40\n 30\n 20\n 10\n 0\n\n(c)\n\n(b)\n\n(a)\n\n0\n\n500\n\n1000\n\n1500\n\n2000\n\n2500\n\n3000\n\n3500\n\n4000\n\nTime (x 5 sec)\n\nFigure 2: Traces of (a) workload intensity, (b) mean response time, and (c) powercap and consumed\npower of the random-walk (RW) powercap policy.\n\nWe have initially used the basic input representation studied in [15], in which the state s is repre-\nsented using a single metric of workload intensity (number of clients nc), and the action a is a single\nscalar variable\u2014the powercap p\u03ba. This scheme robustly produces decent learned policies, with little\nsensitivity to exact learning algorithm parameter settings. In later experiments, we have expanded\nthe state representation to a much larger set of 14 state variables, and \ufb01nd that substantial improve-\nments in learned policies can be obtained, provided that certain data pre-processing techniques are\nused, as detailed below.\n\n3.1 System-speci\ufb01c innovations\n\nIn our research in this application domain, we have devised several innovative \u201ctricks\u201d enabling us\nto achieve substantially improved RL performance. Such tricks are worth mentioning as they are\nlikely to be of more general use in other problem domains with similar characteristics.\n\nFirst, to represent and learn V , we could employ a single output unit, trained on the total util-\nity (reward) using Q-Learning. However, we can take advantage of the fact that total utility Upp\nin equation 1 is a linear combination of performance utility U and power cost \u2212\u01eb \u2217 P wr. Since\nthe separate reward components are generally observable, and since these should have completely\ndifferent functional forms relying on different state variables, we propose training two separate func-\ntion approximators estimating future discounted reward components Vperf and Vpwr respectively.\nThis type of \u201cdecompositional reward\u201d problem has been studied for tabular RL in [13], where it\nis shown that learning the value function components using Sarsa provably converges to the correct\ntotal value function. (Note that Q-Learning cannot be used to train the value function components,\nas it incorrectly assumes that the optimal policy optimizes each individual component function.)\nSecond, we devised a new type of neuronal output unit to learn Vperf . This is motivated by the\nshape of U, which is a piecewise linear function of RT , with constant value for low RT and linearly\ndecreasing for large RT . This functional form is is not naturally approximated by either a linear or\na sigmoidal transfer function. However, by noting that the derivative of U is a step function (chang-\ning from 0 to -1 at the threshold), and that sigmoids give a good approximation to step functions,\nthis suggests using an output transfer function that behaves as the integral of a sigmoid function.\n\nSpeci\ufb01cally, our transfer function has the form Y (x) = 1 \u2212 \u03c7(x) where \u03c7(x) = R \u03c3(x)dx + C,\n\nwhere \u03c3(x) = 1/(1 + exp(\u2212x)) is the standard sigmoid function, and the integration constant C is\nchosen so that \u03c7 \u2192 0 as x \u2192 \u2212\u221e. We \ufb01nd that this type of output unit is easily trained by standard\nbackprop and provides quite a good approximation to the true expected rewards.\nWe have also trained separate neural networks to estimate Vpwr using a similar hidden layer archi-\ntecture and a standard linear output unit. However, we found only a slight improvement in Bellman\nerror over a simple estimator of predicted power \u223c= p\u03ba (although this is not always a good estimate).\n\n5\n\n\fHence for simplicity we used Vpwr = \u2212\u01eb \u2217 p\u03ba in computing the overall learned policy maximizing\nV = Vperf + Vpwr.\nThirdly, we devised a data pre-processing technique to address a speci\ufb01c rate limitation in our sys-\ntem that the powercap decision p\u03ba as well as the number of clients nc can only be changed every\n30 seconds, whereas we collect state data from the system every 5 seconds. This limitation was\nimposed because faster variations in effective CPU speed or in load disrupt WXD\u2019s functionality, as\nits internal models estimate parameters on much slower time scales, and in particular, it assumes that\nCPU speed is a constant. As a result, we cannot do standard RL on the 5 second interval data, since\nthis would presume the policy\u2019s ability to make a new decision every 5 seconds. A simple way to\naddress this would be to discard data points where a decision was not made (5/6 of the data), but this\nwould make the training set much smaller, and we would lose valuable state transition information\ncontained in the discarded samples. As an alternative, we divide the entire training set into six sub-\nsets according to line number mod-6, so that within each subset, adjacent data points are separated\nby 30 second intervals. We then concatenate the subsets to form one large training set, with no loss\nof data, where all adjacent intervals are 30 seconds long. In effect, a sweep through such a dataset\nreplays the experiment six times, corresponding to the six different 5-second phases within the 30-\nsecond decision cycle. As we shall see in the following section, such rearranged datasets result in\nsubstantially more stable policies.\n\nFinally, we realized that in the arti\ufb01cially constructed dataset described above, there is an inaccu-\nracy in training on samples in the \ufb01ve non-decision phases: standard RL would presume that the\npowercap decision is held constant over the full 30 seconds until the next recorded sample, whereas\nwe know that decision actually changes somewhere in the middle of the interval, depending on the\nphase. To obtain the best approximation to a constant decision over such intervals, we compute an\nequally weighted average \u00afp\u03ba of the recorded decisions at times {t, t+5, t+10, t+15, t+20, t+25}\nand train on \u00afp\u03ba as the effective decision that was made at time t. This change results in a signi\ufb01cant\nreduction (\u223c 40%) in Bellman error, and the combination of this with the mod-6 data reordering\nenables us to obtain substanial improvements in policy performance.\n\n4 Results\n\n)\nc\ne\ns\nm\n\n(\n \ne\nm\n\ni\nt\n \ne\ns\nn\no\np\ns\ne\nr\n\n1200\n\n1000\n\n(a)\n\n800\n\n600\n\n400\n\n(c)\n\n)\nt\nn\ne\nc\n(\n \ne\nr\nu\nt\na\nr\ne\np\nm\ne\nt\n\n56\n\n54\n\n52\n\n50\n\n48\n\n46\n\nUN\n\nHC\n\nRW 2NN 15NN 15NNp\n\nUN\n\nHC\n\nRW 2NN 15NN 15NNp\n\n)\ns\nt\nt\na\nw\n\n(\n \nr\ne\nw\no\np\n\ny\nt\ni\nl\ni\nt\nu\n\n110\n\n105\n\n100\n\n95\n\n90\n\n0.25\n\n0.2\n\n0.15\n\n0.1\n\n0.05\n\n0\n\n(b)\n\nUN\n\nHC\n\nRW 2NN 15NN 15NNp\n\n(d)\n\nUN\n\nHC\n\nRW 2NN 15NN 15NNp\n\nFigure 3: Comparison of mean metrics (a) response time, (b) power consumed, (c) temperature and\n(d) utility for six different power management policies: \u201cUN\u201d (unmanaged), \u201cHC\u201d (hand-crafted),\n\u201cRW\u201d (random walk), \u201c2NN\u201d (2-input neural net), \u201c15NN\u201d (15-input neural net, no pre-processing),\n\u201c15NNp\u201d (15-input neural net with pre-processing).\n\nWhile we have conducted experiments in other work involving multiple blade servers, in this section\nwe focus on experiments involving a single blade. Fig. 3 plots various mean performance metrics in\nidentical six-hour test runs using identical workload traces for six different power management poli-\ncies: \u201cUN\u201d and \u201cHC\u201d denote the unmanaged and hand-crafted policies described in Sec. 2.2; \u201cRW\u201d\nis the random-walk policy of Sec. 3; \u201c2NN\u201d denotes a two-input (single state variable) neural net;\n\u201c15NN\u201d refers to a 15-input neural net without any data pre-processing as described in Sec. 3.1, and\n\n6\n\n\f\u201c15NNp\u201d indicates a 15-input neural net using said pre-processing. In the \ufb01gure, the performance\nmetrics plotted are: (a) mean response time, (b) mean power consumed, (c) mean temperature, and\nmost importantly, (d) mean utility. Standard error in estimates of these mean values are quite small,\nas indicated by error bars which lie well within the diamond-shaped data points. Since the runs use\nidentical workload traces, we can also assess signi\ufb01cance of the differences in means across policies\nvia paired T-tests; exhaustive pairwise comparisons show that in all cases, the null hypothesis of no\ndifference in mean metrics is rejected at 1% signi\ufb01cance level with P-value \u2264 10\u22126.\nWe see in Fig. 3 that all RL-based policies, after what is effectively a single round of policy iter-\nation, signi\ufb01cantly outperform the original random walk policy which generated the training data.\nUsing only load intensity as a state variable, 2NN achieves utility close to (but not matching) the\nhand-crafted policy. 15NN is disappointing in that its utility is actually worse than 2NN, for rea-\nsons that we discuss below. Comparing 15NNp with 15NN shows that pre-processing yields great\nimprovements; 15NNp is clearly the best of the six policies. Breaking down overall utility into sep-\narate power and performance components, we note that all RL-based policies achieve greater power\nsavings than HC at the price of somewhat higher mean response times. An additional side bene\ufb01t\nof this is lower mean temperatures, as shown in the lower left plot; this implies both lower cooling\ncosts as well as prolonged machine life.\n\ncap\n\ngoal\n\ncap\n\ngoal\n\n)\ns\nt\nt\na\nw\n\n(\n \nr\ne\nw\no\nP\ng\nv\nA\n\n \n\n)\nc\ne\ns\nm\n\n \n\n(\n \nT\nR\ng\nv\nA\n\n)\ns\nt\nt\na\nw\n\n(\n \nr\ne\nw\no\nP\ng\nv\nA\n\n \n\n)\nc\ne\ns\nm\n\n \n\n(\n \nT\nR\ng\nv\nA\n\ns\nt\nn\ne\n\ni\nl\n\nC\n\n 120\n 110\n 100\n 90\n 80\n 70\n 1600\n\n 1200\n\n 800\n\n 400\n\n 0\n\n 120\n 110\n 100\n 90\n 80\n 70\n 1600\n\n 1200\n\n 800\n\n 400\n\n 0\n 60\n 50\n 40\n 30\n 20\n 10\n 0\n\n(e)\n\n(d)\n\n(c)\n\n(b)\n\n(a)\n\n0\n\n500\n\n1000\n\n1500\n\n2000\n\n2500\n\n3000\n\n3500\n\n4000\n\nTime (x5 sec)\n\n)\ns\nt\nt\na\nw\n\n(\n \nr\ne\nw\no\nP\ng\nv\nA\n\n \n\n)\nc\ne\ns\nm\n\n \n\n(\n \nT\nR\ng\nv\nA\n\n)\ns\nt\nt\na\nw\n\n(\n \nr\ne\nw\no\nP\ng\nv\nA\n\n \n\n)\nc\ne\ns\nm\n\n \n\n(\n \nT\nR\ng\nv\nA\n\n)\ns\nt\nt\na\nw\n\n(\n \nr\ne\nw\no\nP\ng\nv\nA\n\n \n\n)\nc\ne\ns\nm\n\n \n\n(\n \nT\nR\ng\nv\nA\n\n 120\n 110\n 100\n 90\n 80\n 70\n 1600\n\n 1200\n\n 800\n\n 400\n\n 0\n\n 120\n 110\n 100\n 90\n 80\n 70\n 1600\n\n 1200\n\n 800\n\n 400\n\n 0\n\n 120\n 110\n 100\n 90\n 80\n 70\n 1600\n\n 1200\n\n 800\n\n 400\n\n(k)\n\n(j)\n\n(i)\n\n(h)\n\n(g)\n\n(f )\n\ncap\n\ngoal\n\ncap\n\ngoal\n\ncap\n\ngoal\n\n 0\n\n0\n\n500\n\n1000\n\n1500\n\n2000\n\n2500\n\n3000\n\n3500\n\n4000\n\nTime (x5 sec)\n\nFigure 4: Traces of the \ufb01ve non-random policies: (a) workload intensity; (b) UN response time; (c)\nUN powercap; (d) HC response time; (e) HC powercap; (f) 2NN response time; (g) 2NN powercap;\n(h) 15NN response time; (i) 15NN powercap; (j) 15NNp response time; (k) 15NNp powercap.\n\nFig. 4 shows the actual traces of response time, powercap and power consumed in all experiments\nexcept the random walk, which was plotted earlier. The most salient points to note are that 15NNp\nexhibits the steadiest response time, keeping closest to the response time goal, and that the powercap\ndecsions of 15NN show quite large short-term \ufb02uctuations. We attribute the latter behavior to \u201cover-\nreacting\u201d to response time \ufb02uctuations above or below the target value. Such behavior may well\nbe correct if the policy could reset every 5 seconds, as 15NN presumes. In this case, the policy\ncould react to a response time \ufb02ucutation by setting an extreme powercap value in an attempt to\nquickly drive the response time back to the goal value, and then backing off to a less extreme value\n5 seconds later. However, such behavior would be quite poor in the actual system, in which the\nextreme powercap setting is held \ufb01xed for 30 seconds.\n\n7\n\n\f5 Summary and related work\n\nThis paper presented a successful application of batch RL combined with nonlinear function ap-\nproximation in a new and challenging domain of autonomic management of power and performance\nin web application servers. We addressed challenges arising both from operating in real hardware,\nand from limitations imposed by interoperating with commercial middleware. By training on data\nfrom a simple random-walk initial policy, we achieved high-quality management polices that out-\nperformed the best available hand-crafted policy. Such policies save more than 10% on server power\nwhile keeping performance close to a desired target.\n\nIn our ongoing and future work, we are aiming to scale the approach to an entire Blade cluster, and\nto achieve much greater levels of power savings. With the existing approach it appears that power\nsavings closer to 20% could be obtained simply by using more realistic web workload pro\ufb01les in\nwhich high-intensity spikes are brief, and the ratio of peak-to-mean workload is much higher than\nin our current traf\ufb01c model. It also appears that savings of \u223c30% are plausible when using multi-\ncore processors [8]. Finally, we are also aiming to learn policies for powering machines off when\nfeasible; this offers the potential to achieve power savings of 50% or more. In order to scale our\napproach to larger systems, we can leverage the fact that Blade clusters usually have sets of identical\nmachines. All servers within such a homogeneous set can be managed in an identical fashion by the\nperformance and power managers, thereby making the size of the overall state space and the action\nspace more tractable for RL.\n\nAn important component of our future work is also to improve our current RL methodology. Be-\nyond Hybrid RL, there has been much recent research in of\ufb02ine RL methods, including LSPI [10],\nApprenticeship Learning [2], Differential Dynamic Programming [1], and \ufb01tted policy iteration\nminimizing Bellman residuals [3]. These methods are of great interest to us, as they typically have\nstronger theoretical guarantees than Hybrid RL, and have delivered impressive performance in appli-\ncations such as helicopter aerobatics. For powering machines on and off, we are especially interested\nin of\ufb02ine model-based RL approaches: as the number of training samples that can be acquired is\nlikely to be severely limited, it will be important to reduce sample complexity by learning explicit\nstate-transition models.\n\nReferences\n[1] P. Abbeel, A. Coates, M. Quigley, and A. Y. Ng. An application of reinforcement learning to aerobatic\n\nhelicopter \ufb02ight. In Proc. of NIPS-06, 2006.\n\n[2] P. Abbeel and A. Y. Ng. Exploration and apprenticeship learning in reinforcement learning. In Proc. of\n\n[3] A. Antos, C. Szepesvari, and R. Munos. Learning near-optimal policies with bellman-residual minimiza-\n\ntion based \ufb01tted policy iteration and a single sample path. In Proc. of COLT-06, 2006.\n\n[4] L. Baird. Residual algorithms: Reinforcement learning with function approximation. In Proc. of ICML-\n\n[5] Y. Chen et al. Managing server energy and operational costs in hosting centers. In Proc. of SIGMETRICS,\n\nICML-05, 2005.\n\n95, 1995.\n\n2005.\n\n[6] M. Femal and V. Freeh. Boosting data center performance through non-uniform power allocation. In\n\nSecond Intl. Conf. on Autonomic Computing, 2005.\n\n[7] Green Grid Consortium. Green grid. http://www.thegreengrid.org, 2006.\n[8] J. Chen et al. Datacenter power modeling and prediction. UC Berkeley RAD Lab presentation, 2007.\n[9] J. O. Kephart, H. Chan, R. Das, D. Levine, G. Tesauro, F. Rawson, and C. Lefurgy. Coordinating multiple\n\nautonomic managers to achieve speci\ufb01ed power-performance tradeoffs. In Proc. of ICAC-07, 2007.\n\n[10] M. G. Lagoudakis and R. Parr. Least-squares policy iteration. J. of Machine Learning Research, 4:1107\u2013\n\n[11] C. Lefurgy, X. Wang, and M. Ware. Server-level power control. In Proc. of ICAC-07, 2007.\n[12] D. Menasce and V. A. F. Almeida. Capacity Planning for Web Performance: Metrics, Models, and\n\n[13] S. Russell and A. L. Zimdars. Q-decomposition for reinforcement learning agents. In Proc. of ICML-03,\n\n1149, 2003.\n\nMethods. Prentice Hall, 1998.\n\npages 656\u2013663, 2003.\n\n[14] M. S. Squillante, D. D. Yao, and L. Zhang. Internet traf\ufb01c: Periodicity, tail behavior and performance\n\nimplications. In System Performance Evaluation: Methodologies and Applications, 1999.\n\n[15] G. Tesauro, N. K. Jong, R. Das, and M. N. Bennani. A hybrid reinforcement learning approach to auto-\n\nnomic resource allocation. In Proc. of ICAC-06, pages 65\u201373, 2006.\n\n[16] United States Environmental Protection Agency. Letter to Enterprise Server Manufacturers and Other\n\nStakeholders. http://www.energystar.gov, 2006.\n\n[17] M. Wang et al. Adaptive Performance Control of Computing Systems via Distributed Cooperative Con-\n\ntrol: Application to Power Management in Computer Clusters. In Proc. of ICAC-06, 2006.\n\n[18] WebSphere Extended Deployment. http://www.ibm.com/software/webservers/appserv/extend/, 2007.\n\n8\n\n\f", "award": [], "sourceid": 906, "authors": [{"given_name": "Gerald", "family_name": "Tesauro", "institution": null}, {"given_name": "Rajarshi", "family_name": "Das", "institution": null}, {"given_name": "Hoi", "family_name": "Chan", "institution": null}, {"given_name": "Jeffrey", "family_name": "Kephart", "institution": null}, {"given_name": "David", "family_name": "Levine", "institution": null}, {"given_name": "Freeman", "family_name": "Rawson", "institution": null}, {"given_name": "Charles", "family_name": "Lefurgy", "institution": null}]}