{"title": "Welfare Guarantees from Data", "book": "Advances in Neural Information Processing Systems", "page_first": 3768, "page_last": 3777, "abstract": "Analysis of efficiency of outcomes in game theoretic settings has been a main item of study at the intersection of economics and computer science. The notion of the price of anarchy takes a worst-case stance to efficiency analysis, considering instance independent guarantees of efficiency. We propose a data-dependent analog of the price of anarchy that refines this worst-case assuming access to samples of strategic behavior. We focus on auction settings, where the latter is non-trivial due to the private information held by participants. Our approach to bounding the efficiency from data is robust to statistical errors and mis-specification. Unlike traditional econometrics, which seek to learn the private information of players from observed behavior and then analyze properties of the outcome, we directly quantify the inefficiency without going through the private information. We apply our approach to datasets from a sponsored search auction system and find empirical results that are a significant improvement over bounds from worst-case analysis.", "full_text": "Welfare Guarantees from Data\n\nDarrell Hoy\n\nUniversity of Maryland\n\ndarrell.hoy@gmail.com\n\nDenis Nekipelov\n\nUniversity of Virginia\ndenis@virginia.edu\n\nVasilis Syrgkanis\nMicrosoft Research\n\nvasy@microsoft.com\n\nAbstract\n\nAnalysis of ef\ufb01ciency of outcomes in game theoretic settings has been a main item\nof study at the intersection of economics and computer science. The notion of\nthe price of anarchy takes a worst-case stance to ef\ufb01ciency analysis, considering\ninstance independent guarantees of ef\ufb01ciency. We propose a data-dependent analog\nof the price of anarchy that re\ufb01nes this worst-case assuming access to samples of\nstrategic behavior. We focus on auction settings, where the latter is non-trivial\ndue to the private information held by participants. Our approach to bounding the\nef\ufb01ciency from data is robust to statistical errors and mis-speci\ufb01cation. Unlike\ntraditional econometrics, which seek to learn the private information of players\nfrom observed behavior and then analyze properties of the outcome, we directly\nquantify the inef\ufb01ciency without going through the private information. We apply\nour approach to datasets from a sponsored search auction system and \ufb01nd empirical\nresults that are a signi\ufb01cant improvement over bounds from worst-case analysis.\n\n1\n\nIntroduction\n\nA major \ufb01eld at the intersection of economics and computer science is the analysis of the ef\ufb01ciency of\nsystems under strategic behavior. The seminal work of [6, 11] triggered a line of work on quantifying\nthe inef\ufb01ciency of computer systems, ranging from network routing, resource allocation and more\nrecently auction marketplaces [10]. However, the notion of the price of anarchy suffers from the\npessimism of worst-case analysis. Many systems can be inef\ufb01cient in the worst-case over parameters\nof the model, but might perform very well for the parameters that arise in practice.\nDue to the large availability of datasets in modern economic systems, we propose a data-dependent\nanalog of the price of anarchy, which assumes access to a sample of strategic behavior from the\nsystem. We focus our analysis on auction systems where the latter approach is more interesting due\nto the private information held by the participants of the system, i.e. their private value for the item at\nsale. Since ef\ufb01ciency is a function of these private parameters, quantifying the inef\ufb01ciency of the\nsystem from samples of strategic behavior is non-trivial. The problem of estimation of the inef\ufb01ciency\nbecomes an econometric problem where we want to estimate a function of hidden variables from\nobserved strategic behavior. The latter is feasible under the assumption that the observed behavior\nis the outcome of an equilibrium of the strategic setting, which connects observed behavior to\nunobserved private information.\nTraditional econometric approaches to auctions [3, 8], address such questions by attempting to\nexactly pin-point the private parameters from the observed behavior and subsequently measuring the\nquantities of interest, such as the ef\ufb01ciency of the allocation. The latter approach is problematic in\ncomplex auction systems for two main reasons: (i) it leads to statistical inef\ufb01ciency, (ii) it requires\nstrong conditions on the connection between observed behavior and private information. Even for a\nsingle-item \ufb01rst-price auction, uniform estimation of the private value of a player from T samples\nof observed bids, can only be achieved at O(T 1/3)-rates [3]. Moreover, uniquely identifying the\nprivate information from the observed behavior, requires a one-to-one mapping between the two\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fquantities. The latter requires strong assumptions on the distribution of private parameters and can\nonly be applied to simple auction rules.\nOur approach bridges the gap between worst-case price of anarchy analysis and statistically and\nmodeling-wise brittle econometric analysis. We provide a data-dependent analog of recent techniques\nfor quantifying the worst-case inef\ufb01ciency in auctions [13, 4, 10], that do not require characterization\nof the equilibrium structure and which directly quantify the inef\ufb01ciency through best-response\narguments, without the need to pin-point the private information. Our approach makes minimal\nassumptions on the distribution of private parameters and on the auction rule and achieves \u02dcO(\nT )-\nrates of convergence for many auctions used in practice, such as the Generalized Second Price (GSP)\nauction [2, 14]. We applied our approach to a real world dataset from a sponsored search auction\nsystem and we portray the optimism of the data-dependent guarantees as compared to their worst-case\ncounterparts [1].\n\n\u221a\n\n2 Preliminaries\n\nWe consider the single-dimensional mechanism design setting with n bidders. The mechanism\ndesigner wants to allocate a unit of good to the bidders, subject to some feasibility constraint on\nthe vector of allocations (x1, . . . , xn). Let X be the space of feasible allocations. Each bidder i has\na private value vi \u2208 [0, H] per-unit of the good, and her utility when she gets allocation xi and is\nasked to make a payment pi is vi \u00b7 xi \u2212 pi. The value of each bidder is drawn independently from\ndistribution with CDF Fi, supported in Vi \u2286 R+ and let F = \u00d7i Fi be the joint distribution.\nAn auction A solicits a bid bi \u2208 B from each bidder i and decides on the allocation vector based on\nan allocation rule X : Bn \u2192 X and a payment rule p : Bn \u2192 Rn. For a vector of values and bids,\nthe utility of a bidder is:\n(1)\nA strategy \u03c3i : Vi \u2192 B, for each bidder i, maps the value of the bidder to a bid. Given an auction A\nand distribution of values F, a strategy pro\ufb01le \u03c3 is a Bayes-Nash Equilibrium (BNE) if each bidder\ni with any value vi \u2208 Vi maximizes her utility in expectation over her opponents bids, by bidding\n\u03c3i(vi).\nThe welfare of an auction outcome is the expected utility generated for all the bidders, plus the\nrevenue of the auctioneer, which due to the form of bidder utilities boils down to being the total value\nthat the bidders get from the allocation. Thus the expected utility of a strategy pro\ufb01le \u03c3 is\n\nUi(b; vi) = vi \u00b7 Xi(b) \u2212 Pi(b).\n\n\uf8f9\uf8fb\n\nvi \u00b7 Xi(\u03c3(v))\n\n\uf8ee\uf8f0(cid:88)\n\ni\u2208[n]\n\nWELFARE(\u03c3; F) = Ev\u223cF\n\nWe denote with OPT(F) the expected optimal welfare: OPT(F) = Ev\u223cF[maxx\u2208X(cid:80)\n\ni\u2208[n] vi \u00b7 xi].\nWorst-case Bayes-Nash price of anarchy. The Bayesian price of anarchy of an auction is de\ufb01ned\nas the worst-case ratio of welfare in the optimal auction to the welfare in a Bayes-Nash equilibrium\nof the original auction, taken over all value distributions and over all equilibria. Let BN E(A, F) be\nthe set of Bayes-Nash equilibria of an auction A, when values are drawn from distributions F. Then:\n\n(2)\n\nPOA =\n\nsup\n\nF,\u03c3\u2208BN E(F)\n\nOPT(F)\n\nWELFARE(\u03c3; F)\n\n(3)\n\n3 Distributional Price of Anarchy: Re\ufb01ning the POA with Data\nWe will assume that we observe T samples b1:T = {b1, . . . , bT} of bid pro\ufb01les from running T\ntimes an auction A. Each bid pro\ufb01le bt is drawn i.i.d. based on an unknown Bayes-Nash equilibrium\n\u03c3 of the auction, i.e.: let D denote the distribution of the random variable \u03c3(v), when v is drawn\nfrom F. Then bt are i.i.d. samples from D. Our goal is to re\ufb01ne our prediction on the ef\ufb01ciency of the\nauction and compute a bound on the price of anarchy of the auction conditional on the observed data\nset. More formally, we want to derive statements of the form: conditional on b1:T , with probability\nat least 1 \u2212 \u03b4: WELFARE(\u03c3; F) \u2265 1\n\u02c6\u03c1 OPT(F), where \u02c6\u03c1 is the empirical analogue of the worst-case\nprice of anarchy ratio.\n\n2\n\n\fIn\ufb01nite data limit We will tackle this question in two steps, as is standard in estimation theory.\nFirst we will look at the in\ufb01nite data limit where we know the actual distribution of equilibrium bids\nD. We de\ufb01ne a notion of price of anarchy that is tailored to an equilibrium bid distribution, which we\nrefer to as the distributional price of anarchy. In Section 4 we give a distribution-dependent upper\nbound on this ratio for any single-dimensional auction. Subsequently, in Section 5, we show how one\ncan estimate this upper bound on the distributional price of anarchy from samples.\nGiven a value distribution F and an equilibrium \u03c3, let D(F, \u03c3) denote the resulting equilibrium bid\ndistribution. We then de\ufb01ne the distributional price of anarchy as follows:\nDe\ufb01nition 1 (Distributional Price of Anarchy). The distributional price of anarchy DPOA(D) of\nan auction A and a distribution of bid pro\ufb01les D, is the worst-case ratio of welfare in the optimal\nallocation to the welfare in an equilibrium, taken over all distributions of values and all equilibria\nthat could generate the bid distribution D:\n\nDPOA(D) =\n\nsup\n\nF,\u03c3\u2208BN E(F) s.t. D(F,\u03c3)=D\n\nOPT(F)\n\nWELFARE(\u03c3; F)\n\n(4)\n\nThis notion has nothing to do with sampled data-sets, but rather is a hypothetical worst-case quantity\nthat we could calculate had we known the true bid generating distribution D.\nWhat does the extra information of knowing D give us? To answer this question, we \ufb01rst focus\non the optimization problem each bidder faces. At any Bayes-Nash equilibrium each player must\nbe best-responding in expectation over his opponent bids. Observe that if we know the rules of the\nauction and the equilibrium distribution of bids D, then the expected allocation and payment function\nof a player as a function of his bid are uniquely determined:\n\nxi(b;D) = Eb\u2212i\u223cD\u2212i [Xi(b, b\u2212i)]\n\n(5)\nImportantly, these functions do not depend on the distribution of values F, other than through the\ndistribution of bids D. Moreover, the expected revenue of the auction is also uniquely determined:\n\npi(b;D) = Eb\u2212i\u223cD\u2212i [Pi(b, b\u2212i)] .\n\n(cid:34)(cid:88)\n\n(cid:35)\n\nREV(D) = Eb\u223cD\n\nPi(b)\n\n,\n\n(6)\n\ni\n\nThus when bounding the distributional price of anarchy, we can assume that these functions and\nthe expected revenue are known. The latter is unlike the standard price of anarchy analysis, which\nessentially needs to take a worst-case approach to these quantities.\nShorthand notation Through the rest of the paper we will \ufb01x the distribution D. Hence, for brevity\nwe omit it from notation, using xi(b), pi(b) and REV instead of xi(b;D), pi(b;D) and REV(D).\n\n4 Bounding the Distributional Price of Anarchy\n\nWe \ufb01rst upper bound the distributional price of anarchy via a quantity that is relatively easy to\ncalculate as a function of the bid distribution D and hence will also be rather straightforward to\nestimate from samples of D, which we defer to the next section. To give intuition about the upper\nbound, we start with a simple but relevant example of bounding the distributional price of anarchy in\nthe case when the auction A is the single-item \ufb01rst price auction. We then generalize the approach to\nany auction A.\n\n4.1 Example: Single-Item First Price Auction\n\nIn a single item \ufb01rst price auction, the designer wants to auction a single indivisible good. Thus\nthe space of feasible allocations X , are ones where only one player gets allocation xi = 1 and\nother players get allocation 0. The auctioneer solicits bids bi from each bidder and allocates the\ngood to the highest bidder (breaking ties lexicographically), charging him his bid. Let D be the\nequilibrium distribution of bids and let Gi be the CDF of the bid of player i. For simplicity we\nassume that Gi is continuous (i.e. the distribution is atomless). Then the expected allocation of a\nj(cid:54)=i Gj(b) and his expected payment\nis pi(b) = b \u00b7 xi(b), leading to expected utility: ui(b; vi) = (vi \u2212 b)G\u2212i(b).\n\nplayer i from submitting a bid b is equal to xi(b) = G\u2212i(b) =(cid:81)\n\n3\n\n\fThe quantity DPOA is a complex object as it involves the structure of the set of equilibria of the given\nauction. The set of equilibria of a \ufb01rst price auction when bidders values are drawn from different\ndistributions is an horri\ufb01c object.1 However, we can upper bound this quantity by a much simpler\ndata-dependent quantity by simply invoking the fact that under any equilibrium bid distribution no\nplayer wants to deviate from his equilibrium bid. Moreover, this data-dependent quantity can be\nmuch better than its worst-case counterpart used in the existing literature on the price of anarchy.\nLemma 1. Let A be the single item \ufb01rst price auction and let D be the equilibrium distribution of\nbids, then DPOA(D) \u2264 \u00b5(D)\n\nmaxi\u2208[n] Eb\u2212i\u223cD\u2212i [maxj(cid:54)=i bj ]\n\n1\u2212e\u2212\u00b5(D) , where \u00b5(D) =\n\n.\n\nEb\u223cD[maxi\u2208[n] bi]\n\nProof. Let Gi be the CDF of the bid of each player under distribution D. Moreover, let \u03c3 denote the\nequilibrium strategy that leads to distribution D. By the equilibrium condition, we know that for all\nvi \u2208 Vi and for all b(cid:48) \u2208 B,\n\nui(\u03c3i(vi); vi) \u2265 ui(b(cid:48); vi) = (vi \u2212 b(cid:48)) \u00b7 G\u2212i(b(cid:48)).\n\n(7)\n\nexpected maximum other bid which can be expressed as Ti =(cid:82) \u221e\n\nWe will give a special deviating strategy used in the literature [13], that will show that either the\nplayers equilibrium utility is large or the expected maximum other bid is high. Let Ti denote the\n0 1 \u2212 G\u2212i(z)dz. We consider the\nrandomized deviation where the player submits a randomized bid in z \u2208 [0, vi(1 \u2212 e\u2212\u00b5)] with PDF\n(cid:90) vi(1\u2212e\u2212\u00b5)\nf (z) =\n\n\u00b5(vi\u2212z). Then the expected utility from this deviation is:\n\n1\n\nb(cid:48) [ui(b(cid:48); vi)] =\nE\n\n(vi \u2212 z) \u00b7 G\u2212i(z)f (z)dz =\n\n1\n\u00b5\n\n0\n\nG\u2212i(z)dz\n\n(8)\n\nAdding the quantity 1\n\u00b5\n\u00b5 Ti \u2265 vi\n1\nSubsequently, for any x\u2217\n\n1\u2212e\u2212\u00b5\n\n\u00b5\n\n. Invoking the equilibrium condition we get: ui(\u03c3i(vi); vi) + 1\n\n(1 \u2212 G\u2212i(z))dz \u2264 1\n\n\u00b5 Ti on both sides, we get: Eb(cid:48) [ui(b(cid:48); vi)] +\n1\u2212e\u2212\u00b5\n.\n\n\u00b5 Ti \u2265 vi\n\n\u00b5\n\n0\n\n(cid:90) vi(1\u2212e\u2212\u00b5)\n(cid:82) vi(1\u2212e\u2212\u00b5)\ni \u2208 [0, 1]:\n\n0\n\nui(\u03c3i(vi); vi) +\n\n1\n\u00b5\n\nTi \u00b7 x\u2217\n\n1 \u2212 e\u2212\u00b5\n\n\u00b5\n\n.\n\ni\n\ni \u2265 vi \u00b7 x\u2217\n(cid:35)\n\nTiX\u2217\n\n(cid:34)(cid:88)\n\n1\n\u00b5\n\nE\nv\n\n(9)\ni (v) \u2261 1{vi =\n\n(cid:88)\n\nE\nvi\n\ni is the expected allocation of player i under the ef\ufb01cient allocation rule X\u2217\n\nIf x\u2217\nmaxj vj}, then taking expectation of Equation (9) over vi and adding across all players we get:\n\ni\n\n[ui(\u03c3i(vi); vi)] +\n\nThe theorem then follows by invoking the fact that for any feasible allocation x: (cid:80)\n\ni Ti \u00b7 xi \u2264\nmaxi Ti = \u00b5(D)REV(D), using the fact that expected total agent utility plus total revenue at\nequilibrium is equal to expected welfare at equilibrium and setting \u00b5 = \u00b5(D).\n\ni (v)\n\n\u00b5\n\ni\n\n(10)\n\n\u2265 OPT(F)\n\n1 \u2212 e\u2212\u00b5\n\nComparison with worst-case POA In the worst-case, \u00b5(D) is upper bounded by 1, leading to the\nwell-known worst-case price of anarchy ratio of the single-item \ufb01rst price auction of (1 \u2212 1/e)\u22121,\nirrespective of the bid distribution D. However, if we know the distribution D then we can explicitly\nestimate \u00b5, which can lead to a much better ratio (see Figure 1). Moreover, observe that even if\nwe had samples from the bid distribution D, then estimating \u00b5(D) is very easy as it corresponds\nto the ratio of two expectations, each of which can be estimating to within an O( 1\u221a\n) error by a\nsimple average and using standard concentration inequalities. Even thought this improvement, when\ncompared to the worst-case bound might not be that drastic in the \ufb01rst price auction, the extension of\nthe analysis in the next section will be applicable even to auctions where the analogue of the quantity\n\u00b5(D) is not even bounded in the worst-case. In those settings, the empirical version of the price of\nanarchy analysis is of crucial importance to get any ef\ufb01ciency bound.\n\nT\n\n1Even for two bidders with uniformly distributed values U [0, a] and U [0, b], the equilibrium strategy requires\nsolving a complex system of partial differential equations, which took several years of research in economics to\nsolve (see [15, 7])\n\n4\n\n\f4\n\n3\n\n2\n\ny\nh\nc\nr\na\nn\nA\n\nf\no\n\ne\nc\ni\nr\nP\n\nFigure 1: The upper bound on the distributional price of anarchy of an auction\n\n1\u2212e\u2212\u00b5(D as a function of \u00b5(D).\n\u00b5(D)\n\n1\n\n0\n\n1\n\n2\n\u00b5\n\n3\n\n4\n\nknow the bid distribution D we can calculate the equilibrium welfare as Eb\u223cD [(cid:80)\n\nComparison with value inversion approach Apart from being just a primer to our main general\nresult in the next section, the latter result about the data-dependent ef\ufb01ciency bound for the \ufb01rst price\nauction, is itself a contribution to the literature. It is notable to compare the latter result with the\nstandard econometric approach to estimating values in a \ufb01rst price auction pioneered by [3] (see also\n[8]). Traditional non-parametric auction econometrics use the equilibrium best response condition\nto pin-point the value of a player from his observed bid, by what is known as value inversion. In\nparticular, if the function: ui(b(cid:48); vi) = (vi \u2212 b(cid:48)) \u00b7 G\u2212i(b(cid:48)) has a unique maximum for each vi and\nthis maximum is strictly monotone in vi, then given the equilibrium bid of a player bi and given a\ndata distribution D we can reverse engineer the value vi(bi) that the player must have. Thus if we\ni vi(bi) \u00b7 Xi(b)].\nMoreover, we can calculate the expected optimal welfare as: Eb\u223cD [maxi vi(bi)]. Thus we can\npin-point the distributional price of anarchy.\nHowever, the latter approach suffers from two main drawbacks: (i) estimating the value inversion\nfunction vi(\u00b7) uniformly over b from samples, can only happen at very slow rates that are at least\nO(1/T 1/3) and which require differentiability assumptions from the value and bid distribution as\nwell as strong conditions that the density of the value distribution is bounded away from zero in all the\nsupport (with this lower bound constant entering the rates of convergence), (ii) the main assumption\nof the latter approach is that the optimal bid is an invertible function and that given a bid there is\na single value that corresponds to that bid. This assumption might be slightly benign in a single\nitem \ufb01rst price auction, but becomes a harsher assumption when one goes to more complex auction\nschemes. Our result in Lemma 1 suffers neither of these drawbacks: it admits fast estimation rates\nfrom samples, makes no assumption on properties of the value and bid distribution and does not\nrequire invertibility of the best-response correspondence. Hence it provides an upper bound on the\ndistributional price of anarchy that is statistically robust to both sampling and mis-speci\ufb01cation errors.\nThe robustness of our approach comes with the trade-off that we are now only estimating a bound on\nthe ef\ufb01ciency of the outcome, rather than exactly pinpointing it.\n\n4.2 Generalizing to any Single-Dimensional Auction Setting\n\nOur analysis on DPOA is based on the reformulation of the auction rules as an equivalent pay-your-\nbid auction and then bounding the price of anarchy as a function of the ratio of how much a player\nneeds to pay in an equivalent pay-your-bid auction, so as to acquire his optimal allocation vs. how\nmuch revenue is the auctioneer collecting. For any auction, we can re-write the expected utility of a\nbid b:\n\n(11)\n\n(cid:18)\n\n(cid:19)\n\nui(b; vi) = xi(b)\n\nvi \u2212 pi(b)\nxi(b)\n\nThis can be viewed as the same form of utility if the auction was a pay-your-bid auction and the player\nsubmitted a bid of pi(b)\nxi(b). We refer to this term as the price-per-unit and denote it ppu(b) = pi(b)\nxi(b).\nOur analysis will be based on the price-per-unit allocation rule \u02dcx(\u00b7), which determines the expected\nallocation of a player as a function of his price-per-unit. Given this notation, we can re-write\nthe utility that an agent achieves if he submits a bid that corresponds to a price-per-unit of z as:\n\u02dcui(z; vi) = \u02dcx(z)(vi \u2212 z). The latter is exactly the form of a pay-your-bid auction.\nOur upper bound on the DPOA, will be based on the inverse of the PPU allocation rule; let \u03c4i(z) =\n\u02dcx\u22121\n(z) be the price-per-unit of the cheapest bid that achieves allocation at least z. More formally,\n\ni\n\n5\n\n\f\u03c4i(z) = minb|xi(b)\u2265z{ppu(b)}. For simplicity, we assume that any allocation z \u2208 [0, 1] is achieveable\nby some high enough bid b.2 Given this we can de\ufb01ne the threshold for an allocation:\nDe\ufb01nition 2 (Average Threshold). The average threshold for agent i is\n\n(cid:90) 1\n\nTi =\n\n\u03c4i(z) dz\n\n(12)\n\n0\n\nIn Figures 3 and 2 we provide a pictorial representation of these quantities. Connecting with the\nprevious section, for a \ufb01rst price auction, the price-per-unit function is ppu(b) = b, the price-per-unit\nallocation function is \u02dcxi(b) = G\u2212i(b) and the threshold function is \u03c4i(z) = G\u22121\u2212i (z). The average\n0 1 \u2212 G\u2212i(b)db, i.e. the expected maximum other bid.\n\nthreshold Ti is equal to(cid:82) 1\n\n0 G\u22121\u2212i (z)dz =(cid:82) \u221e\n\n1\n\n\u22121\n\u02dcxi(ppu) = \u03c4\ni\n\n(ppu)\n\n]\nn\no\ni\nt\na\nc\no\nl\nl\n\nA\n[\nE\n\nui(b)\n\nppu(b)\n\nPPU\n\nvi\n\nFigure 2: For any bid b with PPC ppu(b), the area\nof a rectangle between (ppu(b), \u02dcxi(ppu(b))) and\n(vi, 0) on the bid allocation rule is the expected\nutility ui(b). The BNE action b\u2217 is chosen to max-\nimize this area.\n\n1\n\n]\nn\no\ni\nt\na\nc\no\nl\nl\n\nA\n[\nE\n\nTi\n\n\u02dcx(ppu)\n\nPPU\n\nFigure 3: The average threshold is the area to the\nleft of the price-per-unit allocation rule, integrate\nfrom 0 to 1.\n\nWe now give our main theorem, which is a distribution-dependent bound on DPOA, that is easy\nto compute give D and which can be easily estimated from samples of D. This theorem is a\ngeneralization of Lemma 1 in the previous section.\nTheorem 2 (Distributional Price of Anarchy Bound). For any auction A in a single dimensional\nsetting and for any bid distribution D, the distributional price of anarchy is bounded by DPOA(D) \u2264\n\n1\u2212e\u2212\u00b5(D) , where \u00b5(D) = maxx\u2208X(cid:80)n\n\n\u00b5(D)\n\nREV(D)\n\ni=1 Ti\u00b7xi\n\n.\n\nT = maxx\u2208X(cid:80)n\n\ni=1 Ti \u00b7 xi,\n\nTheorem 2 provides our main method for bounding the distributional price of anarchy. All we need is\nto compute the revenue REV of the auction and the quantity:\n\n(13)\nunder the given bid distribution D. Both of these are uniquely de\ufb01ned quantities if we are given\nD. Moreover, once we compute Ti, the optimization problem in Equation (13) is simply a welfare\nmaximization problem, where each player\u2019s value per-unit of the good is Ti. Thus, the latter can be\nsolved in polynomial time, whenever the welfare maximization problem over the feasible set X is\npolynomial-time solvable.\nTheorem 2 can be viewed as a bid distribution-dependent analogue of the revenue covering framework\n[4] and of the smooth mechanism framework [13]. In particular, the quantity \u00b5(D) is the data-\ndepenent analogue of the worst-case \u00b5 quantity used in the de\ufb01nition of \u00b5-revenue covering in [4]\nand is roughly related to the \u00b5 quantity used in the de\ufb01nition of a (\u03bb, \u00b5)-smooth mechanism in [13].\n\n5 Distributional Price of Anarchy Bound from Samples\nIn the last section, we assumed we were given distribution D and hence we could compute the\nREV , which gave an upper bound on the DPOA. We now show how we can estimate this\nquantity \u00b5 = T\n2The theory can be easily extended to allow for different maximum achievable allocations by each player, by\n\nsimply integrating the average threshold only up until the largest such allocation.\n\n6\n\n\fquantity \u00b5 when given access to i.i.d. samples b1:T from the bid distribution D. We will separately\nestimate T and REV. The latter is simple expectation and thereby can be easily estimated by an\naverage at\nrates. For the former we \ufb01rst need to estimate Ti for each player i, which requires\nestimation of the allocation and payment functions xi(\u00b7;D) and pi(\u00b7;D).\nSince both of these functions are expected values over the equilibrium bids of opponents, we will\napproximate them by their empirical analogues:\n\n1\u221a\nT\n\n(cid:98)xi(b) =\n\nT(cid:88)\n\nt=1\n\n1\nT\n\n(cid:98)pi(b) =\n\nT(cid:88)\n\nt=1\n\n1\nT\n\nXi(b, bt\u2212i)\n\nPi(b, bt\u2212i).\n\n(14)\n\nTo bound the estimation error of the quantities \u02c6Ti produced by using the latter empirical estimates of\nthe allocation and payment function, we need to provide a uniform convergence property for the error\nof these functions over the bid b.\nSince b takes values in a continuous interval, we cannot simply apply a union bound. We need\nto make assumptions on the structure of the class of functions FXi = {Xi(b,\u00b7) : b \u2208 B} and\nFPi = {Pi(b,\u00b7) : b \u2208 B}, so as uniformly bound their estimation error. For this we resort to the\ntechnology of Rademacher complexity. For a generic class of functions F and a sequence of random\nvariables Z 1:T , the Rademacher complexity is de\ufb01ned as:\n\n(cid:34)\n\nT(cid:88)\n\nt=1\n\nRT (F, Z 1:T ) = E\n\n\u03c31:T\n\n1\nT\n\nsup\nf\u2208F\n\n(cid:35)\n\n\u03c3tf (Z t)\n\n.\n\n(15)\n\nwhere each \u03c3t \u2208 {\u00b11/2} is an i.i.d. Rademacher random variable, which takes each of those values\nwith equal probabilities. The following well known theorem will be useful in our derivations:\nTheorem 3 ([12]). Suppose that for any sample Z 1:T of size T , RT (F, Z 1:T ) \u2264 RT and suppose\nthat functions in F take values in [0, H]. Then with probability 1 \u2212 \u03b4:\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1\n\nT\n\nT(cid:88)\n\nt=1\n\nsup\nf\u2208F\n\n(cid:114)\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) \u2264 2RT + H\n\nf (Zt) \u2212 E[f (Z)]\n\n2 log(4/\u03b4)\n\nT\n\n(16)\n\nThis Theorem reduces our uniform error problem to bounding the Rademacher complexity of classes\nFXi and FPi, since we immediately have the following corollary (where we also use that the\nallocation functions lie in [0, 1] and the payment functions lie in [0, H]):\n|(cid:98)pi(b)\u2212pi(b)|\nCorollary 4. Suppose that for any sample b1:T of size T , the Rademacher complexity of classes FXi\nand FPi is at most RT . Then with probability 1\u2212\u03b4/2, both sup\nb\u2208B\n\n|(cid:98)xi(b)\u2212xi(b)| and sup\n\nare at most 2RT + H(cid:112)2 log(4/\u03b4) / T .\n\nb\u2208B\n\nWe now provide conditions under which the Rademacher complexity of these classes is \u02dcO(1/\nT ).\nLemma 5. Suppose that B = [0, B] and for each bidder i and each bi \u2208 B, the functions Xi(bi,\u00b7) :\n[0, B]n\u22121 (cid:55)\u2192 [0, 1] and Pi(b,\u00b7) : [0, B]n\u22121 (cid:55)\u2192 [0, H] can be computed as \ufb01nite superposition of (i)\ncoordinate-wise multiplication of bid vectors b\u2212i with constants; (ii) comparison indicators 1{\u00b7 > \u00b7}\nof coordinates or constants; (iii) pairwise addition \u00b7 +\u00b7 of coordinates or constants. The Rademacher\ncomplexity for both classes on a sample of size T is O\n\n(cid:16)(cid:112)log(T ) / T\n\n(cid:17)\n\n.\n\n\u221a\n\nThe proof of this Lemma follows by standard arguments of Rademacher calculus, together with VC\narguments on the class of pairwise comparisons. Those arguments can be found in [5, Lemma 9.9]\nand [9, Lemma 11.6.28]. Thereby, we omit its proof. The assumptions of Lemma 5 can be directly\nveri\ufb01ed, for instance, for the sponsored search auctions where the constants that multiply each bid\ncorrespond to quality factors of the bidders, e.g. as in [2] and [14] and then the allocation and the\npayment is a function of the rank of the weighted bid of a player. In that case the price and the\nallocation rule are determined solely by the ranks and the values of the score-weighted bids \u03b3ibi, as\nwell as the position speci\ufb01c quality factors \u03b1j, for each position j in the auction.\nNext we turn to the analysis of the estimation errors on quantities Ti. We consider the following plug-\n\nin estimator for Ti: We consider the empirical analog of function \u03c4i(\u00b7) by(cid:98)\u03c4i(z) =\n\n(cid:98)pi(b)(cid:98)xi(b).\n\nb\u2208[0,B], (cid:98)xi(b)\u2265z\n\ninf\n\n7\n\n\fThen the empirical analog of Ti is obtained by:\n\n1(cid:90)\n\n(cid:98)Ti =\n\n(cid:98)\u03c4i(z) dz.\n\n(17)\n\n0\n\nTo bound the estimation error of (cid:98)Ti, we need to impose an additional condition that ensures that any\n\nnon-zero allocation requires the payment from the bidder at least proportional to that allocation.\nAssumption 6. We assume that pi(x\u22121\ncase interim individually rational, i.e. pi(b) \u2264 H \u00b7 xi(b).\n\u221a\nUnder this assumption we can establish that \u02dcO(\n\nT ) rates of convergence of (cid:98)Ti to Ti and of the\n\n(\u00b7)) is Lipschitz-continuous and that the mechanism is worst-\n\nempirical analog \u02c6T = maxx\u2208X(cid:80)n\nanalog (cid:100)REV of the revenue to REV. Thus the quantity \u02c6\u00b5 = \u02c6T(cid:100)REV\n\n\u02c6Ti \u00b7 xi of the optimized threshold to T as well as the empirical\nREV at\n\n, will also converge to \u00b5 = T\n\nthat rate. This implies the following \ufb01nal conclusion of this section.\nTheorem 7. Under Assumption 6 and the premises of Lemma 5, with probability 1 \u2212 \u03b4:\n\ni=1\n\ni\n\n(cid:114)\n\n(cid:33)\n\nOPT(F)\n\nWELFARE(\u03c3; F)\n\n\u2264\n\nn max{L, H}\n\nH log(n/\u03b4)\n\nT\n\n(18)\n\n(cid:32)\n\n(cid:98)\u00b5\n1 \u2212 e\u2212(cid:98)\u00b5 + \u02dcO\n\n6 Sponsored Search Auction: Model, Methodology and Data Analysis\n\n(cid:80)\n\nWe consider a position auction setting where k ordered positions are assigned to n bidders. An\noutcome m in a position auction is an allocation of positions to bidders. m(j) denotes the bidder who\nis allocated position j; m\u22121(i) refers to the position assigned to bidder i. When bidder i is assigned to\nslot j, the probability of click ci,j is the product of the click-through-rate of the slot \u03b1j and the quality\nscore of the bidder, \u03b3i, so ci,j = \u03b1j\u03b3i (in the data the quality scores for each bidder are varying across\ndifferent auctions and we used the average score as a proxy for the score of a bidder). Each advertiser\nhas a value-per-click (VPC) vi, which is not observed in the data and which we assume is drawn from\nsome distribution Fi. Our benchmark for welfare will be the welfare of the auction that chooses a\nfeasible allocation to maximize the welfare generated, thus OPT = Ev[maxm\nWe consider data generated by advertisers repeatedly participating in a sponsored search auction. The\nmechanism that is being repeated at each stage is an instance of a generalized second price auction\ntriggered by a search query. The rules of each auction are as follows: Each advertiser i is associated\nwith a click probability \u03b3i and a scoring coef\ufb01cient si and is asked to submit a bid-per-click bi.\nAdvertisers are ranked by their rank-score qi = si \u00b7 bi and allocated positions in decreasing order\nof rank-score as long as they pass a rank-score reserve r. All the mentioned sets of parameters\n\u03b8 = (s, \u03b1, \u03b3, r) and the bids b are observable in the data.\nWe will denote with \u03c0b,\u03b8(j) the bidder allocated in slot j under a bid pro\ufb01le b and parameter pro\ufb01le\n\u03b8. We denote with \u03c0\u22121\nb,\u03b8(i) the slot allocated to bidder i. If advertiser i is allocated position j,\nthen he pays only when he is clicked and his payment, i.e. his cost-per-click is the minimal bid\nhe had to place to keep his position, which is: cpcij(b; \u03b8) =\nthis setting to our general model, the allocation function of the auction is Xi(b) = \u03b1\u03c0\nb,\u03b8(i) \u00b7 \u03b3 \u00b7 cpci\u03c0\nthe payment function is Pi(b) = \u03b1\u03c0\nUi(b; vi) = \u03b1\u03c0\nb,\u03b8(i)(b; \u03b8)\n\u22121\n\n. Mapping\nb,\u03b8(i) \u00b7 \u03b3,\nb,\u03b8(i)(b; \u03b8) and the utility function is:\n\u22121\n\nb,\u03b8(i) \u00b7 \u03b3i \u00b7(cid:16)\n\ns\u03c0b,\u03b8 (j+1)\u00b7b\u03c0b,\u03b8 (j+1),r\n\ni \u03b3i\u03b1m\u22121(i)vi].\n\nvi \u2212 cpci\u03c0\n\n(cid:17)\n\n(cid:110)\n\n(cid:111)\n\nmax\n\n\u22121\n\n\u22121\n\n\u22121\n\nsi\n\n.\n\nData Analysis We applied our analysis to the BingAds sponsored search auction system. We\nanalyzed eleven phrases from multiple thematic categories. For each phrase we retrieved data of\nauctions for the phrase for the period of a week. For each phrase and bidder that participated in the\nauctions for the phrase we computed the allocation curve by simulating the auctions for the week\nunder any alternative bid an advertiser could submit (bids are multiples of cents).\nSee Figure 4 for the price-per-unit allocation curves \u02dcxi(\u00b7) = \u03c4\u22121\n(\u00b7) for a subset of the advertisers\nfor a speci\ufb01c search phrase. We estimated the average threshold \u02c6Ti for each bidder by numerically\n\ni\n\n8\n\n\f\u02c6\u00b5 = \u02c6T(cid:100)REV\n\n.511\n.509\n2.966\n1.556\n.386\n.488\n.459\n.419\n.441\n.377\n.502\n\nphrase1\nphrase2\nphrase3\nphrase4\nphrase5\nphrase6\nphrase7\nphrase8\nphrase9\nphrase10\nphrase11\n\nDPOA = 1\u2212e\u2212 \u02c6\u00b5\n\n\u02c6\u00b5\n\n1\n\n.783\n.784\n.320\n.507\n.829\n.791\n.802\n.817\n.809\n.833\n.786\n\nFigure 4: (left) Examples of price-per-unit allocation curves for a subset of six advertisers for a speci\ufb01c keyword\nduring the period of a week. All axes are normalized to 1 for privacy reasons. (right) Distributional Price of\nAnarchy analysis for a set of eleven search phrases on the BingAds system.\n\ni\n\n(cid:80)\n\nSection 3 for each of the search phrases, computing the quantity \u02c6T = maxx\u2208X(cid:80)\nintegrating these allocation curves along the y axis. We then applied the approach described in\n\u02c6Ti \u00b7 xi =\n\u02c6Ti \u00b7 \u03b3i \u00b7 \u03b1m\u22121(i). The latter optimization is simply the optimal assignment problem\nmaxm(\u00b7)\nin decreasing order of \u02c6Ti. We then estimate the expected revenue by the empirical revenue (cid:100)REV.\nwhere each player\u2019s value-per-click is \u02c6Ti and can be performed by greedily assigning players to slots\nWe portray our results on the estimate \u02c6\u00b5 = \u02c6T(cid:100)REV\n\nand the implied bound on the distributional price\nof anarchy for each of the eleven search phrases in Table 4. Phrases are grouped based on thematic\ncategory. Even though the worst-case price of anarchy of this auction is unbounded (since scores\nsi are not equal to qualities \u03b3i, which is required in worst-case POA proofs [1]), we observe that\nempirically the price of anarchy is very good and on average the guarantee is approximately 80% of\nthe optimal. Even if si = \u03b3i the worst-case bound on the POA implies guarantees of approx. 34%\n[1], while the DPOA we estimated implies signi\ufb01cantly higher percentages, portraying the value of\nthe empirical approach we propose.\n\ni\u2208[n]\n\nReferences\n[1] Ioannis Caragiannis, Christos Kaklamanis, Maria Kyropoulou, Brendan Lucier, Renato Paes\nLeme, and \u00c9va Tardos. Bounding the inef\ufb01ciency of outcomes in generalized second price\nauctions. pages 1\u201345, 2014.\n\n[2] Benjamin Edelman, Michael Ostrovsky, and Michael Schwarz. Internet advertising and the\ngeneralized second-price auction: Selling billions of dollars worth of keywords. The American\neconomic review, 97(1):242\u2013259, 2007.\n\n[3] Emmanuel Guerre, Isabelle Perrigne, and Quang Vuong. Optimal nonparametric estimation of\n\n\ufb01rst-price auctions. Econometrica, 68(3):525\u2013574, 2000.\n\n[4] Jason Hartline, Darrell Hoy, and Sam Taggart. Price of Anarchy for Auction Revenue. In ACM\nConference on Economics and Computation, pages 693\u2013710, New York, New York, USA, 2014.\nACM Press.\n\n[5] Michael R Kosorok. Introduction to empirical processes and semiparametric inference. Springer\n\nScience & Business Media, 2007.\n\n[6] Elias Koutsoupias and Christos Papadimitriou. Worst-case equilibria. In STACS 99, pages\n\n404\u2013413. Springer, 1999.\n\n[7] Vijay Krishna. Auction Theory. Academic Press, March 2002.\n\n[8] H. J. Paarsch and H. Hong. An Introduction to the Structural Econometrics of Auction Data.\n\nMIT Press, 2006.\n\n9\n\n\f[9] D. Pollard. Convergence of Stochastic Processes. Springer-Verlag, 1984.\n\n[10] Tim Roughgarden, Vasilis Syrgkanis, and \u00c9va Tardos. The price of anarchy in auctions. CoRR,\n\nabs/1607.07684, 2016.\n\n[11] Tim Roughgarden and Eva Tardos. How bad is sel\ufb01sh routing? J. ACM, 49(2):236\u2013259, March\n\n2002.\n\n[12] S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algo-\nrithms. Understanding Machine Learning: From Theory to Algorithms. Cambridge University\nPress, 2014.\n\n[13] Vasilis Syrgkanis and Eva Tardos. Composable and ef\ufb01cient mechanisms. In ACM Symposium\n\non Theory of Computing, pages 211\u2013220, 2013.\n\n[14] Hal R Varian. Online ad auctions. The American Economic Review, pages 430\u2013434, 2009.\n\n[15] William Vickrey. Counterspeculation, auctions, and competitive sealed tenders. The Journal of\n\nFinance, 16(1):8\u201337, 1961.\n\n10\n\n\f", "award": [], "sourceid": 2088, "authors": [{"given_name": "Darrell", "family_name": "Hoy", "institution": "Tremor Technologies"}, {"given_name": "Denis", "family_name": "Nekipelov", "institution": "University of Virginia"}, {"given_name": "Vasilis", "family_name": "Syrgkanis", "institution": "Microsoft Research"}]}