{"title": "Collaborative PAC Learning", "book": "Advances in Neural Information Processing Systems", "page_first": 2392, "page_last": 2401, "abstract": "We introduce a collaborative PAC learning model, in which k players attempt to learn the same underlying concept. We ask how much more information is required to learn an accurate classifier for all players simultaneously. We refer to the ratio between the sample complexity of collaborative PAC learning and its non-collaborative (single-player) counterpart as the overhead. We design learning algorithms with O(ln(k)) and O(ln^2(k)) overhead in the personalized and centralized variants our model. This gives an exponential improvement upon the naive algorithm that does not share information among players. We complement our upper bounds with an Omega(ln(k)) overhead lower bound, showing that our results are tight up to a logarithmic factor.", "full_text": "Collaborative PAC Learning\n\nToyota Technological Institute at Chicago\n\nAvrim Blum\n\nChicago, IL 60637\navrim@ttic.edu\n\nNika Haghtalab\n\nComputer Science Department\nCarnegie Mellon University\n\nPittsburgh, PA 15213\n\nnhaghtal@cs.cmu.edu\n\nAriel D. Procaccia\n\nComputer Science Department\nCarnegie Mellon University\n\nPittsburgh, PA 15213\n\narielpro@cs.cmu.edu\n\nInstitute for Interdisciplinary Information Sciences\n\nMingda Qiao\n\nTsinghua University\nBeijing, China 100084\n\nqmd14@mails.tsinghua.edu.cn\n\nAbstract\n\nWe consider a collaborative PAC learning model, in which k players attempt to\nlearn the same underlying concept. We ask how much more information is re-\nquired to learn an accurate classi\ufb01er for all players simultaneously. We refer to\nthe ratio between the sample complexity of collaborative PAC learning and its\nnon-collaborative (single-player) counterpart as the overhead. We design learning\nalgorithms with O(ln(k)) and O(ln2(k)) overhead in the personalized and central-\nized variants our model. This gives an exponential improvement upon the na\u00efve\nalgorithm that does not share information among players. We complement our\nupper bounds with an \u2126(ln(k)) overhead lower bound, showing that our results are\ntight up to a logarithmic factor.\n\nIntroduction\n\n1\nAccording to Wikipedia, collaborative learning is a \u201csituation in which two or more people learn ...\nsomething together,\u201d e.g., by \u201ccapitalizing on one another\u2019s resources\u201d and \u201casking one another for\ninformation.\u201d Indeed, it seems self-evident that collaboration, and the sharing of information, can\nmake learning more ef\ufb01cient. Our goal is to formalize this intuition and study its implications.\nAs an example, suppose k branches of a department store, which have sales data for different items in\ndifferent locations, wish to collaborate on learning which items should be sold at each location. In\nthis case, we would like to use the sales information across different branches to learn a good policy\nfor each branch. Another example is given by k hospitals with different patient demographics, e.g.,\nin terms of racial or socio-economic factors, which want to predict occurrence of a disease in patients.\nIn addition to requiring a classi\ufb01er that performs well on the population served by each hospital, it is\nnatural to assume that all hospitals deploy a common classi\ufb01er.\nMotivated by these examples, we consider a model of collaborative PAC learning, in which k players\nattempt to learn the same underlying concept. We then ask how much information is needed for all\nplayers to simultaneously succeed in learning a desirable classi\ufb01er. Speci\ufb01cally, we focus on the\nclassic probably approximately correct (PAC) setting of Valiant [14], where there is an unknown target\nfunction f\u2217 \u2208 F. We consider k players with distributions D1, . . . , Dk that are labeled according to\nf\u2217. Our goal is to learn f\u2217 up to an error of \u0001 on each and every player distribution while requiring\nonly a small number of samples overall.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fA natural but na\u00efve algorithm that forgoes collaboration between players can achieve our objective\nby taking, from each player distribution, a number of samples that is suf\ufb01cient for learning the\nindividual task, and then training a classi\ufb01er over all samples. Such an algorithm uses k times as\nmany samples as needed for learning an individual task \u2014 we say that this algorithm incurs O(k)\noverhead in sample complexity. By contrast, we are interested in algorithms that take advantage\nof the collaborative environment, learn k tasks by sharing information, and incur o(k) overhead in\nsample complexity.\nWe study two variants of the aforementioned model: personalized and centralized. In the personalized\nsetting (as in the department store example), we allow the learning algorithm to return different\nfunctions for different players. That is, our goal is to return classi\ufb01ers f1, . . . , fk that have error of at\nmost \u0001 on player distributions D1, . . . , Dk, respectively. In the centralized setting (as in the hospital\nexample), the learning algorithm is required to return a single classi\ufb01er f that has an error of at most\n\u0001 on all player distributions D1, . . . , Dk. Our results provide upper and lower bounds on the sample\ncomplexity overhead required for learning in both settings.\n1.1 Overview of Results\nIn Section 3, we provide algorithms for personalized and centralized collaborative learning that obtain\nexponential improvements over the sample complexity of the na\u00efve approach. In Theorem 3.1, we\nintroduce an algorithm for the personalized setting that has O(ln(k)) overhead in sample complexity.\nFor the centralized setting, in Theorem 3.2, we develop an algorithm that has O(ln2(k)) overhead in\nsample complexity. At a high level, the latter algorithm \ufb01rst learns a series of functions on adaptively\nchosen mixtures of player distributions. These mixtures are chosen such that for any player a large\nmajority of the functions perform well. This allows us to combine all functions into one classi\ufb01er\nthat performs well on every player distribution. Our algorithm is an improper learning algorithm, as\nthe combination of these functions may not belong to F.\nIn Section 4, we present lower bounds on the sample complexity of collaborative PAC learning for\nthe personalized and centralized variants. In particular, in Theorem 4.1 we show that any algorithm\nthat learns in the collaborative setting requires \u2126(ln(k)) overhead in sample complexity. This shows\nthat our upper bound for the personalized setting, as stated in Theorem 3.1, is tight. Furthermore, in\nTheorem 4.5, we show that obtaining uniform convergence across F over all k player distributions\nrequires \u2126(k) overhead in sample complexity. Interestingly, our centralized algorithm (Theorem 3.2)\nbypasses this lower bound by using arguments that do not depend on uniform convergence. Indeed,\nthis can be seen from the fact that it is an improper learning algorithm.\nIn Appendix D, we discuss the extension of our results to the non-realizable setting. Speci\ufb01cally, we\nconsider a setting where there is a \u201cgood\u201d but not \u201cperfect\u201d target function f\u2217 \u2208 F that has a small\nerror with respect to every player distribution, and prove that our upper bounds carry over.\n1.2 Related Work\nRelated work in computational and statistical learning has examined some aspects of the general\nproblem of learning multiple related tasks simultaneously. Below we discuss papers on multi-task\nlearning [4, 3, 7, 5, 10, 13], domain adaptation [11, 12, 6], and distributed learning [2, 8, 15], which\nare most closely related.\nMulti-task learning considers the problem of learning multiple tasks in series or in parallel. In this\nspace, Baxter [4] studied the problem of model selection for learning multiple related tasks. In their\nwork, each learning task is itself randomly drawn from a distribution over related tasks, and the\nlearner\u2019s goal is to \ufb01nd a hypothesis space that is appropriate for learning all tasks. Ben-David and\nSchuller [5] also studied the sample complexity of learning multiple related tasks. However, in their\nwork similarity between two tasks is represented by existence of \u201ctransfer\u201d functions though which\nunderlying distributions are related.\nMansour et al. [11, 12] consider a multi-source domain adaptation problem, where the learner is given\nk distributions and k corresponding predictors that have error at most \u0001 on individual distributions.\nThe goal of the learner is to combine these predictors to obtain error of k\u0001 on any unknown mixture of\nplayer distributions. Our work is incomparable to this line of work, as our goal is to learn classi\ufb01ers,\nrather than combining existing ones, and our benchmark is to obtain error \u0001 on each individual\ndistribution. Indeed, in our setting one can learn a hypothesis that has error k\u0001 on any mixture of\nplayers with no overhead in sample complexity.\n\n2\n\n\fwell-known that sampling a set S of size m\u0001,\u03b4 = O(cid:0) 1\n\nDistributed learning [2, 8, 15] also considers the problem of learning from k different distributions\nsimultaneously. However, the main objective in this space is to learn with limited communication\nbetween the players, rather than with low sample complexity.\n2 Model\nLet X be an instance space and Y = {0, 1} be the set of labels. A hypothesis is a function\nf : X \u2192 Y that maps any instance x \u2208 X to a label y \u2208 Y. We consider a hypothesis class F\nwith VC dimension d. Given a distribution D over X \u00d7 Y, the error of a hypothesis f is de\ufb01ned as\nerrD(f ) = Pr(x,y)\u223cD [f (x) (cid:54)= y].\nIn the collaborative learning setting, we consider k players with distributions D1, . . . , Dk over\nX \u00d7 Y. We focus on the realizable setting, where all players\u2019 distributions are labeled according to\na common target function f\u2217 \u2208 F, i.e., errDi(f\u2217) = 0 for all i \u2208 [k] (but see Appendix D for an\nextension to the non-realizable setting). We represent an instance of the collaborative PAC learning\nsetting with the 3-tuple (F, f\u2217,{D}i\u2208[k]).\nOur goal is to learn a good classi\ufb01er with respect to every player distribution. We call this (\u0001, \u03b4)-\nlearning in the collaborative PAC setting, and study two variants: the personalized setting, and the\ncentralized setting. In the personalized setting, our goal is to learn functions f1, . . . , fk, such that\nwith probability 1 \u2212 \u03b4, errDi(fi) \u2264 \u0001 for all i \u2208 [k]. In the centralized setting, we require all the\noutput functions to be identical. Put another way, our goal is to return a single f, such that with\nprobability 1 \u2212 \u03b4, errDi(f ) \u2264 \u0001 for all i \u2208 [k]. In both settings, we allow our algorithm to be\nimproper, that is, the learned functions need not belong to F.\nWe compare the sample complexity of our algorithms to their PAC counterparts in the realizable\nsetting. In the traditional realizable PAC setting, m\u0001,\u03b4 denotes the number of samples needed for\n(\u0001, \u03b4)-learning F. That is, m\u0001,\u03b4 is the total number of samples drawn from a realizable distribution D,\nsuch that, with probability 1 \u2212 \u03b4, any classi\ufb01er f \u2208 F that is consistent with the sample set satis\ufb01es\nerrD(f ) \u2264 \u0001. We denote by OF (\u00b7) the function that, for any set S of labeled samples, returns a\nfunction f \u2208 F that is consistent with S if such a function exists (and outputs \u201cnone\u201d otherwise). It is\nsuf\ufb01cient for (\u0001, \u03b4)-learning a hypothesis class F of VC dimension d [1]. We refer to the ratio of the\nsample complexity of an algorithm in the collaborative PAC setting to that of the (non-collaborative)\nrealizable PAC setting as the overhead. For ease of exposition, we only consider the dependence of\nthe overhead on parameters k, d, and \u0001.\n3 Sample Complexity Upper Bounds\nIn this section, we prove upper bounds on the sample complexity of (\u0001, \u03b4)-learning in the collaborative\nPAC setting. We begin by providing a simple algorithm with O(ln(k)) overhead (in terms of sample\ncomplexity, see Section 2) for the personalized setting. We then design and analyze an algorithm for\nthe centralized setting with O(ln2(k)) overhead, following a discussion of additional challenges that\narise in this setting.\n3.1 Personalized Setting\nThe idea underlying the algorithm for the personalized setting is quite intuitive: If we were to learn a\nclassi\ufb01er that is on average good for the players, then we have learned a classi\ufb01er that is good for a\nlarge fraction of the players. Therefore, a large fraction of the players can be simultaneously satis\ufb01ed\nby a single good global classi\ufb01er. This process can be repeated until each player receives a good\nclassi\ufb01er.\n(cid:80)\nIn more detail, let us consider an algorithm that pools together a sample set of total size m\u0001/4,\u03b4 from\ni\u2208[k] Di over individual player distributions, and \ufb01nds f \u2208 F that\nthe uniform mixture D = 1\nk\nis consistent with this set. Clearly, with probability 1 \u2212 \u03b4, f has a small error of \u0001/4 with respect\nto distribution D. However, we would like to understand how well f performs on each individual\nplayer\u2019s distribution.\nSince errD(f ) \u2264 \u0001/4 is also the average error of f on player distributions, with probability 1 \u2212 \u03b4, f\nmust have error of at most \u0001/2 on at least half of the players. Indeed, one can identify such players\nby taking additional \u02dcO( 1\n\u0001 ) samples from each player and asking whether the empirical error of f\non these sample sets is at most 3\u0001/4. Using a variant of the VC theorem, it is not hard to see that\n\n(cid:1)(cid:1)(cid:1), and applying OF (S), is\n\n\u0001\n\n(cid:0)d ln(cid:0) 1\n\n(cid:1) + ln(cid:0) 1\n\n\u03b4\n\n\u0001\n\n3\n\n\ffor any player i such that errDi(f ) \u2264 \u0001/2, the empirical error of f is at most 3\u0001/4, and no player\nwith empirical error at most 3\u0001/4 has true error that is worst than \u0001. Once players with empirical\nerror 3\u0001/4 are identi\ufb01ed, one can output fi = f for any such player, and repeat the procedure for\nthe remaining players. After log(k) rounds, this process terminates with all players having received\nfunctions with error of at most \u0001 on their respective distributions, with probability 1 \u2212 log(k)\u03b4.\nWe formalize the above discussion via Algorithm 1 and Theorem 3.1. For completeness, a more\nrigorous proof of the theorem is given in Appendix A.\n\nAlgorithm 1 PERSONALIZED LEARNING\nN1 \u2190 [k]; \u03b4(cid:48) \u2190 \u03b4/2 log(k);\nfor r = 1, . . . ,(cid:100)log(k)(cid:101) do\nDi;\n\n(cid:80)\n\ni\u2208Nr\n\n\u02dcDr \u2190 1|Nr|\nLet S be a sample of size m\u0001/4,\u03b4(cid:48) drawn from \u02dcDr, and f (r) \u2190 OF (S);\nLet Gr \u2190 TEST(f (r), Nr, \u0001, \u03b4(cid:48));\nNr+1 \u2190 Nr \\ Gr;\nfor i \u2208 Gr do fi \u2190 f (r);\n\nend\nreturn f1, . . . , fk\nTEST(f, N, \u0001, \u03b4):\nfor i \u2208 N do take sample set Ti of size O\nreturn {i | errTi(f ) \u2264 3\nTheorem 3.1. For any \u0001, \u03b4 > 0, and hypothesis class F of VC dimension d, Algorithm 1 (\u0001, \u03b4)-learns\nF in the personalized collaborative PAC setting using m samples, where\n\n(cid:16)|N|\n\nfrom Di ;\n\n(cid:17)(cid:17)\n\n(cid:16) 1\n\n4 \u0001}\n\n\u0001 ln\n\n\u0001\u03b4\n\nm = O\n\n(d + k) ln\n\n+ k ln\n\n.\n\n(cid:18) ln(k)\n\n(cid:18)\n\n\u0001\n\n(cid:18) 1\n\n(cid:19)\n\n\u0001\n\n(cid:18) k\n\n(cid:19)(cid:19)(cid:19)\n\n\u03b4\n\nNote that Algorithm 1 has O(ln(k)) overhead when k = O(d).\n3.2 Centralized Setting\n\nWe next present a learning algorithm with O(ln2(k)) overhead in the centralized setting. Recall that\nour goal is to learn a single function f that has an error of \u0001 on every player distribution, as opposed\nto the personalized setting where players can receive different functions.\nA natural \ufb01rst attempt at learning in the centralized setting is to combine the classi\ufb01ers f1, . . . , fk\nthat we learned in the personalized setting (Algorithm 1), say, through a weighted majority vote.\nOne challenge with this approach is that, in general, it is possible that many of the functions fj\nperform poorly on the distribution of a different player i. The reason is that when Algorithm 1 \ufb01nds a\nsuitable f (r) for players in Gr, it completely removes them from consideration for future rounds;\nsubsequent functions may perform poorly with respect to the distributions associated with those\nplayers. Therefore, this approach may lead to a global classi\ufb01er with large error on some player\ndistributions.\nTo overcome this problem, we instead design an algorithm that continues to take additional samples\nfrom players for whom we have already found suitable classi\ufb01ers. The key idea behind the centralized\nlearning algorithm is to group the players at every round based on how many functions learned so\nfar have large error rates on those players\u2019 distributions, and to learn from data sampled from all\nthe groups simultaneously. This ensures that the function learned in each round performs well on a\nlarge fraction of the players in each group, thereby reducing the likelihood that in later stages of this\nprocess a player appears in a group for which a large fraction of the functions perform poorly.\nIn more detail, our algorithm learns t = \u0398(ln(k)) classi\ufb01ers f (1), f (2), . . . , f (t), such that for any\nplayer i \u2208 [k], at least 0.6t functions among them achieve an error below \u0001(cid:48) = \u0001/6 on Di. The\nalgorithm then returns the classi\ufb01er maj({f (r)}t\nr=1), where, for a set of hypotheses F , maj(F )\ndenotes the classi\ufb01er that, given x \u2208 X , returns the label that the majority of hypotheses in F assign\nto x. Note that any instance that is mislabeled by this classi\ufb01er must be mislabeled by at least 0.1t\n\n4\n\n\fi\n\nr=1)\n\nfunctions among the 0.6t good functions, i.e., 1/6 of the good functions. Hence, maj({f (r)}t\nhas an error of at most 6\u0001(cid:48) = \u0001 on each distribution Di.\nfor any round r \u2208 [t] and player i \u2208 [k],\nThroughout the algorithm, we keep track of counters \u03b1(r)\nwhich, roughly speaking, record the number of classi\ufb01ers among f (1), f (2), . . . , f (r) that have an\nerror of more than \u0001(cid:48) on distribution Di. To learn f (r+1), we \ufb01rst group distributions D1, . . . , Dk\nbased on the values of \u03b1(r)\n, draw about m\u0001(cid:48),\u03b4 samples from the mixture of the distributions in each\ngroup, and return a function f (r+1) that is consistent with all of the samples. Similarly to Section 3.1,\none can show that f (r+1) achieves O(\u0001(cid:48)) error with respect to a large fraction of player distributions\nin each group. Consequently, the counters are increased, i.e., \u03b1(r+1)\n, only for a small fraction\ni \u2264 0.4t for any player i \u2208 [k], i.e., on\nof players. Finally, we show that with high probability, \u03b1(t)\neach distribution Di, at least 0.6t functions achieve error of at most \u0001(cid:48).\nThe algorithm is formally described in Algorithm 2. The next theorem states our sample complexity\nupper bound for the centralized setting.\n\n> \u03b1(r)\n\ni\n\ni\n\ni\n\n(cid:109)\n\nt \u2190(cid:108) 5\n\nAlgorithm 2 CENTRALIZED LEARNING\ni \u2190 0 for each i \u2208 [k];\n\u03b1(0)\n\n; \u0001(cid:48) \u2190 \u0001/6;\n\n2 log8/7(k)\n0 \u2190 [k]; N (0)\nN (0)\nfor r = 1, 2, . . . , t do\nfor c = 0, 1, . . . , t \u2212 1 do\n\nc \u2190 \u2205 for each c \u2208 [t];\n\nif N (r\u22121)\n\n(cid:54)= \u2205 then\n\nc\nDraw a sample set S(r)\n\nc\n\n(cid:17)\n\nelse S(r)\n\nc \u2190 \u2205 ;\n\n(cid:16)(cid:83)t\u22121\n\nc=0 S(r)\n\nend\nf (r) \u2190 OF\nGr \u2190 TEST(f (r), [k], \u0001(cid:48), \u03b4/(2t));\ni \u2190 \u03b1(r\u22121)\nfor i = 1, . . . , k do \u03b1(r)\nc \u2190 {i \u2208 [k] : \u03b1(r)\nfor c = 0, . . . , t do N (r)\n\n;\n\nc\n\ni\n\n+ I [i /\u2208 Gr];\ni = c};\n\nend\nreturn maj({f (r)}t\n\nr=1);\n\nof size m\u0001(cid:48)/16,\u03b4/(2t2) from (cid:101)D(r\u22121)\n\nc\n\n=\n\n1\n\n|N (r\u22121)\n\nc\n\n|\n\n(cid:80)\n\ni\u2208N (r\u22121)\n\nc\n\nDi;\n\nTheorem 3.2. For any \u0001, \u03b4 > 0, and hypothesis class F of VC dimension d, Algorithm 2 (\u0001, \u03b4)-learns\nF in the centralized collaborative PAC setting using m samples, where\n\n(cid:18) ln2(k)\n\n(cid:18)\n\n\u0001\n\n(cid:18) 1\n\n(cid:19)\n\n\u0001\n\n(cid:18) 1\n\n(cid:19)(cid:19)(cid:19)\n\n\u03b4\n\nm = O\n\n(d + k) ln\n\n+ k ln\n\n.\n\nIn particular, Algorithm 2 has O(ln2(k)) overhead when k = O(d).\nTurning to the theorem\u2019s proof, note that in Algorithm 2, N (r\u22121)\n\nwhom c out of the r \u2212 1 functions learned so far have a large error, and (cid:101)D(r\u22121)\neach function f (r) has a small error on (cid:101)D(r\u22121)\n(cid:108) 5\n\nrepresents the set of players for\nrepresents the mixture\nof distribution of players in N (r\u22121)\n. Moreover, Gr is the set of players for whom f (r) has a small\nerror. The following lemma, whose proof appears in Appendix B.1, shows that with high probability\nfor all c. Here and in the following, t stands for\n\n(cid:109)\n\nc\n\nc\n\nc\n\nc\n\nas in Algorithm 2.\n\n2 log8/7(k)\nLemma 3.3. With probability 1 \u2212 \u03b4, the following two properties hold for all r \u2208 [t]:\n1. For any c \u2208 {0, . . . , t \u2212 1} such that N (r\u22121)\n2. For any i \u2208 Gr, errDi(f (r)) \u2264 \u0001(cid:48), and for any i /\u2208 Gr, errDi(f (r)) > \u0001(cid:48)/2.\n\nis non-empty, err(cid:101)D(r\u22121)\n\n(f (r)) \u2264 \u0001(cid:48)/16.\n\nc\n\nc\n\n5\n\n\fThe next lemma gives an upper bound on |N (r)\nlearned functions have a large error.\nLemma 3.4. With probability 1 \u2212 \u03b4, for any r, c \u2208 {0, . . . , t}, we have |N (r)\n\nc\n\nc\n\n| \u2014 the number of players for whom c out of the r\n\n| \u2264(cid:0)r\n\n(cid:1) \u00b7 k\n\nc\n\n8c .\n\nc\n\n| = |{i \u2208 [k] : \u03b1(r)\n\nProof. Let nr,c = |N (r)\ni = c}| be the number of players for whom c functions in\nf (1), . . . , f (r) do not have a small error. We note that n0,0 = k and n0,c = 0 for c \u2208 {1, . . . , t}. The\nnext technical claim, whose proof appears in Appendix B.2, asserts that to prove this lemma, it is\nsuf\ufb01cient to show that for any r \u2208 {1, . . . , t} and c \u2208 {0, . . . , t}, nr,c \u2264 nr\u22121,c + 1\n8 nr\u22121,c\u22121. Here\nwe assume that nr\u22121,\u22121 = 0.\nClaim 3.5. Suppose that n0,0 = k, n0,c = 0 for c \u2208 {1, . . . , t}, and nr,c \u2264 nr\u22121,c + 1\n\nholds for any r \u2208 {1, . . . , t} and c \u2208 {0, . . . , t}. Then for any r, c \u2208 {0, . . . , t}, nr,c \u2264(cid:0)r\n\n8 nr\u22121,c\u22121\n\ni\n\ni\n\nc\n\nc\u22121\n\nnr,c =\n\n\\ Gr\n\n(cid:12)(cid:12)(cid:12) .\n\nc , N (r)\n\n=nr\u22121,c +\n\n, and nr,c, we have\n\nBy de\ufb01nition of \u03b1(r)\n\n= c}(cid:12)(cid:12)(cid:12) +\n\n(cid:12)(cid:12)(cid:12){i \u2208 [k] : \u03b1(r\u22121)\n8 nr\u22121,c\u22121. Recall that (cid:101)D(r\u22121)\n\ni = c}(cid:12)(cid:12)(cid:12) \u2264(cid:12)(cid:12)(cid:12){i \u2208 [k] : \u03b1(r\u22121)\n(cid:12)(cid:12)(cid:12){i \u2208 [k] : \u03b1(r)\n(cid:12)(cid:12)(cid:12)N (r\u22121)\n. By Lemma 3.3, with probability 1\u2212 \u03b4, err(cid:101)D(r\u22121)\n8|N (r\u22121)\nc\u22121\n(cid:12)(cid:12)(cid:12)N (r\u22121)\n: errDi(f (r)) > \u0001(cid:48)/2}(cid:12)(cid:12)(cid:12) \u2264 1\n(cid:12)(cid:12)(cid:12) \u2264(cid:12)(cid:12)(cid:12){i \u2208 N (r\u22121)\n\nIt remains to show that |N (r\u22121)\nc\u22121\ndistributions in N (r\u22121)\nc\u22121\nerrDi (f (r)) < \u0001(cid:48)\n\nis the mixture of all\nc\u22121\n(f (r)) < \u0001(cid:48)/16. Put another\n| players i \u2208 N (r\u22121)\ncan have\nc\u22121\nerrDi(f (r)) > \u0001(cid:48)/2. Moreover, by Lemma 3.3, for any i /\u2208 Gr, we have that errDi(f (r)) > \u0001(cid:48)/2.\n\nway,(cid:80)\nTherefore,(cid:12)(cid:12)(cid:12)N (r\u22121)\n\n|. Thus, at most 1\n\n16 \u00b7 |N (r\u22121)\nc\u22121\n\n\\ Gr| \u2264 1\n\n(cid:12)(cid:12)(cid:12) =\n\ni\u2208N (r\u22121)\n\nc\u22121\n\n\\ Gr\n\nc\u22121\n\nc\u22121\n\nc\u22121\n\n1\n8\n\nnr\u22121,c\u22121.\n\nc\u22121\n\n8\n\nc\n\n8c .\n\n(cid:1) \u00b7 k\n= c \u2212 1 \u2227 i /\u2208 Gr}(cid:12)(cid:12)(cid:12)\n\nThis completes the proof.\n\nWe now prove Theorem 3.2 using Lemma 3.4.\n\nProof of Theorem 3.2. We \ufb01rst show that, with high probability, for any i \u2208 [k], at most 0.4t functions\ni < 0.4t for all i \u2208 [k]. Note that by our\namong f (1), . . . , f (t) have error greater than \u0001(cid:48), i.e., \u03b1(t)\n, we have (8/7)0.4t \u2265 k. By Lemma 3.4 and an upper bound on binomial\nchoice of t =\ncoef\ufb01cients, with probability 1 \u2212 \u03b4, for any integer c \u2208 [0.4t, t],\n\n2 log8/7(k)\n\n(cid:108) 5\n\n(cid:109)\n\n(cid:18)t\n\n(cid:19)\n\nc\n\n(cid:18) et\n\n(cid:19)c \u00b7 k\n\nc\n\n|N (t)\n\nc\n\n| \u2264\n\n\u00b7 k\n8c <\n\n8c <\n\nk\n\n(8/7)c \u2264 1,\n\ni < 0.4t for all i \u2208 [k].\n\nc = \u2205. Therefore, with probability 1 \u2212 \u03b4, \u03b1(t)\n\nwhich implies that N (t)\nNext, we prove that f = maj({f (r)}t\nr=1) has error at most \u0001 on every player distribution. Consider\ndistribution Di of player i. By de\ufb01nition, t \u2212 \u03b1(t)\nfunctions have error at most \u0001(cid:48) on Di. We\nrefer to these functions as \u201cgood\u201d functions. Note that for any instance x that is mislabeled by\nf, at least 0.5t \u2212 \u03b1(t)\ni )\u0001(cid:48) \u2265\n(0.5t \u2212 \u03b1(t)\n\ni good functions must make a wrong prediction. Therefore, (t \u2212 \u03b1(t)\ni < 0.4t for all i \u2208 [k]. Hence,\n\u0001(cid:48) \u2264 \u0001,\n\ni ) \u00b7 errDi(f ). Moreover, with probability 1 \u2212 \u03b4, \u03b1(t)\n\u0001(cid:48) \u2264 0.6t\n0.1t\n\nerrDi(f ) \u2264 t \u2212 \u03b1(t)\n0.5t \u2212 \u03b1(t)\n\ni\n\ni\n\ni\n\nwith probability 1 \u2212 \u03b4. This proves that Algorithm 2 (\u0001, \u03b4)-learns F in the centralized collaborative\nPAC setting.\n\n6\n\n\fFinally, we bound the sample complexity of Algorithm 2. Recall that t = \u0398(ln(k)) and \u0001(cid:48) = \u0001/6. At\neach iteration of Algorithm 2, we draw total of t \u00b7 m\u0001(cid:48)/16,\u03b4/(4t2) samples from t mixtures. Therefore,\nover t time steps, we draw a total of\nt2 \u00b7 m\u0001(cid:48)/16,\u03b4/(4t2) = O\n\n(cid:18) ln2(k)\n\n(cid:18) 1\n\n(cid:18) 1\n\n(cid:19)(cid:19)\n\n+ ln ln(k)\n\n(cid:18)\n\n(cid:19)\n\n(cid:19)\n\n+ ln\n\nd ln\n\nsamples for learning f (1), . . . , f (t). Moreover, the total number samples requested for subroutine\nTEST(f (r), [k], \u0001(cid:48), \u03b4/(4t)) for r = 1 . . . , t is\n\n(cid:18) tk\n\n\u0001\n\nO\n\n(cid:18) k\n\n\u0001\u03b4\n\n\u00b7 ln\n\nk ln\n\n+ k ln\n\n(cid:19)\n\n+\n\nln2(k)\n\n\u0001\n\nk\n\n.\n\n\u0001\n\n\u00b7\n\n(cid:18)\n\n\u00b7\n\n= O\n\n(cid:19)(cid:19)\n(cid:18) ln2(k)\n\n(cid:18) ln(k)\n(cid:18)\n\n\u0001\n\n\u0001\n\n(cid:19)\n\n(cid:18) 1\n(cid:19)\n(cid:18) 1\n\n\u0001\n\n\u0001\n\n\u03b4\n\n(cid:18) 1\n(cid:19)(cid:19)\n(cid:19)(cid:19)(cid:19)\n(cid:18) 1\n\n\u03b4\n\n.\n\n\u03b4\n\nO\n\n\u0001\n\n(d + k) ln\n\n+ k ln\n\nWe conclude that the total sample complexity is\n\nWe remark that Algorithm 2 is inspired by the classic boosting scheme. Indeed, an algorithm that\nis directly adapted from boosting attains a similar performance guarantee as in Theorem 3.2. The\nalgorithm assigns a uniform weight to each player, and learns a classi\ufb01er with O(\u0001) error on the\nmixture distribution. Then, depending on whether the function achieves an O(\u0001) error on each\ndistribution, the algorithm updates the players\u2019 weights, and learns the next classi\ufb01er from the\nweighted mixture of all distributions. An analysis similar to that of AdaBoost [9] shows that the\nmajority vote of all the classi\ufb01ers learned over \u0398(ln(k)) iterations of the above procedure achieves\na small error on every distribution. Similar to Algorithm 2, this algorithm achieves an O(ln2(k))\noverhead for the centralized setting.\n4 Sample Complexity Lower Bounds\nIn this section, we present lower bounds on the sample complexity of collaborative PAC learning. In\nSection 4.1, we show that any learning algorithm for the collaborative PAC setting incurs \u2126(log(k))\noverhead in terms of sample complexity. In Section 4.2, we consider the sample complexity required\nfor obtaining uniform convergence across F in the collaborative PAC setting. We show that \u2126(k)\noverhead is necessary to obtain such results.\n4.1 Tight Lower Bound for the Personalized Setting\nWe now turn to establishing the \u2126(log(k)) lower bound mentioned above. This lower bound implies\nthe tightness of the O(log(k)) overhead upper bound obtained by Theorem 3.1 in the personalized\nsetting. Moreover, the O(log2(k)) overhead obtained by Theorem 3.2 in the centralized setting is\nnearly tight, up to a log(k) multiplicative factor. Formally, we prove the following theorem.\nTheorem 4.1. For any k \u2208 N, \u0001, \u03b4 \u2208 (0, 0.1), and (\u0001, \u03b4)-learning algorithm A in the collaborative\nPAC setting, there exist an instance with k players, and a hypothesis class of VC-dimension k, on\nwhich A requires at least 3k ln[9k/(10\u03b4)]/(20\u0001) samples in expectation.\nHard instance distribution. We show that for any k \u2208 N and \u0001, \u03b4 \u2208 (0, 0.1), there is a distribution\nDk,\u0001 of \u201chard\u201d instances, each with k players and a hypothesis class with VC-dimension k, such that\nany (\u0001, \u03b4)-learning algorithm A requires \u2126(k log(k)/\u0001) samples in expectation on a random instance\ndrawn from the distribution, even in the personalized setting. This directly implies Theorem 4.1,\nsince A must take \u2126(k log(k)/\u0001) samples on some instance in the support of Dk,\u0001. We de\ufb01ne Dk,\u0001\nas follows:\n\u2022 Instance space: Xk = {1, 2, . . . , k,\u22a5}.\n\u2022 Hypothesis class: Fk is the collection of all binary functions on Xk that map \u22a5 to 0.\n\u2022 Target function: f\u2217 is chosen from Fk uniformly at random.\n\u2022 Players\u2019 distributions: The distribution Di of player i is either a degenerate distribution that assigns\nprobability 1 to \u22a5, or a Bernoulli distribution on {i,\u22a5} with Di(i) = 2\u0001 and Di(\u22a5) = 1 \u2212 2\u0001.\nDi is chosen from these two distributions independently and uniformly at random.\n\n7\n\n\fNote that the VC-dimension of Fk is k. Moreover, on any instance in the support of Dk,\u0001, learning\nin the personalized setting is equivalent to learning in the centralized setting. This is due to the fact\nthat given functions f1, f2, . . . , fk for the personalized setting, where fi is the function assigned to\nplayer i, we can combine these functions into a single function f \u2208 Fk for the centralized setting\nby de\ufb01ning f (\u22a5) = 0 and f (i) = fi(i) for all i \u2208 [k]. Then, errDi(f ) \u2264 errDi(fi) for all i \u2208 [k].1\nTherefore, without loss of generality we focus below on the centralized setting.\n\nLower bound for k = 1. As a building block in our proof of Theorem 4.1, we establish a lower\nbound for the special case of k = 1. For brevity, let D\u0001 denote the instance distribution D1,\u0001. We say\nthat A is an (\u0001, \u03b4)-learning algorithm for the instance distribution D\u0001 if and only if on any instance\nin the support of D\u0001, with probability at least 1 \u2212 \u03b4, A outputs a function f with error below \u0001.\nThe following lemma, proved in Appendix C, states that any (\u0001, \u03b4)-learning algorithm for D\u0001 takes\n\u2126(log(1/\u03b4)/\u0001) samples on a random instance drawn from D\u0001.2\nLemma 4.2. For any \u0001, \u03b4 \u2208 (0, 0.1) and (\u0001, \u03b4)-learning algorithm A for D\u0001, A takes at least\nln(1/\u03b4)/(6\u0001) samples in expectation on a random instance drawn from D\u0001. Here the expecta-\ntion is taken over both the randomness in the samples and the randomness in drawing the instance\nfrom D\u0001.\nNow we prove Theorem 4.1 by Lemma 4.2 and a reduction from a random instance sampled from D\u0001\nto instances sampled from Dk,\u0001. Intuitively, a random instance drawn from Dk,\u0001 is equivalent to k\nindependent instances from D\u0001. We show that any learning algorithm A that simultaneously solves k\ntasks (i.e., an instance from Dk,\u0001) with probability 1 \u2212 \u03b4 can be transformed into an algorithm A(cid:48) that\nsolves a single task (i.e., an instance from D\u0001) with probability 1 \u2212 O(\u03b4/k). Moreover, the expected\nsample complexity of A(cid:48) is only an O(1/k) fraction of the complexity of A. This transformation,\ntogether with Lemma 4.2, gives a lower bound on the sample complexity of A.\nProof Sketch of Theorem 4.1. We construct an algorithm A(cid:48) for the instance distribution D\u0001 from an\nalgorithm A that (\u0001, \u03b4)-learns in the centralized setting. Recall that on an instance drawn from D\u0001,\nA(cid:48) has access to a distribution D, i.e., the single player\u2019s distribution.\n\u2022 A(cid:48) generates an instance (Fk, f\u2217,{Di}i\u2208[k]) from the distribution Dk,\u0001 (speci\ufb01cally, A(cid:48) knows\nthe target function f\u2217 and the distributions), and then chooses l \u2208 [k] uniformly at random.\n\u2022 A(cid:48) simulates A on instance (Fk, f\u2217,{Di}i\u2208[k]), with Dl replaced by the distribution D. Specif-\nically, every time A draws a sample from Dj for some j (cid:54)= l, A(cid:48) samples Dj and forwards the\nsample to A. When A asks for a sample from Dl, A(cid:48) samples the distribution D instead and\nreplies to A accordingly, i.e., A(cid:48) returns l, together with the label, if the sample is 1 (recall that\nX1 = {1,\u22a5}), and returns \u22a5 otherwise.\n\u2022 Finally, when A terminates and returns a function f on Xk, A(cid:48) checks whether errDj (f ) < \u0001\nfor every j (cid:54)= l. If so, A(cid:48) returns the function f(cid:48) de\ufb01ned as f(cid:48)(1) = f (l) and f(cid:48)(\u22a5) = f (\u22a5).\nOtherwise, A(cid:48) repeats the simulation process on a new instance drawn from Dk,\u0001.\n\nLet mi be the expected number of samples drawn from the i-th distribution when A runs on an instance\ndrawn from Dk,\u0001. We have the following two claims, whose proofs are relegated to Appendix C.\nClaim 4.3. A(cid:48) is an (\u0001, 10\u03b4/(9k))-learning algorithm for D\u0001.\n\nClaim 4.4. A(cid:48) takes at most 10/(9k)(cid:80)k\nApplying Lemma 4.2 to A(cid:48) gives(cid:80)k\n\ni=1 mi samples in expectation.\n\ni=1 mi \u2265 3k ln[9k/(10\u03b4)]\n\n20\u0001\n\n, which proves Theorem 4.1.\n\n4.2 Lower Bound for Uniform Convergence\nWe next examine the sample complexity required for obtaining uniform convergence across the\nhypothesis class F in the centralized collaborative PAC setting, and establish an overhead lower\nbound of \u2126(k). Interestingly, our centralized learning algorithm (Algorithm 2) achieves O(log2(k))\noverhead \u2014 it circumvents the lower bound by not relying on uniform convergence.\n\n1In fact, when fi \u2208 Fk, errDi (f ) = errDi (fi) for all i \u2208 [k].\n2 Here we only assume that A is correct for instances in the support of D\u0001, rather than being correct on every\n\ninstance.\n\n8\n\n\fTo be more formal, we \ufb01rst need to de\ufb01ne uniform convergence in the cooperative PAC learning\nsetting. We say that a hypothesis class F has the uniform convergence property with sample size\n\u0001,\u03b4 if for any k distributions D1, . . . , Dk, there exist integers m1, . . . , mk that sum up to m(k)\nm(k)\n\u0001,\u03b4 ,\nsuch that when mi samples are drawn from Di for each i \u2208 [k], with probability 1 \u2212 \u03b4, any function\nin F that is consistent with all the m(k)\n\u0001,\u03b4 samples achieves error at most \u0001 on every distribution Di.\nNote that the foregoing de\ufb01nition is a relatively weak adaptation of uniform convergence to the\ncooperative setting, as the integers mi are allowed to depend on the distributions Di. But this\nobservation only strengthens our lower bound, which holds despite the weak requirement.\nTheorem 4.5. For any k, d \u2208 N and (\u0001, \u03b4) \u2208 (0, 0.1), there exists a hypothesis class F of VC-\ndimension d, such that m(k)\n\n\u0001,\u03b4 \u2265 dk(1 \u2212 \u03b4)/(4\u0001).\n\nProof Sketch of Theorem 4.5. Fix k, d \u2208 N and \u0001, \u03b4 \u2208 (0, 0.1). We de\ufb01ne instance (F, f\u2217,{Di}k\ni=1)\nas follows. The instance space is X = ([k]\u00d7[d])\u222a{\u22a5}, and the hypothesis class F contains all binary\nfunctions on X that map \u22a5 to 0 and take value 1 on at most d points. The target function f\u2217 maps\nevery element in X to 0. Finally, the distribution of each player i \u2208 [k] is given by Di((i, j)) = 2\u0001/d\nfor any j \u2208 [d] and Di(\u22a5) = 1 \u2212 2\u0001.\nNote that if a sample set contains strictly less than d/2 elements in {(i\u2217, 1), (i\u2217, 2), . . . , (i\u2217, d)} for\nsome i\u2217, there is a consistent function in F with error strictly greater than \u0001 on Di\u2217, namely, the\nfunction that maps (i, j) to 1 if and only if i = i\u2217 and (i\u2217, j) is not in the sample set.\nTherefore, to achieve uniform convergence, at least d/2 elements from X \\ {\u22a5} must be drawn from\neach distribution. Since the probability that each sample is different from \u22a5 is 2\u0001, drawing d/2 such\nsamples from k distribution requires \u2126(dk/\u0001) samples.\n\nA complete proof of Theorem 4.5 appears in Appendix C.\nAcknowledgments\nWe thank the anonymous reviewers for their helpful remarks and suggesting an alternative boosting-\nbased approach for the centralized setting. This work was partially supported by the NSF grants\nCCF-1525971, CCF-1536967, CCF-1331175, IIS-1350598, IIS-1714140, CCF-1525932, and CCF-\n1733556, Of\ufb01ce of Naval Research grants N00014-16-1-3075 and N00014-17-1-2428, a Sloan\nResearch Fellowship, and a Microsoft Research Ph.D. fellowship. This work was done while Avrim\nBlum was working at Carnegie Mellon University.\nReferences\n[1] M. Anthony and P. L. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge\n\nUniversity Press, 1999.\n\n[2] Maria Florina Balcan, Avrim Blum, Shai Fine, and Yishay Mansour. Distributed learning, com-\nmunication complexity and privacy. In Proceedings of the 25th Conference on Computational\nLearning Theory (COLT), pages 26.1\u201326.22, 2012.\n\n[3] Jonathan Baxter. A Bayesian/information theoretic model of learning to learn via multiple task\n\nsampling. Machine learning, 28(1):7\u201339, 1997.\n\n[4] Jonathan Baxter. A model of inductive bias learning. Journal of Arti\ufb01cial Intelligence Research,\n\n12:149\u2013198, 2000.\n\n[5] Shai Ben-David and Reba Schuller. Exploiting task relatedness for mulitple task learning.\nIn Proceedings of the 16th Conference on Computational Learning Theory (COLT), pages\n567\u2013580, 2003.\n\n[6] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen-\nnifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):\n151\u2013175, 2010.\n\n[7] Rich Caruana. Multitask learning. Machine Learning, 28(1):41\u201375, 1997.\n\n9\n\n\f[8] Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online\nprediction. In Proceedings of the 28th International Conference on Machine Learning (ICML),\npages 713\u2013720, 2011.\n\n[9] Yoav Freund and Robert E Schapire. A desicion-theoretic generalization of on-line learning and\nan application to boosting. In Proceedings of the 2nd European Conference on Computational\nLearning Theory (EuroCOLT), pages 23\u201337, 1995.\n\n[10] Abhishek Kumar and Hal Daum\u00e9 III. Learning task grouping and overlap in multi-task learning.\nIn Proceedings of the 29th International Conference on Machine Learning (ICML), pages\n1103\u20131110, 2012.\n\n[11] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning\nbounds and algorithms. In Proceedings of the 22nd Conference on Computational Learning\nTheory (COLT), pages 19\u201330, 2009.\n\n[12] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple\nsources. In Proceedings of the 23rd Annual Conference on Neural Information Processing\nSystems (NIPS), pages 1041\u20131048, 2009.\n\n[13] Massimiliano Pontil and Andreas Maurer. Excess risk bounds for multitask learning with trace\nnorm regularization. In Proceedings of the 26th Conference on Computational Learning Theory\n(COLT), pages 55\u201376, 2013.\n\n[14] Leslie G. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134\u20131142,\n\n1984.\n\n[15] Jialei Wang, Mladen Kolar, and Nathan Srerbo. Distributed multi-task learning. In Proceedings\nof the 19th International Conference on Arti\ufb01cial Intelligence and Statistics (AISTATS), pages\n751\u2013760, 2016.\n\n10\n\n\f", "award": [], "sourceid": 1404, "authors": [{"given_name": "Avrim", "family_name": "Blum", "institution": "Toyota Technological Institute at Chicago"}, {"given_name": "Nika", "family_name": "Haghtalab", "institution": "Carnegie Mellon University"}, {"given_name": "Ariel", "family_name": "Procaccia", "institution": "Carnegie Mellon University"}, {"given_name": "Mingda", "family_name": "Qiao", "institution": "Tsinghua University"}]}