{"title": "Achieving budget-optimality with adaptive schemes in crowdsourcing", "book": "Advances in Neural Information Processing Systems", "page_first": 4844, "page_last": 4852, "abstract": "Adaptive schemes, where tasks are assigned based on the data collected thus far, are widely used in practical crowdsourcing systems to efficiently allocate the budget. However, existing theoretical analyses of crowdsourcing systems suggest that the gain of adaptive task assignments is minimal. To bridge this gap, we investigate this question under a strictly more general probabilistic model, which has been recently introduced to model practical crowdsourcing data sets. Under this generalized Dawid-Skene model, we characterize the fundamental trade-off between budget and accuracy, and introduce a novel adaptive scheme that matches this fundamental limit. We further quantify the gain of adaptivity, by comparing the trade-off with the one for non-adaptive schemes, and confirm that the gain is significant and can be made arbitrarily large depending on the distribution of the difficulty level of the tasks at hand.", "full_text": "AchievingBudget-optimalitywithAdaptiveSchemesinCrowdsourcingAshishKhetanandSewoongOhDepartmentofISE,UniversityofIllinoisatUrbana-ChampaignEmail:{khetan2,swoh}@illinois.eduAbstractAdaptiveschemes,wheretasksareassignedbasedonthedatacollectedthusfar,arewidelyusedinpracticalcrowdsourcingsystemstoef\ufb01cientlyallocatethebudget.However,existingtheoreticalanalysesofcrowdsourcingsystemssuggestthatthegainofadaptivetaskassignmentsisminimal.Tobridgethisgap,weinvestigatethisquestionunderastrictlymoregeneralprobabilisticmodel,whichhasbeenrecentlyintroducedtomodelpracticalcrowdsourcingdatasets.UnderthisgeneralizedDawid-Skenemodel,wecharacterizethefundamentaltrade-offbetweenbudgetandaccuracy.Weintroduceanoveladaptiveschemethatmatchesthisfundamentallimit.Agivenbudgetisallocatedovermultiplerounds.Ineachround,asubsetoftaskswithhighenoughcon\ufb01denceareclassi\ufb01ed,andincreasingbudgetisallocatedonremainingonesthatarepotentiallymoredif\ufb01cult.Oneachround,decisionsaremadebasedontheleadingeigenvectorof(weighted)non-backtrackingoperatorcorrespondingtothebipartiteassignmentgraph.Wefurtherquantifythegainofadaptivity,bycomparingthetradeoffwiththeonefornon-adaptiveschemes,andcon\ufb01rmthatthegainissigni\ufb01cantandcanbemadearbitrarilylargedependingonthedistributionofthedif\ufb01cultylevelofthetasksathand.1IntroductionCrowdsourcingplatformsprovidelabormarketsinwhichpiecesofmicro-tasksareelectronicallydistributedtoapoolofworkers.Intypicalcrowdsourcingscenarios,suchasthoseonAmazon\u2019sMechanicalTurk,arequesterpostsacollectionoftasks,andabatchispickedupbyanyworkerwhoiswillingtocompleteit.Theworkerissubsequentlyrewardedforeachtaskhe/shecompletes.However,someworkersarespammerstryingtomakeeasymoney.Moreover,sincetherewardissmallandtasksaretedious,errorsarecommonevenamongthosewhotry.Tocorrectfortheerrors,acommonapproachistointroduceredundancybyassigningeachtasktomultipleworkersandaggregatingtheirresponsesusingsomeschemessuchasmajorityvoting.Afundamentalproblemofinterestishowtomaximizetheaccuracyofthusinferredsolutions,whileusingassmallnumberofrepetitionsaspossible.Therearetwochallengesinachievingsuchanoptimaltradeoffbetweenaccuracyandthebudget:(a)weneedaschemefordecidingwhichtaskstoassigntowhichworkers;and(b)atthesametimeinferthetruesolutionsfromtheirresponses.Sincetheworkersare\ufb02eeting,therequesterhasnocontroloverwhogetstoworkonwhichtasks.Itisimpossibletomakeatrustrelationshipwiththeworkers.Inparticular,itdoesnotmakesensetoexplorereliableworkers,andexploittheminsubsequentsteps.Eacharrivingworkeriscompletelynewandyoumaynevergethimback.Nevertheless,bycomparingresponsesfrommultipleworkers,wecanestimatethetrueanswertothetask,anduseitinsubsequentstepstolearnthereliabilityoftheworkers.Ourbeliefsonthetrueanswersaswellasthedif\ufb01cultyofthetasksandthereliabilityoftheworkerscanbeiterativelyre\ufb01ned,andonecanpotentiallychoosetoassignmoreworkerstothemoredif\ufb01culttasks.Wewouldliketounderstandsuchintricateinterplayoftaskassignmentandinference.30thConferenceonNeuralInformationProcessingSystems(NIPS2016),Barcelona,Spain.\fSetup.Wehavembinaryclassi\ufb01cationtaskstobecompletedbyworkers.Weassumearecentgener-alizationoftheDawid-Skenemodelintroducedin[22]tomodeltheresponses,whichcapturestheheterogeneityinthetasksaswellastheworkers.Precisely,eachnewarrivingworkerisparametrizedbyaqualityparameterpj\u2208[0,1](forthej-tharrivingworker),whichisi.i.d.accordingtosomepriordistributionF.Eachtaskisparametrizedbyadif\ufb01cultyparameterqi\u2208[0,1](forthei-thtask),whichisdrawni.i.d.accordingtosomepriordistributionG.Whenaworkerjisassignedataski,thetaskisperceivedasapositivetaskwithprobabilityqi,andasanegativetaskotherwise.Hence,ifqiisclosetoahalfthenitisconfusinganddif\ufb01culttocorrectlyclassify,andeasyifclosetooneorzero.Whentaskiisassignedtoworkerj,theresponseisanoisyperceptionofthetask:Aij=(cid:26)1,w.p.qipj+\u00afqi\u00afpj,\u22121,w.p.\u00afqipj+qi\u00afpj.,(1)where\u00afqi=1\u2212qiand\u00afpj=1\u2212pj.Withprobabilitypj,theworkeranswerstruthfullyasheperceivesthetask,andotherwisegivestheoppositeanswer.Hence,ifpjisclosetoonethenhetellsthetruth(inhisopinion)andifitisclosetohalfhegivesrandomanswers.Ifitiszero,heisalsoreliable,inthesensethatarequesterwhocancorrectlydecodehisreliabilitycanextractthetruthsexactly.Wede\ufb01nethegroundtruthofataskaswhatthemajorityoftheworkersagreeon,hadweaskedalltheworkers.Accordingly,weassumethatEF[pj]>1/2andthetruelabelsarede\ufb01nedasti=I{qi>(1/2)}\u2212I{qi<(1/2)}.Otherwise,wedonotimposeanyconditiononthedistributionofpj\u2019s.However,weassumeqi\u2019sarediscreterandomvariableswithsupportatKpoints.OurresultsdonotdirectlydependonthissupportsizeK,andthereforeKcanbemadearbitrarilylarge.Notethatwefocusononlybinarytaskswithtwotypesofclasses,andalsotheworkersareassumedtobesymmetric,i.e.theerrorprobabilitydoesnotdependontheperceivedlabelofthetask.TheoriginalDawid-Skenemodelintroducedin[3]andanalyzedin[9]isaspecialcase,whenalltasksareequallyeasy,i.e.qi\u2019sareeitheroneorzero.Thismakesinferenceeasierasalltasksareperceivedtheirtrueclass;theonlysourceoferrorisinworkers\u2019noisyresponses.Weassumethefollowingtaskassignmentscenariotomodelpracticalcrowdsourcingsystems.Itisadiscretetimesystem,whereatthebeginningofeachtimesteptherequestercancreateabatchoftasks.Thisbatchispickedupbyanewarrivingworker,andhis/herresponsesarecollected.Tomodelreal-worldconstraintsweassumethereisalimitonhowmanytasksasingleworkercancomplete,whichwedenotebyr.Therequester(alsocalledtaskmaster)hasnocontroloverwhoisarrivingnext,buthehascontroloverwhichofthemtasksaretobesolvedbythenextarrivingworker.Thisallowsforadaptivetaskassignmentschemes,wheretherequestercanchoosetoincludethosetasksthatheismostuncertainaboutbasedonallthehistoryofresponsescollectedthusfar.Weconsiderallrandomizedtaskassignmentschemes,whoseexpectednumberofassignmentpertaskis\u2018,andallinferencealgorithms.WestudytheminimaxratewhenthenaturechoosestheworstcasepriorsFandG(fromafamilyofpriorsparametrizedbyaverageworkerreliability\u03b2andaveragetaskdif\ufb01culty\u03bbde\ufb01nedin(2)),andwechoosethebestpossibleadaptivetaskassignmenttogetherwiththebestpossibleinferencealgorithm.Wefurtherproposeanoveladaptiveapproachthatachievesthisminimaxrateuptoaconstantfactor.Ourapproachisdifferentfromexistingadaptiveschemesin[5],wheretherearemultipletypesoftasksandthemainsourceofuncertaintyiswhichtypethenextarrivingworkerisexperton.Goldentaskswithknownanswersareusedtoexploreexpertiseandtasksareassignedaccordingly.Relatedwork.ExistingworkoncrowdsourcingsystemsstudythestandardDawid-Skene(DS)model[3],wherealltasksareequallydif\ufb01cultandhenceqi\u2208{0,1}foralltasks.Severalinferencealgorithmshavebeenproposed[3,17,6,16,4,7,11,23,10,21,2,8,14],andthequestionoftaskassignmentisaddressedin[9],wheretheminimaxrateontheprobabilityoferrorischaracterizedandamatchingtaskassignmentschemeandaninferencealgorithmareproposed.Perhapssurprisingly,forthestandardDSmodel,anon-adaptivetaskassignmentschemeachievesthefundamentallimit.Namely,givenmtasksandatotalbudgetform\u2018responses,therequester\ufb01rstconstructsabipartitetask-assignmentgraphwithmtasknodes,n=m\u2018/rworkernodes,andedgesdrawnuniformlyatrandomwithdegree\u2018forthetasknodesandrfortheworkernodes.Then,j-tharrivingworkerisassignedabatchofrtasksthatareadjacenttothej-thworkernode.TogetherwithaninferencealgorithmexplainedindetailinSection2,thisachievesanear-optimalperformance.Namely,toachieveanaverageprobabilityoferror\u03b5,itissuf\ufb01cienttohavetotalbudgetO((m/\u03b2)log(1/\u03b5)),where\u03b2=EF[(2pj\u22121)2]isthequalityoftheworkersde\ufb01nedin(2).Perhapssurprisingly,noadaptiveassignmentcanimproveuponit.Eventhebestadaptiveschemeandthebestinference2\falgorithmstillrequires\u2126((m/\u03b2)log(1/\u03b5))totalbudget.Hence,thereisnogaininadaptivity.ThisnegativeresultreliescruciallyinthefactthatunderthestandardDSmodel,alltasksareinherentlyequallydif\ufb01cult.Hence,adaptivelyassigningmoreworkerstorelativelymoreambiguoustaskshasonlyamarginalgain.However,simpleadaptiveschemesarewidelyusedinpractice,wheresigni\ufb01cantgainsareachieved;inreal-worldsystems,tasksarewidelyheterogeneous.Tocapturesuchvaryingdif\ufb01cultiesinthetasks,generalizationsoftheDSmodelwereproposedin[19,18,22,15]andsigni\ufb01cantimprovementhasbeenreportedoninferenceproblemsforrealdatasets.ThegeneralizedDSmodelservesasthemissingpieceinbridgingthegapbetweenpracticalgainsofadaptivityandtheoreticallimitationsofadaptivity.Weinvestigatethefundamentalquestionof\u201cdoadaptivetaskassignmentsimproveaccuracy?\u201dunderthisgeneralizedDawid-SkenemodelofEq.(1).Contributions.Toinvestigatethegainofadaptivity,we\ufb01rstcharacterizethefundamentallowerboundonthebudgetrequiredtoachieveatargetaccuracy.Tomatchthisfundamentallimit,weintroduceanoveladaptivetaskassignmentscheme.Ourapproachconsistsofmultipleroundsofnon-adaptiveschemes,andweprovidesharpanalysesontheperformanceateachround,whichguidesthedesignofthetaskassignmentineachroundadaptivelyusingthedatafrompreviousrounds.Theproposedadaptivetaskassignmentissimpletoapplyinpractice,andnumericalsimulationscon\ufb01rmthesuperioritycomparedtostate-of-the-artnon-adaptiveschemes.Underacertainassumptiononthechoiceofparametersinthealgorithm,whichrequiresamoderateaccesstoanoracle,wecanprovethattheperformanceoftheproposedadaptiveschemematchesthatofthefundamentallimituptoaconstantfactor.Finally,wequantifythegainofadaptivitybyprovingastrictlylargerlowerboundonthebudgetrequiredforanynon-adaptiveschemes.Precisely,weshowthattheminimaxrateonthebudgetrequiredtoachieveatargetaverageerrorrateof\u03b5scalesas\u0398((m/\u03bb\u03b2)log(1/\u03b5)).ThedependenceonthepriorFandGaresolelycapturedin\u03b2(thequalityofthecrowdasawhole)and\u03bb(thequalityofthetasksasawhole).Weshowthatthefundamentaltradeofffornon-adaptiveschemesis\u0398((m/\u03bbmin\u03b2)log(1/\u03b5)),requiringafactorof\u03bb/\u03bbminlargerbudgetfornon-adaptiveschemes.Thisfactorof\u03bb/\u03bbminispreciselyhowmuchwegainbyadaptivity,andthisgaincanbemadearbitrarilylargeintheworstcasedistributionG.2MainResultsThefollowingquantitiesarefundamentalincapturingthedependenceoftheminimaxrateonthedistributionoftaskdif\ufb01cultiesandworkerreliabilities:\u03bb\u2261EG(cid:20)1(2qi\u22121)2(cid:21)\u22121,\u03b1\u2261EG[(2qi\u22121)2],and\u03b2\u2261EF[(2pj\u22121)2].(2)Letndenotethetotalnumberofworkersused,andTjdenotethesetofalltasksassignedtoworkerj\u2208[n]andWidenotethesetofallworkersassignedtotaski\u2208[m]untiltheadaptivetaskassignmentschemehasterminated.WeconsiderdiscretedistributionGwithKtypesoftasksofvaryingdif\ufb01cultylevels.De\ufb01neeffectivedif\ufb01cultylevelofeachtaskitobe\u03bbi\u2261(2qi\u22121)2,and\u03bbmin=mini\u2208[m]\u03bbi.Ataskwithasmall\u03bbiismoredif\ufb01cult,sinceqicloseto1/2meansthetaskismoreambiguous.Let\u03b4adenotefractionoftotaltaskshavingdif\ufb01cultylevel\u03bbafora\u2208[K]suchthatPa\u2208[K]\u03b4a=1,and\u03b4max\u2261maxa\u2208[K]\u03b4a,\u03b4min\u2261mina\u2208[K]\u03b4a.2.1FundamentallimitundertheadaptivescenarioWeprovealowerboundontheminimaxerrorrate:theerrorthatisachievedbythebestinferencealgorithm\u02c6tusingthebestadaptivetaskassignmentscheme\u03c4underaworstcaseworkerdistributionFandtheworst-casetrueanswerstforthegivendistributionofdif\ufb01cultylevel\u03bbi\u2019s.Notethatgiven\u03bbi,eitherqi=(1+\u221a\u03bbi)/2inwhichcaseti=1orqi=(1\u2212\u221a\u03bbi)/2andti=\u22121.LetT\u2018bethesetofalltaskassignmentschemesthatuseatmostm\u2018queriesintotal,andletF\u03b2bethesetofalltheworkerdistributionssuchthatexpectationofworkerqualityis\u03b2,i.e.F\u03b2\u2261{F|EF[(2pj\u22121)2]=\u03b2}.Thenwecanshowthefollowinglowerboundontheminimaxrateontheprobabilityoferror.AproofofthistheoremisprovidedinSection4inthesupplementarymaterial.Theorem2.1.When\u03b2<1,thereexistsapositiveconstantC0suchthatforeachtaski\u2208[m],min\u03c4\u2208T\u2018,\u02c6tmaxt\u2208{\u00b11}m,F\u2208F\u03b2P[ti6=\u02c6ti|\u03bbi]\u226512e\u2212C0\u03bbi\u03b2E[|Wi||\u03bbi].3\fThisprovesalowerboundonpertaskprobabilityoferrorthatdecaysexponentiallywithexponentscalingas\u03bbi\u03b2E[|Wi||\u03bbi].Theeasierthetask(\u03bbilarge),themorereliabletheworkersare(\u03b2large),andthemoreworkersassignedtothattask(|Wi|large),thesmallertheachievableerror.Togetalowerboundontheaverageprobabilityoferror,supposeweknowthedif\ufb01cultiesofthetasksandassign\u2018aworkerstotasksofdif\ufb01culty\u03bba.WithaveragebudgetconstraintPa\u2208[K]\u2018a\u03b4a\u2264\u2018,min\u03c4\u2208T\u2018,\u02c6tmaxt\u2208{\u00b11}m,F\u2208F\u03b21mmXi=1P[ti6=\u02c6ti]\u2265min\u2018a:Pa\u2208[K]\u03b4a\u2018a=\u2018KXa=112\u03b4ae\u2212C0\u2018a\u03bba\u03b2(3)=12e\u2212C0\u2018\u03bb\u03b2(cid:18)KXa=1\u03b4ae\u2212\u03bbPa6=a0(\u03b4a0/\u03bba0)log(\u03bba/\u03bba0)(cid:19),wheretheequalityfollowsfromsolvingtheoptimizationproblem.Notethatthesummandinthebounddoesnotdependuponthebudget\u2018,anditislowerboundedby\u03b4min>0.Theerrorscalesase\u2212C0\u2018\u03bb\u03b2,where\u03bb=1/(E[1/\u03bbi])asde\ufb01nedin(2),andcaptureshowdif\ufb01cultthesetoftasksarecollectively.Thisgivesalowerboundonthebudget\u0393requiredtoachieveerror\u03b5;thereexistsaconstantC00suchthatif\u0393\u03b5\u2264C00m\u03bb\u03b2log(cid:18)\u03b4min\u03b5(cid:19),(4)thennotaskassignmentscheme(adaptiveornot)withanyinferencealgorithmcanachieveerrorlessthan\u0001.Intuitively,\u03b2capturesthe(collective)qualityoftheworkersasspeci\ufb01edbyFand\u03bbcapturesthe(collective)dif\ufb01cultyofthetasksasspeci\ufb01edbyG.ThisrecoverstheknownfundamentallimitforstandardDSmodelwherealltaskshave\u03bbi=1andhence\u03bb=1in[9]:\u0393\u03b5>C000m\u03b2log(cid:0)1\u0001(cid:1).2.2UpperboundontheachievableerrorrateWepresentanadaptivetaskassignmentschemeandaniterativeinferencealgorithmthatasymp-toticallyachieveanerrorrateofC1e\u2212(C\u03b4/4)\u2018\u03bb\u03b2,whenmgrowslargeand\u2018=\u0398(logm)whereC1=log2(2\u03b4max/\u03b4min)log2(\u03bb1/\u03bbK).Thismatchesthelowerboundin(3)andtheexpectednum-berofqueries(ortask-workerassignments)isboundedbym\u2018.ComparingittoafundamentallowerboundinTheorem2.1establishesthenear-optimalityofourapproach,andthesuf\ufb01cientconditiontoachieveaverageerror\u03b5isfortheaveragetotalbudgettobelargerthan,\u0393\u03b5\u2265C0m\u03bb\u03b2log(cid:16)C1\u03b5(cid:17).(5)2.2.1AdaptivealgorithmSincedif\ufb01cultylevelisvaryingacrossthetasks,itisintuitivetoassignfewerworkerstoeasytasksandmoreworkerstohardtasks.Supposeweknowthedif\ufb01cultylevels,thenoptimizingthelowerbound(3)over\u02dc\u2018i\u2019s,itsuggeststoassign\u02dc\u2018i\u2019\u2018(\u03bb/\u03bbi)workerstothetaskiwithdif\ufb01culty\u03bbi,whengivena\ufb01xedbudgetof\u2018workerspertaskonaverage.However,thedif\ufb01cultylevelsarenotknown.Non-adaptiveschemescanbearbitrarilyworse(seeTheorem2.4).Weproposeanovelapproachofadaptivelyassigningworkersinmultiplerounds,re\ufb01ningourbeliefon\u03bbi,andmakingdecisionsonthetaskswithhighercon\ufb01dence.Themainalgorithmiccomponentisthesub-routineinline8-13ofAlgorithm1.Forachoiceofthe(pertask)budget\u2018t,wecollectresponsesaccordingtoa(\u2018t,rt=\u2018t)regularrandomgraphon|M|tasksand|M|workers.Theleadingeigen-vectorofthenon-backtrackingoperatoronthisbipartitegraph,weightedbythe\u00b11responsesrevealsanoisyobservationofthetrueclassandthedif\ufb01cultylevelsofthetasks.Letx\u2208R|M|denotethetoplefteigenvector,computedasperAlgorithm2.Thenthei-thentryxiasymptoticallyconvergesinthelargenumberoftasksmlimittoaGaussianrandomvariablewithmeanproportionaltothedif\ufb01cultylevel(2qi\u22121),withmeanandvariancespeci\ufb01edinLemma5.1inthethesupplementarymaterial.Thisnon-backtrackingoperatorapproachtocrowdsourcingwas\ufb01rstintroducedin[7]forthestandardDSmodel,isasingle-roundnon-adaptivescheme,andusesathresholdofzerotoclassifytasksbasedonthesignofxi\u2019s.WegeneralizetheiranalysistothisgeneralizedDSmodelinTheorem2.3for\ufb01nitesampleregime,andfurthergiveasharpercharacterizationbasedoncentrallimittheoremintheasymptoticregime(Lemma5.1inthesupplementarymaterial).4\fThisprovidesusasub-routinethatreveals(2qi\u22121)\u2019swewant,corruptedbyadditiveGaussiannoise.Thisresemblesthesettinginracingalgorithmsintroducedin[12]wherethegoalistochoosethevariable(i.e.task)withlargestmean(i.e.easiest)withminimalbudget.However,ourgoalistoidentifythesignofthemeanofthevariables(i.e.classes)withsuf\ufb01cientaccuracy.Thekeyideaistoclassifytheeasiertasks\ufb01rstwithminimalbudget,andthenclassifytheremainingmoredif\ufb01culttaskswithmorebudgetallocatedpertask.WecansetathresholdXt,uateachround,andmakeapermanentdecisiononasubsetoftasksthathavelargexi\u2019sinabsolutevalue,sincethosearethetaskswearemostcon\ufb01dentaboutinitsclass,i.e.sign(2qi\u22121).Wearenowlefttochoosethebudget\u2018tandthethresholdXt,uforeachround.Weprescribeachoiceusingfollowingnotations.Assumethat\u03bba\u2019sareindexedsuchthat\u03bb1>\u03bb2>...>\u03bbK.Forsimplicity,assumethat\u03bbK=\u03bb12\u2212(T\u22121)forsomeT\u2208Z+\\{1}.Giventhedistribution{\u03bba,\u03b4a}a\u2208[K],we\ufb01rstbinittogetanotherdistribution{\u02dc\u03bba,\u02dc\u03b4a}a\u2208[T]whichissupportedatmostatTpoints.Wetake\u02dc\u03bb1=\u03bb1and\u02dc\u03bba+1=\u02dc\u03bba2\u22121foreacha\u2208[T\u22121].\u02dc\u03b4aisthetotalfractionoftaskswhosedif\ufb01culty\u03bbiissmallerthan\u03bb12\u2212(a\u22122)andlargerthan\u03bb12\u2212(a\u22121).Precisely,\u02dc\u03b4a=Pa0\u2208[K]\u03b4a0I(cid:8)\u03bb1/2(a\u22121)\u2264\u03bba0<\u03bb1/2(a\u22122)(cid:9),fora\u2208[T].Thechoiceof2fortheratioof\u02dc\u03bba\u2019sisarbitraryandcanbefurtheroptimizedforagivendistributionof\u03bbi\u2019s.Foreaseofnotationsinwritingthealgorithm,were-indexthebinneddistributiontoget{\u02dc\u03bba,\u02dc\u03b4a}a\u2208[\u02dcT],for\u02dcT\u2264T,suchthat\u02dc\u03b4a6=0foralla\u2208\u02dcT.Notethat\u02dcT\u2264dlog2(\u03bb1/\u03bbK)e.WestartwithasetofalltasksM=[m].Afractionoftasksareclassi\ufb01edineachroundandtheun-classi\ufb01edonesaretakentothenextround.Atroundt\u2208{1,...,\u02dcT},ourgoalistoclassifysuf\ufb01cientfractionofthosetasksinthesamedif\ufb01cultygroup{i\u2208M:\u03bbi=\u03bbt}tobeclassi\ufb01edwithdesiredlevelofaccuracy.If\u2018tistoolowand/orthresholdXt,utoosmall,thenmisclassi\ufb01cationratewillbetoolarge.If\u2018tistoolarge,wearewastingourbudgetunnecessarily.IfXt,uistoolarge,notenoughtaskswillbeclassi\ufb01ed.Wechoose\u2018t=\u2018C\u03b4\u02dc\u03bb/\u02dc\u03bbtandanappropriateXt,utoensurethatthemisclassi\ufb01cationprobabilityisatmostC1e\u2212(C\u03b4/4)\u03bb\u03b2\u2018basedonthecentrallimittheoremontheleadingeigenvector(see(21)inthesupplementarymaterial).Werunthissub-routinest=max{0,dlog2(\u02dc\u03b4t(1+\u03b3t)/\u02dc\u03b4t+1\u03b3t+1)e}timestoensurethatenoughfractionfromt-thgroupisclassi\ufb01ed.Wemakesurethattheexpectednumberofunclassi\ufb01edtasksisatmostequaltothenumberoftasksinthenextgroup,i.e.,dif\ufb01cultylevel\u03bbi=\u03bbt+1.Weprovideanear-optimalperformanceguaranteefor\u03b3t=1forallt\u2208[\u02dcT],and\u03b3tprovidesanextradegreeoffreedomforpractitionerstofurtheroptimizetheef\ufb01ciency.Notethatstatistically,thefractionofthet-thgroup(i.e.taskswithdif\ufb01culty\u02dc\u03bbt)thatgetclassi\ufb01edbeforethet-throundisverysmallasthethresholdsetintheseroundsismorethantheirabsolutemeanmessage.Mosttaskswith\u02dc\u03bbtwillgetclassi\ufb01edinroundt.Further,thebinningoftheoriginalgivendistributiontoget{\u02dc\u03bba,\u02dc\u03b4a}ensuresthat\u2018t+1\u22652\u2018t.Itensuresthatthetotalextraneousbudgetspenton\u02dc\u03bbttasksisnotmorethanaconstanttimestheallocatedbudgetofthosetasks,andtheconstantcanbemadeone,bychangingtheinitialchoiceof\u20181byaconstantfactor.2.2.2PerformanceGuaranteeSincewearenotwastinganybudgetonanyofthetasks,withtherightchoiceoftheconstantC\u03b4,weareguaranteedthatthisalgorithmusesatmostm\u2018assignmentsinexpectation.Onecaveatisthat,thethresholdXt,udependson\u03b1t,u=(1/|M|)Pi\u2208[M]\u03bbi,whichistheaveragedif\ufb01cultyoftheremainingtasks.Astheremainingtasksarechangingoverthecourseofthealgorithm,weneedtoestimatethisvalueineachsub-routine.Weprovideanestimatorof\u03b1t,uinAlgorithm3(inthesupplementarymaterial)thatonlyusesthesampledresponsesthatarealreadycollected.Allnumericalresultsarebasedonthisestimator.However,analyzingthesensitivityoftheperformancewithrespecttotheestimationerrorin\u03b1t,uisquitechallenging,andforatheoreticalanalysis,weassumewehaveaccesstoanoraclethatprovidestheexactvalueof\u03b1t,u,replacingAlgorithm3.Theorem2.2.SupposeAlgorithm3returnstheexactvalueof\u03b1t,u=(1/|M|)Pi\u2208[M]\u03bbi.Withthechoiceof\u03b3a=1foralla\u2208[\u02dcT]andC\u03b4=(4+dlog(2\u03b4max/\u03b4min)e)\u22121foranygivendis-tributionoftaskdif\ufb01culty{\u03bba,\u03b4a}a\u2208[K]ofmtasksandanaveragenumberofworkerspertask\u2018=\u0398(logm),theexpectednumberofqueriesmadebyAlgorithm1isasymptoticallyboundedbylimm\u2192\u221ePt\u2208[\u02dcT],u\u2208st\u2018tE[|Mt,u|]/(m\u2018)\u22641,whereMt,uisthenumberoftasksremainingatround5\f(t,u).Further,Algorithm1returnsestimates{\u02c6ti}i\u2208[m]thatasymptoticallyachieves,limm\u2192\u221e1mmXi=1P[ti6=\u02c6ti]\u2264C1e\u2212(C\u03b4/4)\u2018\u03bb\u03b2,(6)whereC1=log2(2\u03b4max/\u03b4min)log2(\u03bb1/\u03bbK)for\u03bb\u03b2scalingas1/\u2018suchthat\u2018\u03bb\u03b2=\u0398(1).AproofofthistheoremisprovidedinSection5inthesupplementarymaterial.Thisshowsthenear-optimalsuf\ufb01cientconditionofourapproachin(5).TheconstantC\u03b4canbeimprovedbyoptimizingoverthechoiceof\u03b3a\u2019sbyminimizingtheexpectednumberofqueriesthatthealgorithmmakes.Algorithm1AdaptiveTaskAssignmentandInferenceAlgorithmRequire:m,{\u02dc\u03bba,\u02dc\u03b4a}a\u2208[\u02dcT],\u2018,C\u03b4,{\u03b3a}a\u2208[\u02dcT],\u03b1,\u03b2,\u00b5=E[2pj\u22121]Ensure:Estimate{\u02c6ti}i\u2208[m]1:M\u2190{1,2,\u00b7\u00b7\u00b7,m},\u02dc\u03bb=(cid:16)Pa\u2208[\u02dcT](\u02dc\u03b4a/\u02dc\u03bba)(cid:17)\u221212:forallt=1,2,\u00b7\u00b7\u00b7,\u02dcTdo3:\u2018t\u2190(\u2018C\u03b4\u02dc\u03bb)/\u02dc\u03bbt,rt\u2190\u2018t4:st\u2190maxn0,llog(cid:16)\u02dc\u03b4t(1+\u03b3t)\u02dc\u03b4t+1\u03b3t+1(cid:17)moI{t<\u02dcT}+1I{t=\u02dcT}5:forallu=1,2,\u00b7\u00b7\u00b7,stdo6:ifM6=\u2205then7:n\u2190|M|,k\u2190plog|M|8:DrawE\u2208{0,1}|M|\u00d7n\u223c(\u2018t,rt)-regularrandomgraph9:Collectanswers{Ai,j\u2208{1,\u22121}}(i,j)\u2208E10:{xi}i\u2208M\u2190Algorithm2(cid:2)E,{Ai,j}(i,j)\u2208E,k(cid:3)11:\u03b1t,u\u2190Algorithm3[E,{Ai,j}(i,j)\u2208E,\u2018t,rt]12:Xt,u\u2190p\u02dc\u03bbt\u00b5\u2018t(cid:0)(\u2018t\u22121)(rt\u22121)\u03b1t,u\u03b2(cid:1)k\u22121I{t<\u02dcT}+0I{t=\u02dcT}13:(cid:8)\u02c6ti=I{xi>Xt,u}\u2212I{xi<\u2212Xt,u}(cid:9)i\u2208M,M\u2190{i\u2208M:|xi|\u2264Xt,u}14:endif15:endfor16:endforAlgorithm2Message-PassingAlgorithmRequire:E\u2208{0,1}|M|\u00d7n,{Aij\u2208{1,\u22121}}(i,j)\u2208E,kmaxEnsure:{xi\u2208R}i\u2208[|M|]1:forall(i,j)\u2208Edo2:Initializey(0)j\u2192iwithrandomZj\u2192i\u223cN(1,1)3:endfor4:forallk=1,2,\u00b7\u00b7\u00b7,kmaxdo5:forall(i,j)\u2208Edo6:x(k)i\u2192j\u2190Pj0\u2208Wi\\jAij0yk\u22121j0\u2192i7:endfor8:forall(i,j)\u2208Edo9:y(k)j\u2192i\u2190Pi0\u2208Tj\\iAi0jxki0\u2192j10:endfor11:endfor12:foralli\u2208[m]do13:xi\u2190Pj\u2208WiAijykmax\u22121j\u2192i14:endforInFigure1,wecompareperformanceofouralgorithmwithmajorityvotingandalsonon-adaptiveversionofourAlgorithm1,whereweassigntoeachtask\u2018(thegivenbudget)numberofworkersin6\foneroundandsetclassi\ufb01cationthresholdXt,u=0soastoclassifyallthetasks.Thisnon-adaptivespecialcasehasbeenintroducedforthestandardDSmodelin[9].Wemakeaslightmodi\ufb01cationtoAlgorithm1.Inthe\ufb01nalround,whentheclassi\ufb01cationthresholdissettozero,weincludealltheresponsescollectedthusfarwhenrunningthemessagepassingAlgorithm2,andnotjustthefreshsamplescollectedinthatround.Thiscreatesdependenciesbetweenrounds,whichmakestheanalysischallenging.However,inpracticeweseeimprovedperformanceanditallowsustousethegiven\ufb01xedbudgetef\ufb01ciently.Werunsyntheticexperimentswithm=1800and\ufb01xn=1800forthenon-adaptiveversion.Thecrowdsaregeneratedfromthespammer-hammermodelwithhammerprobabilityequalto0.3.Intheleftpanel,wetakedif\ufb01cultylevel\u03bbatobeuniformlydistributedover{1,1/4,1/16},thatgives\u03bb=1/7.Intherightpanel,wetake\u03bba=1withprobability3/4,otherwisewetakeittobe1/4or1/16withequalprobability,thatgives\u03bb=4/13.Aspredictedfromthetheoreticalanalysis,ouradaptivealgorithmimprovessigni\ufb01cantlyoveritsnon-adaptiveversion.Inparticular,fortheleftpanel,thenon-adaptivealgorithm\u2019serrorscalingdependsonsmallest\u03bbithatis1/16whilefortheadaptivealgorithmitscaleswith\u03bb=1/7.Inthe\ufb01gure,itcanbeseenthattheadaptivealgorithmrequiresapproximately(7/16)\u2018queriestoacheivethesameerrorasachievedbythenon-adaptiveoneusing\u2018queries.Thisgapwidensintherightpaneltoapproximately(13/64)aspredicted,andtheadaptivealgorithmachieveszeroerrorasthenumberofqueriesincrease.Forafaircomparisonwiththenon-adaptiveversion,we\ufb01xtotalbudgettobem\u2018andassignworkersineachrounduntilthebudgetisexhausted.C\u03b4is1andst=1fort\u2208{1,2,3}.1e-0061e-0050.00010.0010.010.1 50 100 150 200 250 300 350Majority votingNon-adaptiveAdaptiveprobabilityoferrornumberofqueriespertask\u20181e-0061e-0050.00010.0010.010.1 50 100 150 200 250 300Majority votingNon-adaptiveAdaptiveprobabilityoferrornumberofqueriespertask\u2018Figure1:Algorithm1improvessigni\ufb01cantlyoveritsnon-adaptiveversionandmajorityvoting.2.3Achievableerrorrateunderthenon-adaptivescenarioConsideranon-adaptiveversionofourapproachwhereweapplyitforoneroundusingan(\u2018,r)randomregulargraph,where\u2018isthegivenbudget.Naturally,theclassi\ufb01cationthresholdissettoXt,u=0soastoclassifyallthetasks.Weprovideasharpupperboundontheachievederror,thatholdsforall(non-asymptotic)regimesofm.De\ufb01ne\u03c32kas\u03c32k\u22612\u03b2\u00b52(cid:0)\u02c6\u2018\u02c6r(\u03b1\u03b2)2(cid:1)k\u22121+3(cid:18)1+1\u02c6r\u03b1\u03b2(cid:19)1\u22121/(cid:0)\u02c6\u2018\u02c6r(\u03b1\u03b2)2(cid:1)k\u221211\u22121/(cid:0)\u02c6\u2018\u02c6r(\u03b1\u03b2)2(cid:1).(7)Thiscapturestheeffectivevarianceinthesub-Gaussiantailofthemessagesxi\u2019safterkiterationsoftheinferencealgorithm(Algorithm2),asshownintheproofofthefollowingtheorem(seethesupplementarymaterialinSection6).Theorem2.3.Forany\u2018>1andr>1,supposemtasksareassignedaccordingtoarandom(\u2018,r)-regulargraphdrawnfromthecon\ufb01gurationmodel.If\u00b5>0,\u02c6\u2018\u02c6r\u03b12\u03b22>1,and\u02c6r\u03b1>1,thenforanyt\u2208{\u00b11}m,theestimate\u02c6ti=sign(xi)afterkiterationsofAlgorithm2achievesP(cid:2)ti6=\u02c6t(k)i(cid:12)(cid:12)\u03bbi(cid:3)\u2264e\u2212\u2018\u03b2\u03bbi/(2\u03c32k)+3\u2018rm(\u02c6\u2018\u02c6r)2k\u22122.(8)Therefore,theaverageerrorrateisboundedby1mmXi=1P[ti6=\u02c6t(k)i]\u2264EG(cid:20)e\u2212\u2018\u03b2\u03bbi2\u03c32k(cid:21)+3\u2018rm(\u02c6\u2018\u02c6r)2k\u22122.(9)7\fThesecondterm,whichistheprobabilitythattheresulting(\u2018,r)regularrandomgraphisnotlocallytree-like,canbemadesmallforlargemaslongask=O(\u221alogm)(whichisthechoicewemakeinAlgorithm1).Hence,thedominanttermintheerrorboundisthe\ufb01rstterm.Further,whenwerunouralgorithmforlargeenoughnumbersofiterations,\u03c32kconvergeslinearlytoa\ufb01nitelimit\u03c32\u221e\u2261limk\u2192\u221e\u03c32ksuchthat\u03c32\u221e=3(cid:0)1+1/(\u02c6r\u03b1\u03b2)(cid:1)(\u02c6\u2018\u02c6r\u03b1\u03b2)2/((\u02c6\u2018\u02c6r\u03b1\u03b2)2\u22121),whichforlargeenough\u02c6r\u03b1\u03b2and\u02c6\u2018\u02c6risupperboundedbyaconstant.Hence,forawiderangeofparameters,theaverageerrorin(9)isdominatedbyEG(cid:2)e\u2212\u2018\u03b2\u03bbi/2\u03c32k(cid:3)=Pa\u03b4ae\u2212C\u2018\u03b2\u03bba.Whenall\u03b4\u2019sarestrictlypositive,theerrorisdominatedbythedif\ufb01culttaskswith\u03bbmin=mina\u03bba,asillustratedinFigure2.Hence,itissuf\ufb01cienttohavebudget\u0393\u03b5\u2265C00m/(\u03bbmin\u03b2)log(1/\u03b5)toachieveanaverageerrorof\u03b5>0.Suchascalingisalsonecessaryasweshowinthenextsection.ThisisfurtherillustratedinFigure2.Theerrordecaysexponentiallyin\u2018and\u03b2aspredicted,buttherateofdecaycruciallyhingesonthedif\ufb01cultylevel.Werunsyntheticexperimentswithm=n=1000andthecrowdsaregeneratedfromthespammer-hammermodelwherepj=1withprobability\u03b2and1/2otherwise.We\ufb01x\u03b2=0.3andvary\u2018intheleft\ufb01gureand\ufb01x\u2018=30andvary\u03b2intheright\ufb01gure.Weletqi\u2019stakevaluesin{0.6,0.8,1}withequalprobabilitysuchthat\u03b1=1.4/3.Theerrorrateofeachtaskgroupedbytheirdif\ufb01cultyisplottedinthedashedlines,matchingpredictede\u2212\u2126(\u2018\u03b2(2qi\u22121)2).Theaverageerrorratesinsolidlinesaredominatedbythoseofthedif\ufb01culttasks,whichisauniversaldrawbackforallnon-adaptiveschemes.1e-0050.00010.0010.010.11 0 5 10 15 20 25 30q = 1.0q = 0.8q = 0.6Mean Errorprobabilityoferrornumberofqueriespertask\u20181e-0050.00010.0010.010.11 0 0.1 0.2 0.3 0.4q = 1.0q = 0.8q = 0.6Mean Errorprobabilityoferrorcrowdquality\u03b2Figure2:Non-adaptiveschemessufferasaverageerrorisdominatedbydif\ufb01culttasks.2.4Fundamentallimitunderthenon-adaptivescenarioTheorem2.3impliesthatitsuf\ufb01cestoassign\u2018\u2265(c/(\u03b2\u03bbi))log(1/\u03b5)toachieveanerrorsmallerthan\u03b5forataski.Weshowinthefollowingtheoremthatthisscalingisalsonecessary.Hence,applyingoneroundofAlgorithm1isnear-optimalinthenon-adaptivescenariocomparedtoaminimaxratewherethenaturechoosestheworstdistributionofworkerpj\u2019samongthesetofdistributionswiththesame\u03b2.WeprovideaproofofthetheoreminSection7inthesupplementarymaterial.Theorem2.4.ThereexistsapositiveconstantC0andadistributionFofworkerswithaveragereliabilityE[(2pj\u22121)2]=\u03b2s.t.when\u03bbi<1,ifthenumberofworkersassignedtotaskibyanynon-adaptivetaskassignmentschemeislessthan(C0/(\u03b2\u03bbi))log(1/\u0001),thennoalgorithmcanachieveconditionalprobabilityoferrorontaskilessthan\u0001foranymandr.Sinceinthisnon-adaptivescheme,taskassignmentsaredoneapriori,thereareonaverage\u2018workersassignedtoanysetoftasksofthesamedif\ufb01culty.Hence,ifthetotalbudgetislessthan\u0393\u03b5\u2264C0m\u03bbmin\u03b2log\u03b4min\u03b5,(10)thennoalgorithmcanachieveaverageerrorlessthan\u03b5,where\u03bbmin=mina\u03bba.Comparedtotheadaptivecasein(4)(nearlyachievedin(5)),thegainofadaptivityisafactorof\u03bb/\u03bbmin.TheRHSisnegativewhen\u03b4min<\u03b5,andcanbetightenedtoC0(m/\u03bba\u03b2)log(Pab=1\u03b4b/\u03b5)whereaisthesmallestintegersuchthatPab=1\u03b4b>\u03b5.AcknowledgementsThisworkissupportedbyNSFSaTCawardCNS-1527754,andNSFCISEawardCCF-1553452.8\fReferences[1]N.AlonandJ.H.Spencer.Theprobabilisticmethod.JohnWileyandSons,2004.[2]N.Dalvi,A.Dasgupta,R.Kumar,andV.Rastogi.Aggregatingcrowdsourcedbinaryratings.InProceedingsofthe22ndinternationalconferenceonWorldWideWeb,pages285\u2013294,2013.[3]A.P.DawidandA.M.Skene.Maximumlikelihoodestimationofobservererror-ratesusingtheemalgorithm.Appliedstatistics,pages20\u201328,1979.[4]A.Ghosh,S.Kale,andP.McAfee.Whomoderatesthemoderators?:crowdsourcingabusedetectioninuser-generatedcontent.InProceedingsofthe12thACMconferenceonElectroniccommerce,pages167\u2013176.ACM,2011.[5]C.Ho,S.Jabbari,andJ.W.Vaughan.Adaptivetaskassignmentforcrowdsourcedclassi\ufb01cation.InProceedingsofthe30thInternationalConferenceonMachineLearning(ICML-13),pages534\u2013542,2013.[6]R.JinandZ.Ghahramani.Learningwithmultiplelabels.InAdvancesinneuralinformationprocessingsystems,pages921\u2013928,2003.[7]D.R.Karger,S.Oh,andD.Shah.Iterativelearningforreliablecrowdsourcingsystems.InAdvancesinneuralinformationprocessingsystems,pages1953\u20131961,2011.[8]D.R.Karger,S.Oh,andD.Shah.Ef\ufb01cientcrowdsourcingformulti-classlabeling.InProceedingsoftheACMSIGMETRICS/internationalconferenceonMeasurementandmodelingofcomputersystems,pages81\u201392,2013.[9]D.R.Karger,S.Oh,andD.Shah.Budget-optimaltaskallocationforreliablecrowdsourcingsystems.OperationsResearch,62:1\u201324,2014.[10]H.LiandB.Yu.Errorrateboundsanditerativeweightedmajorityvotingforcrowdsourcing.arXivpreprintarXiv:1411.4086,2014.[11]Q.Liu,J.Peng,andA.Ihler.Variationalinferenceforcrowdsourcing.InAdvancesinNeuralInformationProcessingSystems25,pages701\u2013709,2012.[12]O.MaronandA.W.Moore.Hoeffdingraces:Acceleratingmodelselectionsearchforclassi\ufb01cationandfunctionapproximation.RoboticsInstitute,page263,1993.[13]M.MezardandA.Montanari.Information,physics,andcomputation.OxfordUniversityPress,2009.[14]J.Ok,S.Oh,J.Shin,andY.Yi.Optimalityofbeliefpropagationforcrowdsourcedclassi\ufb01cation.InInternationalConferenceonMachineLearning,2016.[15]N.B.Shah,S.Balakrishnan,andM.J.Wainwright.Apermutation-basedmodelforcrowdlabeling:Optimalestimationandrobustness.arXivpreprintarXiv:1606.09632,2016.[16]V.SSheng,F.Provost,andP.G.Ipeirotis.Getanotherlabel?improvingdataqualityanddataminingusingmultiple,noisylabelers.InProceedingsofthe14thACMSIGKDD,pages614\u2013622.ACM,2008.[17]P.Smyth,U.Fayyad,M.Burl,P.Perona,andP.Baldi.Inferringgroundtruthfromsubjectivelabellingofvenusimages.InNIPS,pages1085\u20131092,1995.[18]P.Welinder,S.Branson,S.Belongie,andP.Perona.Themultidimensionalwisdomofcrowds.InAdvancesinNeuralInformationProcessingSystems,pages2424\u20132432,2010.[19]J.Whitehill,P.Ruvolo,T.Wu,J.Bergsma,andJ.Movellan.Whosevoteshouldcountmore:Optimalintegrationoflabelsfromlabelersofunknownexpertise.InAdvancesinNeuralInformationProcessingSystems,volume22,pages2035\u20132043,2009.[20]D.Williams.Probabilitywithmartingales.Cambridgeuniversitypress,1991.[21]Y.Zhang,X.Chen,D.Zhou,andM.I.Jordan.Spectralmethodsmeetem:Aprovablyoptimalalgorithmforcrowdsourcing.InAdvancesinneuralinformationprocessingsystems,pages1260\u20131268,2014.[22]D.Zhou,Q.Liu,J.C.Platt,C.Meek,andN.B.Shah.Regularizedminimaxconditionalentropyforcrowdsourcing.arXivpreprintarXiv:1503.07240,2015.[23]D.Zhou,J.Platt,S.Basu,andY.Mao.Learningfromthewisdomofcrowdsbyminimaxentropy.InAdvancesinNeuralInformationProcessingSystems25,pages2204\u20132212,2012.9\f", "award": [], "sourceid": 2454, "authors": [{"given_name": "Ashish", "family_name": "Khetan", "institution": "University of Illinois Urbana-"}, {"given_name": "Sewoong", "family_name": "Oh", "institution": "UIUC"}]}