{"title": "Parallel Double Greedy Submodular Maximization", "book": "Advances in Neural Information Processing Systems", "page_first": 118, "page_last": 126, "abstract": "Many machine learning problems can be reduced to the maximization of submodular functions. Although well understood in the serial setting, the parallel maximization of submodular functions remains an open area of research with recent results only addressing monotone functions. The optimal algorithm for maximizing the more general class of non-monotone submodular functions was introduced by Buchbinder et al. and follows a strongly serial double-greedy logic and program analysis. In this work, we propose two methods to parallelize the double-greedy algorithm. The first, coordination-free approach emphasizes speed at the cost of a weaker approximation guarantee. The second, concurrency control approach guarantees a tight 1/2-approximation, at the quantifiable cost of additional coordination and reduced parallelism. As a consequence we explore the trade off space between guaranteed performance and objective optimality. We implement and evaluate both algorithms on multi-core hardware and billion edge graphs, demonstrating both the scalability and tradeoffs of each approach.", "full_text": "ParallelDoubleGreedySubmodularMaximizationXinghaoPan1StefanieJegelka1JosephGonzalez1JosephBradley1MichaelI.Jordan1,21DepartmentofElectricalEngineeringandComputerScience,and2DepartmentofStatisticsUniversityofCalifornia,Berkeley,Berkeley,CAUSA94720{xinghao,stefje,jegonzal,josephkb,jordan}@eecs.berkeley.eduAbstractManymachinelearningproblemscanbereducedtothemaximizationofsub-modularfunctions.Althoughwellunderstoodintheserialsetting,theparallelmaximizationofsubmodularfunctionsremainsanopenareaofresearchwithrecentresults[1]onlyaddressingmonotonefunctions.Theoptimalalgorithmformaximizingthemoregeneralclassofnon-monotonesubmodularfunctionswasintroducedbyBuchbinderetal.[2]andfollowsastronglyserialdouble-greedylogicandprogramanalysis.Inthiswork,weproposetwomethodstoparallelizethedouble-greedyalgorithm.The\ufb01rst,coordination-freeapproachemphasizesspeedatthecostofaweakerapproximationguarantee.Thesecond,concurrencycontrolapproachguaranteesatight1/2-approximation,atthequanti\ufb01ablecostofadditionalcoordinationandreducedparallelism.Asaconsequenceweexplorethetradeoffspacebetweenguaranteedperformanceandobjectiveoptimality.Weimplementandevaluatebothalgorithmsonmulti-corehardwareandbillionedgegraphs,demonstratingboththescalabilityandtradeoffsofeachapproach.1IntroductionManyimportantproblemsincludingsensorplacement[3],imageco-segmentation[4],MAPinferencefordeterminantalpointprocesses[5],in\ufb02uencemaximizationinsocialnetworks[6],anddocumentsummarization[7]maybeexpressedasthemaximizationofasubmodularfunction.Thesubmodularformulationenablestheuseoftargetedalgorithms[2,8]thatoffertheoreticalworst-caseguaranteesonthequalityofthesolution.Forseveralmaximizationproblemsofmonotonesubmodularfunctions(satisfyingF(A)\u2264F(B)forallA\u2286B),asimplegreedyalgorithm[8]achievestheoptimalapproximationfactorof1\u22121e.Theoptimalresultforthewider,importantclassofnon-monotonefunctions\u2014anapproximationguaranteeof1/2\u2014ismuchmorerecent,andachievedbyadoublegreedyalgorithmbyBuchbinderetal.[2].Whiletheoreticallyoptimal,inpracticethesealgorithmsdonotscaletolargerealworldproblems,sincetheinherentlyserialnatureofthealgorithmsposesachallengetoleveragingadvancesinparallelhardware.Thislimitationraisesthequestionofparallelalgorithmsforsubmodularmaximizationthatideallypreservethetheoreticalbounds,orweakenthemgracefully,inaquanti\ufb01ablemanner.Inthispaper,weaddressthechallengeofparallelizationofgreedyalgorithms,inparticularthedoublegreedyalgorithm,fromtheperspectiveofparalleltransactionprocessingsystems.Thisalternativeperspectiveallowsustoapplyadvancesindatabaseresearchrangingfromfastcoordination-freeapproacheswithlimitedguaranteestosophisticatedconcurrencycontroltechniqueswhichensureadirectcorrespondencebetweenparallelandserialexecutionsattheexpenseofincreasedcoordination.Wedeveloptwoparallelalgorithmsforthemaximizationofnon-monotonesubmodularfunctionsthatoperateatdifferentpointsalongthecoordinationtradeoffcurve.WeproposeCF-2gasacoordination-freealgorithmandcharacterizetheeffectofreducedcoordinationontheapproximationratio.ByboundingthepossibleoutcomesofconcurrenttransactionsweintroducetheCC-2galgorithmwhich1\fguaranteesserializableparallelexecutionandretainstheoptimalityofthedoublegreedyalgorithmattheexpenseofincreasedcoordination.Theprimarycontributionsofthispaperare:1.Weproposetwoparallelalgorithmsforunconstrainednon-monotonesubmodularmaximiza-tion,whichtradeoffparallelismandtightapproximationguarantees.2.WeprovideapproximationguaranteesforCF-2gandanalyticallyboundtheexpectedlossinobjectivevalueforset-coverwithcostsandmax-cutasrunningexamples.3.WeprovethatCC-2gpreservestheoptimalityoftheserialdoublegreedyalgorithmandanalyticallyboundtheadditionalcoordinationoverheadforcoveringwithcostsandmax-cut.4.Wedemonstrateempiricallyusingtwosyntheticandfourrealdatasetsthatourparallelalgorithmsperformwellintermsofbothspeedandobjectivevalues.Therestofthepaperisorganizedasfollows.Sec.2discussestheproblemofsubmodularmaximiza-tionandintroducesthedoublegreedyalgorithm.Sec.3providesbackgroundonconcurrencycontrolmechanisms.WedescribeandprovideintuitionforourCF-2gandCC-2galgorithmsinSec.4andSec.5,andthenanalyzethealgorithmsboththeoretically(Sec.6)andempirically(Sec.7).2SubmodularMaximizationAsetfunctionF:2V\u2192Rde\ufb01nedoversubsetsofagroundsetVissubmodularifitsatis\ufb01esdiminishingmarginalreturns:forallA\u2286B\u2286Vande/\u2208B,itholdsthatF(A\u222a{e})\u2212F(A)\u2265F(B\u222a{e})\u2212F(B).Throughoutthispaper,wewillassumethatFisnonnegativeandF(\u2205)=0.Submodularfunctionshaveemergedinareassuchasgametheory[9],graphtheory[10],combinatorialoptimization[11],andmachinelearning[12,13].Castingmachinelearningproblemsassubmodularoptimizationenablestheuseofalgorithmsforsubmodularmaximization[2,8]thatoffertheoreticalworst-caseguaranteesonthequalityofthesolution.Whilethosealgorithmsconferstrongguarantees,theirdesignisinherentlyserial,limitingtheirusabilityinlarge-scaleproblems.Recentworkhasaddressedfaster[14]andparallel[1,15,16]versionsofthegreedyalgorithmbyNemhauseretal.[8]formaximizingmonotonesubmodularfunctionsthatsatisfyF(A)\u2264F(B)foranyA\u2286B\u2286V.However,manyimportantapplicationsinmachinelearningleadtonon-monotonesubmodularfunctions.Forexample,graphicalmodelinference[5,17],ortradingoffanysubmodulargainmaximizationwithcosts(functionsoftheformF(S)=G(S)\u2212\u03bbM(S),whereG(S)ismonotonesubmodularandM(S)alinear(modular)costfunction),suchasforutility-privacytradeoffs[18],requiremaximizingnon-monotonesubmodularfunctions.Fornon-monotonefunctions,thesimplegreedyalgorithmin[8]canperformarbitrarilypoorly(seeAppendixH.1foranexample).Intuitively,theintroductionofadditionalelementswithmonotonesubmodularfunctionsneverdecreasestheobjectivewhileintroducingelementswithnon-monotonesubmodularfunctionscandecreasetheobjectivetoitsminimum.Fornon-monotonefunctions,Buchbinderetal.[2]recentlyproposedanoptimaldoublegreedyalgorithmthatworkswellinaserialsetting.Inthispaper,westudyparallelizationsofthisalgorithm.Theserialdoublegreedyalgorithm.TheserialdoublegreedyalgorithmofBuchbinderetal.[2](Ser-2g,inAlg.3)maintainstwosetsAi\u2286Bi.Initially,A0=\u2205andB0=V.Initerationi,thesetAi\u22121containstheitemsselectedbeforeitem/iterationi,andBi\u22121containsAiandtheitemsthataresofarundecided.ThealgorithmseriallypassesthroughtheitemsinVanddeterminesonlinewhethertokeepitemi(addtoAi)ordiscardit(removefromBi),basedonathresholdthattradesoffthegain\u2206+(i)=F(Ai\u22121\u222ai)\u2212F(Ai\u22121)ofaddingitothecurrentlyselectedsetAi\u22121,andthegain\u2206\u2212(i)=F(Bi\u22121\\i)\u2212F(Bi\u22121)ofremovingifromthecandidateset,estimatingitscomplementaritytootherremainingelements.Foranyelementordering,thisalgorithmachievesatight1/2-approximationinexpectation.3ConcurrencyPatternsforParallelMachineLearningInthispaperweadoptatransactionalviewoftheprogramstateandexploreparallelizationstrategiesthroughthelensofparalleltransactionprocessingsystems.Werecasttheprogramstate(thesetsAandB)asdata,andtheoperations(addingelementstoAandremovingelementsfromB)as2\ftransactions.Morepreciselywereformulatethedoublegreedyalgorithm(Alg.3)asaseriesofexchangeable,Read-Writetransactionsoftheform:Te(A,B),((A\u222ae,B)ifue\u2264[\u2206+(A,e)]+[\u2206+(A,e)]++[\u2206\u2212(B,e)]+(A,B\\e)otherwise.(1)ThetransactionTeisafunctionfromthesetsAandBtonewsetsAandBbasedontheelemente\u2208Vandthepredeterminedrandombitsueforthatelement.BycomposingthetransactionsTn(Tn\u22121(...T1(\u2205,V)))werecovertheserialdouble-greedyalgo-rithmde\ufb01nedinAlg.3.Infact,anyorderingoftheserialcompositionofthetransactionsrecoversapermutedexecutionofAlg.3andthereforetheoptimalapproximationalgorithm.However,thisraisesthequestion:isitpossibletoapplytransactionsinparallel?IfweexecutetransactionsTiandTj,withi6=j,inparallelweneedamethodtomergetheresultingprogramstates.Inthecontextofthedoublegreedyalgorithm,wecouldde\ufb01netheparallelexecutionoftwotransactionsas:Ti(A,B)+Tj(A,B),(Ti(A,B)A\u222aTj(A,B)A,Ti(A,B)B\u2229Tj(A,B)B),(2)theunionoftheresultingAandtheintersectionoftheresultingB.WhilewecaneasilygeneralizeEq.(2)tomanyparalleltransactions,wecannotalwaysguaranteethattheresultwillcorrespondtoaserialcompositionoftransactions.Asaconsequence,wecannotdirectlyapplytheanalysisofBuchbinderetal.[2]toderivestrongapproximationguaranteesfortheparallelexecution.Fortunately,severaldecadesofresearch[19,20]indatabasesystemshaveexploredef\ufb01cientparalleltransactionprocessing.Inthispaperweadoptacoordinatedboundsapproachtoparalleltransactionprocessinginwhichparalleltransactionsareconstructedunderboundsonthepossibleprogramstate.Ifthetransactioncouldviolatetheboundthenitisprocessedseriallyontheserver.Byadjustingthede\ufb01nitionoftheboundwecanspanaspaceofcoordination-freetoserializableexecutions.Algorithm1:Generalizedtransactions1forp\u2208{1,...,P}doinparallel2while\u2203elementtoprocessdo3e=nextelementtoprocess4(ge,i)=requestGuarantee(e)5\u2202i=propose(e,ge)6commit(e,i,\u2202i)//Non-blockingAlgorithm2:Committransactioni1waituntil\u2200j<i,processed(j)=true2Atomically3if\u2202i=FAILthen//Deferredproposal4\u2202i=propose(e,S)//Advancetheprogramstate5S\u2190\u2202i(S)Figure1:Algorithmforgeneralizedtransactions.Eachtransactionrequestsitspositioniinthecommitordering,aswellastheboundsgethatareguaranteedtoholdwhenitcommits.Transactionsarealsoguaranteedtobecommittedaccordingtothegivenordering.InFig.1wedescribethecoordinatedboundstransactionpattern.Theclients(Alg.1),inparallel,constructandcommittransactionsunderboundedassumptionsabouttheprogramstateS(i.e.,thesetsAandB).TransactionsareconstructedbyrequestingthelatestboundgeonSatlogicaltimeiandcomputingachange\u2202itoS(e.g.,AddetoA).Iftheboundisinsuf\ufb01cienttoconstructthetransactionthen\u2202i=FAILisreturned.Theclientthensendstheproposedchange\u2202itotheservertobecommittedatomicallyandproceedstothenextelementwithoutwaitingforaresponse.Theserver(Alg.2)seriallyappliesthetransactionsadvancingtheprogramstate(i.e.,addingelementstoAorremovingelementsfromB).Iftheboundswereinsuf\ufb01cientandthetransactionfailedattheclient(i.e.,\u2202i=FAIL)thentheserverseriallyreconstructsandappliesthetransactionunderthetrueprogramstate.Moreover,theserverisresponsibleforderivingbounds,processingtransactionsinthelogicalorderi,andproducingtheserializableoutput\u2202n(\u2202n\u22121(...\u22021(S))).Thismodelachievesahighdegreeofparallelismwhenthecostofconstructingthetransactiondominatesthecostofapplyingthetransaction.Forexample,inthecaseofsubmodularmaximization,thecostofconstructingthetransactiondependsonevaluatingthemarginalgainswithrespecttochangesinAandBwhilethecostofapplyingthetransactionreducestosettingabit.Itisalsoessentialthatonlyafewtransactionsfailattheclient.Indeed,theanalysisofthesesystemsfocusesonensuringthatthemajorityofthetransactionssucceed.3\fAlgorithm3:Ser-2g:serialdoublegreedy1A0=\u2205,B0=V2fori=1tondo3\u2206+(i)=F(Ai\u22121\u222ai)\u2212F(Ai\u22121)4\u2206\u2212(i)=F(Bi\u22121\\i)\u2212F(Bi\u22121)5Drawui\u223cUnif(0,1)6ifui<[\u2206+(i)]+[\u2206+(i)]++[\u2206\u2212(i)]+then7Ai:=Ai\u22121\u222ai;Bi:=Bi\u221218elseAi:=Ai\u22121;Bi:=Bi\u22121\\iAlgorithm4:CF-2g:coord-freedoublegreedy1bA=\u2205,bB=V2forp\u2208{1,...,P}doinparallel3while\u2203elementtoprocessdo4e=nextelementtoprocess5bAe=bA;bBe=bB6\u2206max+(e)=F(bAe\u222ae)\u2212F(bAe)7\u2206max\u2212(e)=F(bBe\\e)\u2212F(bBe)8Drawue\u223cUnif(0,1)9ifue<[\u2206max+(e)]+[\u2206max+(e)]++[\u2206max\u2212(e)]+then10bA(e)\u2190111elsebB(e)\u21900Algorithm5:CC-2g:concurrencycontrol1bA=eA=\u2205,bB=eB=V2fori=1,...,|V|doprocessed(i)=false3\u03b9=04forp\u2208{1,...,P}doinparallel5while\u2203elementtoprocessdo6e=nextelementtoprocess7(bAe,eAe,bBe,eBe,i)=getGuarantee(e)8(result,ue)=propose(e,bAe,eAe,bBe,eBe)9commit(e,i,ue,result)Algorithm6:CC-2ggetGuarantee(e)1eA(e)\u21901;eB(e)\u219002i=\u03b9;\u03b9\u2190\u03b9+13bAe=bA;bBe=bB4eAe=eA;eBe=eB5return(bAe,eAe,bBe,eBe,i)Algorithm7:CC-2gpropose1\u2206min+(e)=F(eAe)\u2212F(eAe\\e)2\u2206max+(e)=F(bAe\u222ae)\u2212F(bAe)3\u2206min\u2212(e)=F(eBe)\u2212F(eBe\u222ae)4\u2206max\u2212(e)=F(bBe\\e)\u2212F(bBe)5Drawue\u223cUnif(0,1)6ifue<[\u2206min+(e)]+[\u2206min+(e)]++[\u2206max\u2212(e)]+then7result\u219018elseifue>[\u2206max+(e)]+[\u2206max+(e)]++[\u2206min\u2212(e)]+then9result\u2190\u2212110elseresult\u2190FAIL11return(result,ue)Algorithm8:CC-2g:commit(e,i,ue,result)1waituntil\u2200j<i,processed(j)=true2ifresult=FAILthen3\u2206exact+(e)=F(bA\u222ae)\u2212F(bA)4\u2206exact\u2212(e)=F(bB\\e)\u2212F(bB)5ifue<[\u2206exact+(e)]+[\u2206exact+(e)]++[\u2206exact\u2212(e)]+thenresult\u219016elseresult\u2190\u221217ifresult=1thenbA(e)\u21901;eB(e)\u219018elseeA(e)\u21900;bB(e)\u219009processed(i)=true4Coordination-FreeDoubleGreedyAlgorithmThecoordination-freeapproachattemptstoreducetheneedtocoordinateguaranteesandthelogicalordering.Thisisachievedbyoperatingonpotentiallystalestates:thetransactionguaranteereducestorequiringgebeastaleversionofS,andthelogicalorderingisimplicitlyde\ufb01nedbythetimeofcommit.Inusingtheseweakguarantees,CF-2gisoverlyoptimisticallyassumingthatconcurrenttransactionsareindependent,whichcouldpotentiallyleadtoerroneousdecisions.Alg.4isthecoordination-freeparalleldoublegreedyalgorithm.1CF-2gcloselyresemblestheserialSer-2g,buttheelementse\u2208Varenolongerprocessedina\ufb01xedorder.Thus,thesetsA,Barereplacedbypotentiallystalelocalestimates(bounds)bA,bB,wherebAisasubsetofthetrueAandbBisasupersetoftheactualBoneachiteration.Theseboundingsetsallowustocomputebounds\u2206max+,\u2206max\u2212whichapproximate\u2206+,\u2206\u2212fromtheserialalgorithm.Wenowformalizethisidea.ToanalyzetheCF-2galgorithmweordertheelementse\u2208Vaccordingtothecommittime(i.e.,whenAlg.4line8isexecuted).Let\u03b9(e)bethepositionofeinthistotalorderingonelements.This1Wepresentonlytheparallelizedprobabilisticversionsof[2].Bothparallelalgorithmscanbeeasilyextendedtothedeterministicversionof[2];CF-2gcanalsobeextendedtothemultilinearversionof[2].4\fui Add A!Rem. B!0 1 162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215Algorithm3:Seq-2g:Sequentialdoublegreedy1A0=;,B0=V2fori=1tondo3+(i)=F(Ai1[i)F(Ai1)4(i)=F(Bi1\\i)F(Bi1)5Drawui\u21e0Unif(0,1)6ifui<[+(i)]+[+(i)]++[(i)]+then7Ai:=Ai1[i;Bi:=Bi18elseAi:=Ai1;Bi:=Bi1\\iAlgorithm4:CF-2g:coord-freedoublegreedy1\u02c6A=;,\u02c6B=V2forp2{1,...,P}doinparallel3while9elementtoprocessdo4e=nextelementtoprocess5max+(e)=F(\u02c6A[e)F(\u02c6A)6max(e)=F(\u02c6B\\e)F(\u02c6B)7Drawue\u21e0Unif(0,1)8ifue<[max+(e)]+[max+(e)]++[max(e)]+then9\u02c6A(e) 110else\u02c6B(e) 0Algorithm5:CC-2g:concurrencycontrol1\u02c6A=\u02dcA=;,\u02c6B=\u02dcB=V2fori=1,...,|V|doprocessed(i)=false3\u25c6=04forp2{1,...,P}doinparallel5while9elementtoprocessdo6e=nextelementtoprocess7(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)=getGuarantee(e)8(result,ue)=propose(e,\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe)9commit(e,i,ue,result)Algorithm6:CC-2ggetGuarantee(e)1\u02dcA(e) 1;\u02dcB(e) 02i=\u25c6;\u25c6 \u25c6+13\u02c6Ae=\u02c6A;\u02c6Be=\u02c6B4\u02dcAe=\u02dcA;\u02dcBe=\u02dcB5return(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)Algorithm7:CC-2gpropose1min+(e)=F(\u02dcAe)F(\u02dcAe\\e)2max+(e)=F(\u02c6Ae[e)F(\u02c6Ae)3min(e)=F(\u02dcBe)F(\u02dcBe[e)4max(e)=F(\u02c6Be\\e)F(\u02c6Be)5Drawue\u21e0Unif(0,1)6ifue<[min+(e)]+[min+(e)]++[max(e)]+then7result 18elseifue>[max+(e)]+[max+(e)]++[min(e)]+then9result 110elseresult fail11return(result,ue)Algorithm8:CC-2g:commit(e,i,ue,result)1waituntil8j<i,processed(j)=true2ifresult=failthen3exact+(e)=F(\u02c6A[e)F(\u02c6A)4exact(e)=F(\u02c6B\\e)F(\u02c6B)5ifue<[exact+(e)]+[exact+(e)]++[exact(e)]+thenresult 16elseresult 17ifresult=1then\u02c6A(e) 1;\u02dcB(e) 18else\u02dcA(e) 0;\u02c6B(e) 09processed(i)=true(a)(b)(c)4CoordinationFreeDoubleGreedyAlgorithmThecoordination-freeapproachattemptstoreducetheneedtocoordinateguaranteesandlogicalordering.Thisisachievedbyoperatingonpotentiallystalestates\u2013theguaranteereducestorequiringgebeastaleversionofS,andlogicalorderingisimplicitlyde\ufb01nedbythetimeofcommit.Inusingtheseweakguarantees,CF-2gisoverlyoptimisticallyassumingthatconcurrenttransactionsareindependent,whichcouldpotentiallyleadtoerroneousdecisions.Alg.4isthecoordinationfreeparalleldoublegreedyalgorithm.1CF-2gcloselyresemblestheserialSeq-2g,buttheelementse2Varenolongerprocessedina\ufb01xedorder.Thus,thesetsA,Barereplacedbypotentiallystale\u201cbounds\u201d\u02c6A,\u02c6B,where\u02c6Aisasubsetofthe\u201ctrue\u201dAand\u02c6Bisasuperset1Wepresentonlytheparallelizedprobabilisticversionsof[1].Bothparallelalgorithmscanbeeasilyextendedtothedeterministicversionof[1];CF-2gcanalsobeextendedtothemultilinearversionof[1].4(a)Ser-2gue Add A!Rem. B!0 1 162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215Algorithm3:Seq-2g:Sequentialdoublegreedy1A0=;,B0=V2fori=1tondo3+(i)=F(Ai1[i)F(Ai1)4(i)=F(Bi1\\i)F(Bi1)5Drawui\u21e0Unif(0,1)6ifui<[+(i)]+[+(i)]++[(i)]+then7Ai:=Ai1[i;Bi:=Bi18elseAi:=Ai1;Bi:=Bi1\\iAlgorithm4:CF-2g:coord-freedoublegreedy1\u02c6A=;,\u02c6B=V2forp2{1,...,P}doinparallel3while9elementtoprocessdo4e=nextelementtoprocess5max+(e)=F(\u02c6A[e)F(\u02c6A)6max(e)=F(\u02c6B\\e)F(\u02c6B)7Drawue\u21e0Unif(0,1)8ifue<[max+(e)]+[max+(e)]++[max(e)]+then9\u02c6A(e) 110else\u02c6B(e) 0Algorithm5:CC-2g:concurrencycontrol1\u02c6A=\u02dcA=;,\u02c6B=\u02dcB=V2fori=1,...,|V|doprocessed(i)=false3\u25c6=04forp2{1,...,P}doinparallel5while9elementtoprocessdo6e=nextelementtoprocess7(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)=getGuarantee(e)8(result,ue)=propose(e,\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe)9commit(e,i,ue,result)Algorithm6:CC-2ggetGuarantee(e)1\u02dcA(e) 1;\u02dcB(e) 02i=\u25c6;\u25c6 \u25c6+13\u02c6Ae=\u02c6A;\u02c6Be=\u02c6B4\u02dcAe=\u02dcA;\u02dcBe=\u02dcB5return(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)Algorithm7:CC-2gpropose1min+(e)=F(\u02dcAe)F(\u02dcAe\\e)2max+(e)=F(\u02c6Ae[e)F(\u02c6Ae)3min(e)=F(\u02dcBe)F(\u02dcBe[e)4max(e)=F(\u02c6Be\\e)F(\u02c6Be)5Drawue\u21e0Unif(0,1)6ifue<[min+(e)]+[min+(e)]++[max(e)]+then7result 18elseifue>[max+(e)]+[max+(e)]++[min(e)]+then9result 110elseresult fail11return(result,ue)Algorithm8:CC-2g:commit(e,i,ue,result)1waituntil8j<i,processed(j)=true2ifresult=failthen3exact+(e)=F(\u02c6A[e)F(\u02c6A)4exact(e)=F(\u02c6B\\e)F(\u02c6B)5ifue<[exact+(e)]+[exact+(e)]++[exact(e)]+thenresult 16elseresult 17ifresult=1then\u02c6A(e) 1;\u02dcB(e) 18else\u02dcA(e) 0;\u02c6B(e) 09processed(i)=true(a)(b)(c)4CoordinationFreeDoubleGreedyAlgorithmThecoordination-freeapproachattemptstoreducetheneedtocoordinateguaranteesandlogicalordering.Thisisachievedbyoperatingonpotentiallystalestates\u2013theguaranteereducestorequiringgebeastaleversionofS,andlogicalorderingisimplicitlyde\ufb01nedbythetimeofcommit.Inusingtheseweakguarantees,CF-2gisoverlyoptimisticallyassumingthatconcurrenttransactionsareindependent,whichcouldpotentiallyleadtoerroneousdecisions.Alg.4isthecoordinationfreeparalleldoublegreedyalgorithm.1CF-2gcloselyresemblestheserialSeq-2g,buttheelementse2Varenolongerprocessedina\ufb01xedorder.Thus,thesetsA,Barereplacedbypotentiallystale\u201cbounds\u201d\u02c6A,\u02c6B,where\u02c6Aisasubsetofthe\u201ctrue\u201dAand\u02c6Bisasuperset1Wepresentonlytheparallelizedprobabilisticversionsof[1].Bothparallelalgorithmscanbeeasilyextendedtothedeterministicversionof[1];CF-2gcanalsobeextendedtothemultilinearversionof[1].4162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215Algorithm3:Ser-2g:serialdoublegreedy1A0=;,B0=V2fori=1tondo3+(i)=F(Ai1[i)F(Ai1)4(i)=F(Bi1\\i)F(Bi1)5Drawui\u21e0Unif(0,1)6ifui<[+(e)]+[+(e)]++[(e)]+then78Ai:=Ai1[i;Bi:=Bi19elseAi:=Ai1;Bi:=Bi1\\iAlgorithm4:CF-2g:coord-freedoublegreedy1\u02c6A=;,\u02c6B=V2forp2{1,...,P}doinparallel3while9elementtoprocessdo4e=nextelementtoprocess5\u02c6Ae=\u02c6A;\u02c6Be=\u02c6B6max+(e)=F(\u02c6Ae[e)F(\u02c6Ae)7max(e)=F(\u02c6Be\\e)F(\u02c6Be)8Drawue\u21e0Unif(0,1)9ifue<[max+(e)]+[max+(e)]++[max(e)]+then10\u02c6A(e) 111else\u02c6B(e) 0Algorithm5:CC-2g:concurrencycontrol1\u02c6A=\u02dcA=;,\u02c6B=\u02dcB=V2fori=1,...,|V|doprocessed(i)=false3\u25c6=04forp2{1,...,P}doinparallel5while9elementtoprocessdo6e=nextelementtoprocess7(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)=getGuarantee(e)8(result,ue)=propose(e,\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe)9commit(e,i,ue,result)Algorithm6:CC-2ggetGuarantee(e)1\u02dcA(e) 1;\u02dcB(e) 02i=\u25c6;\u25c6 \u25c6+13\u02c6Ae=\u02c6A;\u02c6Be=\u02c6B4\u02dcAe=\u02dcA;\u02dcBe=\u02dcB5return(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)Algorithm7:CC-2gpropose1min+(e)=F(\u02dcAe)F(\u02dcAe\\e)2max+(e)=F(\u02c6Ae[e)F(\u02c6Ae)3min(e)=F(\u02dcBe)F(\u02dcBe[e)4max(e)=F(\u02c6Be\\e)F(\u02c6Be)5Drawue\u21e0Unif(0,1)6ifue<[min+(e)]+[min+(e)]++[max(e)]+then7result 18elseifue>[max+(e)]+[max+(e)]++[min(e)]+then9result 110elseresult FAIL11return(result,ue)Algorithm8:CC-2g:commit(e,i,ue,result)1waituntil8j<i,processed(j)=true2ifresult=FAILthen3exact+(e)=F(\u02c6A[e)F(\u02c6A)4exact(e)=F(\u02c6B\\e)F(\u02c6B)5ifue<[exact+(e)]+[exact+(e)]++[exact(e)]+thenresult 16elseresult 17ifresult=1then\u02c6A(e) 1;\u02dcB(e) 18else\u02dcA(e) 0;\u02c6B(e) 09processed(i)=true4Coordination-FreeDoubleGreedyAlgorithmThecoordination-freeapproachattemptstoreducetheneedtocoordinateguaranteesandthelogicalordering.Thisisachievedbyoperatingonpotentiallystalestates:thetransactionguaranteereducestorequiringgebeastaleversionofS,andthelogicalorderingisimplicitlyde\ufb01nedbythetimeofcommit.Inusingtheseweakguarantees,CF-2gisoverlyoptimisticallyassumingthatconcurrenttransactionsareindependent,whichcouldpotentiallyleadtoerroneousdecisions.Alg.4isthecoordination-freeparalleldoublegreedyalgorithm.1CF-2gcloselyresemblestheserialSer-2g,buttheelementse2Varenolongerprocessedina\ufb01xedorder.Thus,thesetsA,Barereplacedbypotentiallystalelocalestimates(bounds)\u02c6A,\u02c6B,where\u02c6AisasubsetofthetrueAand\u02c6BisasupersetoftheactualBoneachiteration.Theseboundingsetsallowustocomputeboundsmax+,maxwhichapproximate+,fromtheserialalgorithm.Wenowformalizethisidea.1Wepresentonlytheparallelizedprobabilisticversionsof[2].Bothparallelalgorithmscanbeeasilyextendedtothedeterministicversionof[2];CF-2gcanalsobeextendedtothemultilinearversionof[2].4(b)CF-2gue Add A!Rem. B!0 1 Uncertainty !162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215Algorithm3:Seq-2g:Sequentialdoublegreedy1A0=;,B0=V2fori=1tondo3+(i)=F(Ai1[i)F(Ai1)4(i)=F(Bi1\\i)F(Bi1)5Drawui\u21e0Unif(0,1)6ifui<[+(i)]+[+(i)]++[(i)]+then7Ai:=Ai1[i;Bi:=Bi18elseAi:=Ai1;Bi:=Bi1\\iAlgorithm4:CF-2g:coord-freedoublegreedy1\u02c6A=;,\u02c6B=V2forp2{1,...,P}doinparallel3while9elementtoprocessdo4e=nextelementtoprocess5max+(e)=F(\u02c6A[e)F(\u02c6A)6max(e)=F(\u02c6B\\e)F(\u02c6B)7Drawue\u21e0Unif(0,1)8ifue<[max+(e)]+[max+(e)]++[max(e)]+then9\u02c6A(e) 110else\u02c6B(e) 0Algorithm5:CC-2g:concurrencycontrol1\u02c6A=\u02dcA=;,\u02c6B=\u02dcB=V2fori=1,...,|V|doprocessed(i)=false3\u25c6=04forp2{1,...,P}doinparallel5while9elementtoprocessdo6e=nextelementtoprocess7(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)=getGuarantee(e)8(result,ue)=propose(e,\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe)9commit(e,i,ue,result)Algorithm6:CC-2ggetGuarantee(e)1\u02dcA(e) 1;\u02dcB(e) 02i=\u25c6;\u25c6 \u25c6+13\u02c6Ae=\u02c6A;\u02c6Be=\u02c6B4\u02dcAe=\u02dcA;\u02dcBe=\u02dcB5return(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)Algorithm7:CC-2gpropose1min+(e)=F(\u02dcAe)F(\u02dcAe\\e)2max+(e)=F(\u02c6Ae[e)F(\u02c6Ae)3min(e)=F(\u02dcBe)F(\u02dcBe[e)4max(e)=F(\u02c6Be\\e)F(\u02c6Be)5Drawue\u21e0Unif(0,1)6ifue<[min+(e)]+[min+(e)]++[max(e)]+then7result 18elseifue>[max+(e)]+[max+(e)]++[min(e)]+then9result 110elseresult fail11return(result,ue)Algorithm8:CC-2g:commit(e,i,ue,result)1waituntil8j<i,processed(j)=true2ifresult=failthen3exact+(e)=F(\u02c6A[e)F(\u02c6A)4exact(e)=F(\u02c6B\\e)F(\u02c6B)5ifue<[exact+(e)]+[exact+(e)]++[exact(e)]+thenresult 16elseresult 17ifresult=1then\u02c6A(e) 1;\u02dcB(e) 18else\u02dcA(e) 0;\u02c6B(e) 09processed(i)=true(a)(b)(c)4CoordinationFreeDoubleGreedyAlgorithmThecoordination-freeapproachattemptstoreducetheneedtocoordinateguaranteesandlogicalordering.Thisisachievedbyoperatingonpotentiallystalestates\u2013theguaranteereducestorequiringgebeastaleversionofS,andlogicalorderingisimplicitlyde\ufb01nedbythetimeofcommit.Inusingtheseweakguarantees,CF-2gisoverlyoptimisticallyassumingthatconcurrenttransactionsareindependent,whichcouldpotentiallyleadtoerroneousdecisions.Alg.4isthecoordinationfreeparalleldoublegreedyalgorithm.1CF-2gcloselyresemblestheserialSeq-2g,buttheelementse2Varenolongerprocessedina\ufb01xedorder.Thus,thesetsA,Barereplacedbypotentiallystale\u201cbounds\u201d\u02c6A,\u02c6B,where\u02c6Aisasubsetofthe\u201ctrue\u201dAand\u02c6Bisasuperset1Wepresentonlytheparallelizedprobabilisticversionsof[1].Bothparallelalgorithmscanbeeasilyextendedtothedeterministicversionof[1];CF-2gcanalsobeextendedtothemultilinearversionof[1].4162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215Algorithm3:Seq-2g:Sequentialdoublegreedy1A0=;,B0=V2fori=1tondo3+(i)=F(Ai1[i)F(Ai1)4(i)=F(Bi1\\i)F(Bi1)5Drawui\u21e0Unif(0,1)6ifui<[+(i)]+[+(i)]++[(i)]+then7Ai:=Ai1[i;Bi:=Bi18elseAi:=Ai1;Bi:=Bi1\\iAlgorithm4:CF-2g:coord-freedoublegreedy1\u02c6A=;,\u02c6B=V2forp2{1,...,P}doinparallel3while9elementtoprocessdo4e=nextelementtoprocess5max+(e)=F(\u02c6A[e)F(\u02c6A)6max(e)=F(\u02c6B\\e)F(\u02c6B)7Drawue\u21e0Unif(0,1)8ifue<[max+(e)]+[max+(e)]++[max(e)]+then9\u02c6A(e) 110else\u02c6B(e) 0Algorithm5:CC-2g:concurrencycontrol1\u02c6A=\u02dcA=;,\u02c6B=\u02dcB=V2fori=1,...,|V|doprocessed(i)=false3\u25c6=04forp2{1,...,P}doinparallel5while9elementtoprocessdo6e=nextelementtoprocess7(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)=getGuarantee(e)8(result,ue)=propose(e,\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe)9commit(e,i,ue,result)Algorithm6:CC-2ggetGuarantee(e)1\u02dcA(e) 1;\u02dcB(e) 02i=\u25c6;\u25c6 \u25c6+13\u02c6Ae=\u02c6A;\u02c6Be=\u02c6B4\u02dcAe=\u02dcA;\u02dcBe=\u02dcB5return(\u02c6Ae,\u02dcAe,\u02c6Be,\u02dcBe,i)Algorithm7:CC-2gpropose1min+(e)=F(\u02dcAe)F(\u02dcAe\\e)2max+(e)=F(\u02c6Ae[e)F(\u02c6Ae)3min(e)=F(\u02dcBe)F(\u02dcBe[e)4max(e)=F(\u02c6Be\\e)F(\u02c6Be)5Drawue\u21e0Unif(0,1)6ifue<[min+(e)]+[min+(e)]++[max(e)]+then7result 18elseifue>[max+(e)]+[max+(e)]++[min(e)]+then9result 110elseresult fail11return(result,ue)Algorithm8:CC-2g:commit(e,i,ue,result)1waituntil8j<i,processed(j)=true2ifresult=failthen3exact+(e)=F(\u02c6A[e)F(\u02c6A)4exact(e)=F(\u02c6B\\e)F(\u02c6B)5ifue<[exact+(e)]+[exact+(e)]++[exact(e)]+thenresult 16elseresult 17ifresult=1then\u02c6A(e) 1;\u02dcB(e) 18else\u02dcA(e) 0;\u02c6B(e) 09processed(i)=true(a)(b)(c)4CoordinationFreeDoubleGreedyAlgorithmThecoordination-freeapproachattemptstoreducetheneedtocoordinateguaranteesandlogicalordering.Thisisachievedbyoperatingonpotentiallystalestates\u2013theguaranteereducestorequiringgebeastaleversionofS,andlogicalorderingisimplicitlyde\ufb01nedbythetimeofcommit.Inusingtheseweakguarantees,CF-2gisoverlyoptimisticallyassumingthatconcurrenttransactionsareindependent,whichcouldpotentiallyleadtoerroneousdecisions.Alg.4isthecoordinationfreeparalleldoublegreedyalgorithm.1CF-2gcloselyresemblestheserialSeq-2g,buttheelementse2Varenolongerprocessedina\ufb01xedorder.Thus,thesetsA,Barereplacedbypotentiallystale\u201cbounds\u201d\u02c6A,\u02c6B,where\u02c6Aisasubsetofthe\u201ctrue\u201dAand\u02c6Bisasuperset1Wepresentonlytheparallelizedprobabilisticversionsof[1].Bothparallelalgorithmscanbeeasilyextendedtothedeterministicversionof[1];CF-2gcanalsobeextendedtothemultilinearversionof[1].4(c)CC-2gFigure2:Illustrationofalgorithms.(a)Ser-2gcomputesathresholdbasedonthetruevalues\u2206+,\u2206\u2212,andchoosesanactionbasedbycomparingauniformrandomuiagainstthethreshold.(b)CF-2gapproximatesthethresholdbasedonstalebA,bB,possiblychoosingthewrongaction.(c)CC-2gcomputestwothresholdsbasedontheboundsonA,B,whichde\ufb01nesanuncertaintyregionwhereitisnotpossibletochoosethecorrectactionlocally.IftherandomvalueuefallsinsidetheuncertaintyintervalthanthetransactionFAILSandmustberecomputedseriallybytheserver;otherwisethetransactionholdsunderallpossibleglobalstates.orderingallowsustode\ufb01nemonotonicallynon-decreasingsetsAi={e0:e0\u2208A,\u03b9(e0)<i}whereAisthe\ufb01nalreturnedset,andmonotonicallynon-increasingsetsBi=Ai\u222a{e0:\u03b9(e0)\u2265i}.ThesetsAi,BiprovideaserializationagainstwhichwecancompareCF-2g;inthisserialization,Alg.3computes\u2206+(e)=F(A\u03b9(e)\u22121\u222ae)\u2212F(A\u03b9(e)\u22121)and\u2206\u2212(e)=F(B\u03b9(e)\u22121\\e)\u2212F(B\u03b9(e)\u22121).Ontheotherhand,CF-2gusesstaleversions2bAe,bBe:Alg.4computes\u2206max+(e)=F(bAe\u222ae)\u2212F(bAe)and\u2206max\u2212(e)=F(bBe\\e)\u2212F(bBe).ThenextlemmashowsthatbAe,bBeareboundingsetsfortheserialization\u2019ssetsA\u03b9(e)\u22121,B\u03b9(e)\u22121.Intuitively,theboundsholdbecausebAe,bBearestaleversionsofA\u03b9(e)\u22121,B\u03b9(e)\u22121,whicharemonotonicallynon-decreasingandnon-increasingsets.AppendixAgivesadetailedproof.Lemma4.1.InCF-2g,foranye\u2208V,bAe\u2286A\u03b9(e)\u22121,andbBe\u2287B\u03b9(e)\u22121.Corollary4.2.SubmodularityofFimpliesforCF-2g\u2206+(e)\u2264\u2206max+(e),and\u2206\u2212(e)\u2264\u2206max\u2212(e).TheerrorinCF-2gdependsonthetightnessoftheboundsinCor.4.2.WeanalyzethisinSec.6.1.5ConcurrencyControlfortheDoubleGreedyAlgorithmTheconcurrencycontrol-baseddoublegreedyalgorithm1,CC-2g,ispresentedinAlg.5,andcloselyfollowsthemeta-algorithmofAlg.1andAlg.2.UnlikeinCF-2g,theconcurrencycontrolmecha-nismsofCC-2gensurethatconcurrenttransactionsareserializedwhentheyarenotindependent.SerializabilityisachievedbymaintainingsetsbA,eA,bB,eB,whichserveasupperandlowerboundsonthetruestateofAandBatcommittime.Eachthreadcandeterminelocallyifadecisiontoincludeorexcludeanelementcanbetakensafely.Otherwise,theproposalisdeferredtothecommitprocess(Alg.8)whichwaitsuntilitiscertainaboutAandBbeforeproceeding.Thecommitorderisgivenby\u03b9(e),whichisthevalueof\u03b9inline2ofAlg.5.Wede\ufb01neA\u03b9(e)\u22121,B\u03b9(e)\u22121asbeforewithCF-2g.Additionally,letbAe,bBe,eAe,andeBebethesetsthatarereturnedbyAlg.6.2Indeed,thesesetsareguaranteedtobeboundsonA\u03b9(e)\u22121,B\u03b9(e)\u22121:Lemma5.1.InCC-2g,\u2200e\u2208V,bAe\u2286A\u03b9(e)\u22121\u2286eAe\\e,andbBe\u2287B\u03b9(e)\u22121\u2287eBe\u222ae.Intuitively,theseboundsaremaintainedbyrecordingpotentialeffectsofconcurrenttransactionsineA,eB,andonlyrecordingtheactualeffectsinbA,bB;weleavethefullprooftoAppendixA.Furthermore,bycommittingtransactionsinorder\u03b9,wehavebA=A\u03b9(e)\u22121andbB=B\u03b9(e)\u22121duringcommit.Lemma5.2.InCC-2g,whencommittingelemente,wehavebA=A\u03b9(e)\u22121andbB=B\u03b9(e)\u22121.2Forclarity,wepresentthealgorithmascreatingacopyofbA,bB,eA,andeBforeachelement.Inpractice,itismoreef\ufb01cienttoupdateandaccesstheminsharedmemory.Nevertheless,ourtheoremsholdforbothsettings.5\fCorollary5.3.SubmodularityofFimpliesthatthe\u2206\u2019scomputedbyCC-2gsatisfy\u2206min+(e)\u2264\u2206exact+(e)=\u2206+(e)\u2264\u2206max+(e)and\u2206min\u2212(e)\u2264\u2206exact\u2212(e)=\u2206\u2212(e)\u2264\u2206max\u2212(e).Byusingthesebounds,CC-2gcandeterminewhenitissafetoconstructthetransactionlocally.Forfailedtransactions,theserverisabletoconstructthecorrecttransactionusingthetrueprogramstate.AsaconsequencewecanguaranteethattheparallelexecutionofCC-2gisserializable.6AnalysisofAlgorithmsOurtwoalgorithmstradeoffperformanceandstrongapproximationguarantees.TheCF-2galgo-rithmemphasizesspeedattheexpenseoftheapproximationobjective.Ontheotherhand,CC-2gemphasizesthetight1/2-approximationattheexpenseofincreasedcoordination.Inthissectionwecharacterizethereductionintheapproximationobjectiveaswellastheincreasedcoordina-tion.OuranalysisconnectsthedegradationinCC-2gscalabilitywiththedegradationintheCF-2gapproximationfactorviathemaximuminter-processormessagedelay\u03c4.6.1ApproximationofCF-2gdoublegreedyTheorem6.1.LetFbeanon-negativesubmodularfunction.CF-2gsolvestheunconstrainedproblemmaxA\u2282VF(A)withworst-caseapproximationfactorE[F(ACF)]\u226512F\u2217\u221214PNi=1E[\u03c1i],whereACFistheoutputofthealgorithm,F\u2217istheoptimalvalue,and\u03c1i=max{\u2206max+(e)\u2212\u2206+(e),\u2206max\u2212(e)\u2212\u2206\u2212(e)}isthemaximumdiscrepancyinthemarginalgainduetothebounds.Theproof(AppendixC)ofThm.6.1followsthestructurein[2].Thm.6.1capturesthedeviationfromoptimalityasafunctionofwidthoftheboundswhichwecharacterizefortwocommonapplications.Example:maxgraphcut.Forthemaxcutobjectiveweboundtheexpecteddiscrepancyinthemarginalgain\u03c1iintermsofthesparsityofthegraphandthemaximuminter-processormessagedelay\u03c4.ByapplyingThm.6.1weobtaintheapproximationfactorE[F(AN)]\u226512F\u2217\u2212\u03c4#edges2Nwhichdecreaseslinearlyinboththemessagedelaysandgraphdensity.Inacompletegraph,F\u2217=12#edges,soE[F(AN)]\u2265F\u2217(cid:0)12\u2212\u03c4N(cid:1),whichmakesitpossibletoscale\u03c4linearlywithNwhileretainingthesameapproximationfactor.Example:setcover.Considerthesimplesetcoverfunction,F(A)=PLl=1min(1,|A\u2229Sl|)\u2212\u03bb|A|=|{l:A\u2229Sl6=\u2205}|\u2212\u03bb|A|,with0<\u03bb\u22641.Weassumethatthereissomeboundeddelay\u03c4.SupposealsotheSl\u2019sformapartition,soeachelementebelongstoexactlyoneset.Then,PeE[\u03c1e]\u2265\u03c4+L(1\u2212\u03bb\u03c4),whichislinearin\u03c4butindependentofN.6.2CorrectnessofCC-2gTheorem6.2.CC-2gisserializableandthereforesolvestheunconstrainedsubmodularmaximizationproblemmaxA\u2282VF(A)withapproximationE[F(ACC)]\u226512F\u2217,whereACCistheoutputofthealgorithm,andF\u2217istheoptimalvalue.Thekeychallengeintheproof(AppendixB)ofThm.6.2istodemonstratethatCC-2gguaranteesaserializableexecution.Itsuf\ufb01cestoshowthatCC-2gtakesthesamedecisionasSer-2gforeachelement\u2013locallyifitissafetodoso,andotherwisedeferringthecomputationtotheserver.Asanimmediateconsequenceofserializability,werecovertheoptimalapproximationguaranteesoftheserialSer-2galgorithm.6.3ScalabilityofCC-2gWheneveratransactionisreconstructedontheserver,theserverneedstowaitforallearlierelementstobecommitted,andisalsoblockedfromcommittingalllaterelements.Eachfailedtransactioneffectivelyconstitutesabarriertotheparallelprocessing.Hence,thescalabilityofCC-2gisdependentonthenumberoffailedtransactions.Wecandirectlyboundthenumberoffailedtransactions(detailsinAppendixD)forboththemax-cutandsetcoverexampleproblems.Forthemax-cutproblemwithamaximuminter-processormessage6\fdelay\u03c4weobtaintheupperbound2\u03c4#edgesN.Similarlyforsetcovertheexpectednumberoffailedtransactionsisupper-boundedby2\u03c4.Asaconsequence,thecoordinationcostsofCC-2ggrowsatthesamerateasthereductioninaccuracyofCF-2g.Moreover,theCC-2galgorithmwillslowdowninsettingswheretheCF-2galgorithmproducessub-optimalsolutions.7EvaluationWeimplementedtheparallelandserialdoublegreedyalgorithmsinJava/Scala.ExperimentswereconductedonAmazonEC2usingonecc2.8xlargemachine,upto16threads,for10repetitions.Wemeasuredtheruntimeandspeedup(ratioofruntimeon1threadtoruntimeonpthreads).ForCF-2g,wemeasuredF(ACF)\u2212F(ASer),thedifferencebetweentheobjectivevalueonthesetsreturnedbyCF-2gandSer-2g.Weveri\ufb01edthecorrectnessofCC-2gbycomparingtheoutputofCC-2gwithSer-2g.WealsomeasuredthefractionoftransactionsthatfailinCC-2g.Ourparallelalgorithmsweretestedonthemaxgraphcutandsetcoverproblemswithtwosyntheticgraphsandthreerealdatasets(Table1).Wefoundthatverticesweretypicallyindexedsuchthatnearbyverticesinthegraphwerealsocloseintheirindices.Toreducethisdependency,werandomlypermutedtheorderingofvertices.Graph#vertices#edgesDescriptionErdos-Renyi20,000,000\u22482\u00d7109Eachedgeisincludedwithprobability5\u00d710\u22126.ZigZag25,000,0002,025,000,000Expandergraph.The81-regularzig-zagproductbetweentheCayleygraphonZ2500000withgeneratingset{\u00b11,...,\u00b15},andthecompletegraphK10.Friendster10,000,000625,279,786Subgraphofsocialnetwork.[21]Arabic-200522,744,080631,153,6692005crawlofArabicwebsites[22,23,24].UK-200539,459,925921,345,0782005crawlofthe.ukdomain[22,23,24].IT-200441,291,5941,135,718,9092004crawlofthe.itdomain[22,23,24].Table1:Syntheticandrealgraphsusedintheevaluationofourparallelalgorithms.05101500.511.522.53# threadsRuntime relative to sequentialRuntime, relative to sequential Ser\u22122gCC\u22122gCF\u22122g(a)051015051015# threadsSpeedupSpeedup for Max Graph Cut IdealCC\u22122g, IT\u22122004CF\u22122g, IT\u22122004CC\u22122g, ZigZagCF\u22122g, ZigZag(b)051015051015# threadsSpeedupSpeedup for Set Cover IdealCC\u22122g, IT\u22122004CF\u22122g, IT\u22122004CC\u22122g, ZigZagCF\u22122g, ZigZag(c)051015\u2212101234x 10\u22123# threads% decrease in F(A)CF\u22122g % Decrease in F(A)Max Graph Cut FriendsterArabic\u22122005UK\u22122005IT\u22122004ZigZagErdos\u2212Renyi(d)05101501234x 10\u22124# threads% decrease in F(A)CF\u22122g % Decrease in F(A)Set Cover FriendsterArabic\u22122005UK\u22122005IT\u22122004ZigZagErdos\u2212Renyi(e)05101500.0050.010.015# threads% failed txnsCC\u22122g % Failed TxnsMax Graph Cut FriendsterArabic\u22122005UK\u22122005IT\u22122004ZigZagErdos\u2212Renyi(f)Figure3:Experimentalresults.Fig.3a\u2013runtimeoftheparallelalgorithmsasaratiotothatoftheserialalgorithm.EachcurveshowstheruntimeofaparallelalgorithmonaparticulargraphforaparticularfunctionF.Fig.3b,3c\u2013speedup(ratioofruntimeononethreadtothatonpthreads).Fig.3d,3e\u2013%differencebetweenobjectivevaluesofSer-2gandCF-2g,i.e.[F(ACF)/F(ASer)\u22121]\u00d7100%.Fig.3f\u2013percentageoftransactionsthatfailinCC-2gonthemaxgraphcutproblem.WesummarizeofthekeyresultsherewithmoredetailedexperimentsanddiscussioninAppendixG.Runtime,Speedup:Bothparallelalgorithmsarefasterthantheserialalgorithmwiththreeormorethreads,andshowgoodspeeduppropertiesasmorethreadsareadded(\u223c10xormoreforallgraphsandbothfunctions).Objectivevalue:TheobjectivevalueofCF-2gdecreaseswiththenumberofthreads,butdiffersfromtheserialobjectivevaluebylessthan0.01%.Failedtransactions:CC-2gfailsmoretransactionsasthreadsareadded,butevenwith16threads,lessthan0.015%transactionsfail,whichhasnegligibleeffectontheruntime/speedup.7\f051015050100150200250300Runtime on EC2: Ring Set CoverNumber of threadsRuntime / s Ser\u22122gCC\u22122gCF\u22122g(a)051015051015Speed\u2212up on EC2: Ring Set CoverNumber of threadsSpeed\u2212up factor IdealCC\u22122gCF\u22122g(b)05101500.20.40.60.81CC\u22122g: Fraction of txns failedNumber of threadsRing Set Cover 05101500.20.40.60.81CF\u22122g: Fraction of F(A) decreaseCC\u22122g: failed txnsCC2F: F(A) decrease(c)Figure4:Experimentalresultsforsetcoverproblemonaringexpandergraphdemonstratingthatforadversari-allyconstructedinputswecanreducetheoptimalityofCF-2gandincreasecoordinationcostsforCC-2g.7.1AdversarialorderingTohighlightthedifferencesinapproachesbetweenthetwoparallelalgorithms,weconductedexperimentsonaringCayleyexpandergraphonZ106withgeneratingset{\u00b11,...,\u00b11000}.Thealgorithmsarepresentedwithanadversarialordering,withoutpermutation,soverticescloseintheorderingareadjacenttooneanother,andtendtobeprocessedconcurrently.ThiscausesCF-2gtomakemoremistakes,andCC-2gtofailmoretransactions.Whilemoresophisticatedpartitioningschemescouldimprovescalabilityandeliminatetheeffectofadversarialordering,weusethedefaultdatapartitioninginourexperimentstohighlightthedifferencesbetweenthetwoalgorithms.AsFig.4shows,CC-2gsacri\ufb01cesspeedtoensureaserializableexecution,eventuallyfailingon>90%oftransactions.Ontheotherhand,CF-2gfocusesonspeed,resultinginfasterruntime,butachievesanobjectivevaluethatis20%ofF(ASer).WeemphasizethatwecontrivedthisexampletohighlightdifferencesbetweenCC-2gandCF-2g,andwedonotexpecttoseesuchorderingsinpractice.8RelatedWorkSimilarapproach:Coordination-freesolutionshavebeenproposedforstochasticgradientdescent[25]andcollapsedGibbssampling[26].Moregenerally,parameterservers[27,28]applytheCFapproachtolargerclassesofproblems.Panetal.[29]appliedconcurrencycontroltoparallelizesomeunsupervisedlearningalgorithms.Similarproblem:Distributedandparallelgreedysubmodularmaximizationisaddressedin[1,15,16],butonlyformonotonefunctions.9ConclusionandFutureWorkByadoptingthetransactionprocessingmodelfromparalleldatabasesystems,wepresentedtwoapproachestoparallelizingthedoublegreedyalgorithmforunconstrainedsubmodularmaximization.Wequanti\ufb01edtheweakerapproximationguaranteeofCF-2gandtheadditionalcoordinationofCC-2g,allowingonetotradeoffbetweenperformanceandobjectiveoptimality.Ourevaluationonlargescaledatademonstratesthescalabilityandtradeoffsofthetwoapproaches.Moreover,astheapproximationqualityoftheCF-2galgorithmdecreasessodoesthescalabilityoftheCC-2galgorithm.Thechoicebetweenthealgorithmthenreducestoachoiceofguaranteedperformanceandguaranteedoptimality.Webelievethereareanumberofareasforfuturework.OnecanimagineasystemthatallowsasmoothinterpolationbetweenCF-2gandCC-2g.WhilebothCF-2gandCC-2gcanbeimmediatelyimplementedasdistributedalgorithms,highercommunicationcostsanddelaysmayposeadditionalchallenges.Finally,otherproblemssuchasconstrainedmaximizationofmonotone/non-monotonefunctionscouldpotentiallybeparallelizedwiththeCFandCCframeworks.Acknowledgments.ThisresearchissupportedinpartbyNSFCISEExpeditionsAwardCCF-1139158,LBNLAward7076018,andDARPAXDataAwardFA8750-12-2-0331,andgiftsfromAmazonWebServices,Google,SAP,TheThomasandStaceySiebelFoundation,Adobe,Apple,Inc.,Bosch,C3Energy,Cisco,Cloudera,EMC,Ericsson,Facebook,GameOnTalis,Guavus,HP,Huawei,Intel,Microsoft,NetApp,Pivotal,Splunk,Virdata,VMware,andYahoo!.ThisresearchwasinpartfundedbytheOf\ufb01ceofNavalResearchundercontract/grantnumberN00014-11-1-0688.X.Pan\u2019sworkisalsosupportedbyaDSONationalLaboratoriesPostgraduateScholarship.8\fReferences[1]B.Mirzasoleiman,A.Karbasi,R.Sarkar,andA.Krause.Distributedsubmodularmaximization:Identifyingrepresentativeelementsinmassivedata.InAdvancesinNeuralInformationProcessingSystems26.2013.[2]N.Buchbinder,M.Feldman,J.Naor,andR.Schwartz.Atightlineartime(1/2)-approximationforunconstrainedsubmodularmaximization.InFOCS,2012.[3]A.KrauseandC.Guestrin.Submodularityanditsapplicationsinoptimizedinformationgathering:Anintroduction.ACMTransactionsonIntelligentSystemsandTechnology,2(4),2011.[4]G.Kim,E.P.Xing,F.Li,andT.Kanade.Distributedcosegmentationviasubmodularoptimizationonanisotropicdiffusion.InInt.ConferenceonComputerVision(ICCV),2011.[5]J.Gillenwater,A.Kulesza,andB.Taskar.Near-optimalMAPinferencefordeterminantalpointprocesses.InAdvancesinNeuralInformationProcessingSystems(NIPS),2012.[6]D.Kempe,J.Kleinberg,andE.Tardos.Maximizingthespreadofin\ufb02uencethroughasocialnetwork.InACMSIGKDDConferenceonKnowledgeDiscoveryandDataMining(KDD),2003.[7]H.LinandJ.Bilmes.Aclassofsubmodularfunctionsfordocumentsummarization.InThe49thAnnualMeetingoftheAssociationforComputationalLinguistics:HumanLanguageTechnologies,2011.[8]G.L.Nemhauser,L.A.Wolsey,andM.L.Fisher.Ananalysisofapproximationsformaximizingsubmodularsetfunctions\u2014I.MathematicalProgramming,14(1):265\u2013294,1978.[9]L.S.Shapley.Coresofconvexgames.InternationalJournalofGameTheory,1(1):11\u201326,1971.[10]A.Frank.Submodularfunctionsingraphtheory.DiscreteMathematics,111:231\u2013243,1993.[11]A.Schrijver.CombinatorialOptimization\u2013Polyhedraandef\ufb01ciency.Springer,2002.[12]A.KrauseandS.Jegelka.Submodularityinmachinelearning\u2013newdirections.ICMLTutorial,2013.[13]J.Bilmes.Deepmathematicalpropertiesofsubmodularitywithapplicationstomachinelearning.NIPSTutorial,2013.[14]A.BadanidiyuruandJ.Vondr\u00b4ak.Fastalgorithmsformaximizingsubmodularfunctions.InSODA,2014.[15]R.Kumar,B.Moseley,S.Vassilvitskii,andA.Vattani.FastgreedyalgorithmsinMapReduceandstreaming.InSPAA,2013.[16]K.Wei,R.Iyer,andJ.Bilmes.Fastmulti-stagesubmodularmaximization.InInt.ConferenceonMachineLearning(ICML),2014.[17]C.ReedandZ.Ghahramani.ScalingtheIndianBuffetProcessviasubmodularmaximization.InInt.ConferenceonMachineLearning(ICML),2013.[18]A.KrauseandE.Horvitz.Autility-theoreticapproachtoprivacyinonlineservices.JAIR,39,2010.[19]M.TamerOzsu.PrinciplesofDistributedDatabaseSystems.PrenticeHallPress,UpperSaddleRiver,NJ,USA,3rdedition,2007.ISBN9780130412126.[20]H.KungandJ.T.Robinson.Onoptimisticmethodsforconcurrencycontrol.TODS,6(2),1981.[21]J.Leskovec.Stanfordnetworkanalysisproject,2011.URLhttp://snap.stanford.edu/.[22]P.BoldiandS.Vigna.TheWebGraphframeworkI:Compressiontechniques.InWWW,2004.[23]P.Boldi,M.Rosa,M.Santini,andS.Vigna.Layeredlabelpropagation:Amultiresolutioncoordinate-freeorderingforcompressingsocialnetworks.InWWW.ACMPress,2011.[24]P.Boldi,B.Codenotti,M.Santini,andS.Vigna.Ubicrawler:Ascalablefullydistributedwebcrawler.Software:Practice&Experience,34(8):711\u2013726,2004.[25]B.Recht,C.Re,S.J.Wright,andF.Niu.Hogwild:Alock-freeapproachtoparallelizingstochasticgradientdescent.InAdvancesinNeuralInformationProcessingSystems(NIPS)24,Granada,2011.[26]A.Ahmed,M.Aly,J.Gonzalez,S.Narayanamurthy,andA.J.Smola.Scalableinferenceinlatentvariablemodels.InProc.ofthe5thACMInternationalConferenceonWebSearchandDataMining,2012.[27]MuLi,LiZhou,ZichaoYang,AaronLi,FeiXia,DavidGAndersen,andAlexanderSmola.Parameterserverfordistributedmachinelearning.InBigLearnworkshop,atNIPS,LakeTahoe,2013.[28]Q.Ho,J.Cipar,H.Cui,S.Lee,J.K.Kim,P.B.Gibbons,G.A.Gibson,G.Ganger,andE.Xing.Moreeffectivedistributedmlviaastalesynchronousparallelparameterserver.InNIPS.2013.[29]X.Pan,J.E.Gonzalez,S.Jegelka,T.Broderick,andM.I.Jordan.Optimisticconcurrencycontrolfordistributedunsupervisedlearning.InAdvancesinNeuralInformationProcessingSystems26.2013.9\f", "award": [], "sourceid": 117, "authors": [{"given_name": "Xinghao", "family_name": "Pan", "institution": "UC Berkeley"}, {"given_name": "Stefanie", "family_name": "Jegelka", "institution": "UC Berkeley"}, {"given_name": "Joseph", "family_name": "Gonzalez", "institution": "UC Berkeley"}, {"given_name": "Joseph", "family_name": "Bradley", "institution": "University of California, Berkeley"}, {"given_name": "Michael", "family_name": "Jordan", "institution": "UC Berkeley"}]}