{"title": "GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration", "book": "Advances in Neural Information Processing Systems", "page_first": 7576, "page_last": 7586, "abstract": "Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call. BBMM reduces the asymptotic complexity of exact GP inference from O(n^3) to O(n^2). Adapting this algorithm to scalable approximations and complex GP models simply requires a routine for efficient matrix-matrix multiplication with the kernel and its derivative. In addition, BBMM uses a specialized preconditioner to substantially speed up convergence. In experiments we show that BBMM effectively uses GPU hardware to dramatically accelerate both exact GP inference and scalable approximations. Additionally, we provide GPyTorch, a software platform for scalable GP inference via BBMM, built on PyTorch.", "full_text": "GPyTorch:BlackboxMatrix-MatrixGaussianProcessInferencewithGPUAccelerationJacobR.Gardner\u2217,GeoffPleiss\u2217,DavidBindel,KilianQ.Weinberger,AndrewGordonWilsonCornellUniversity{jrg365,kqw4,andrew}@cornell.edu,{geoff,bindel}@cs.cornell.eduAbstractDespiteadvancesinscalablemodels,theinferencetoolsusedforGaussianpro-cesses(GPs)haveyettofullycapitalizeondevelopmentsincomputinghardware.Wepresentanef\ufb01cientandgeneralapproachtoGPinferencebasedonBlackboxMatrix-Matrixmultiplication(BBMM).BBMMinferenceusesamodi\ufb01edbatchedversionoftheconjugategradientsalgorithmtoderivealltermsfortrainingandinferenceinasinglecall.BBMMreducestheasymptoticcomplexityofexactGPinferencefromO(n3)toO(n2).Adaptingthisalgorithmtoscalableapproxima-tionsandcomplexGPmodelssimplyrequiresaroutineforef\ufb01cientmatrix-matrixmultiplicationwiththekernelanditsderivative.Inaddition,BBMMusesaspecial-izedpreconditionertosubstantiallyspeedupconvergence.InexperimentsweshowthatBBMMeffectivelyusesGPUhardwaretodramaticallyacceleratebothexactGPinferenceandscalableapproximations.Additionally,weprovideGPyTorch,asoftwareplatformforscalableGPinferenceviaBBMM,builtonPyTorch.1IntroductionThepastyearshavewitnessedunprecedentedinnovationindeeplearning.Thisprogresshasinvolvedinnovationsinnetworkdesigns[18,20,24,26,30],butitalsohasbene\ufb01tedvastlyfromimprovementsinoptimization[6],andexcellentsoftwareimplementationssuchasPyTorch,MXNet,TensorFlowandCaffe[1,8,28,38].Broadlyspeaking,thegainsinoptimizationoriginateinlargepartfrominsightsinstochasticgradientoptimization[6,7,23,27,29,31],effectivelytradingoffunnecessaryexactnessforspeedandinsomecasesregularization.Moreover,theadvantagesofmodernsoftwareframeworksfordeeplearningincluderapidprototyping,easyaccesstospecialtycomputehardware(suchasGPUs),andblackboxoptimizationthroughautomaticdifferentiation.Similarly,Gaussianprocessresearchhasundergonesigni\ufb01cantinnovationsinrecentyears[9,21,45,49\u201351]\u2014inparticulartoimprovescalabilitytolargedatasets.However,thetoolsmostcommonlyusedforGPinferencedonoteffectivelyutilizemodernhardware,andnewmodelsrequiresigni\ufb01cantimplementationefforts.Often,infact,themodelandtheinferenceenginearetightlycoupledandconsequentlymanycomplexmodelslikemulti-outputGPsandscalableGPapproximationsrequirecustominferenceprocedures[5,22].Thisentanglementofmodelspeci\ufb01cationandinferenceprocedureimpedesrapidprototypingofdifferentmodeltypes,andobstructsinnovationinthe\ufb01eld.Inthispaper,weaddressthisgapbyintroducingahighlyef\ufb01cientframeworkforGaussianprocessinference.WhereaspreviousinferenceapproachesrequiretheusertoprovideroutinesforcomputingthefullGPmarginalloglikelihoodforasuf\ufb01cientlycomplexmodel,ourframeworkonlyrequiresaccesstoablackboxroutinethatperformsmatrix-matrixmultiplicationswiththekernelmatrixanditsderivative.Accordingly,werefertoourmethodasBlackboxMatrix-Matrix(BBMM)Inference.\u2217Equalcontribution.32ndConferenceonNeuralInformationProcessingSystems(NeurIPS2018),Montr\u00e9al,Canada.\fIncontrasttotheCholeskydecomposition,whichisattheheartofmanyexistinginferenceengines,matrix-matrixmultiplicationsfullyutilizeGPUacceleration.Wewilldemonstratethatthismatrix-matrixapproachalsosigni\ufb01cantlyeasesimplementationforawideclassofexistingGPmodelsfromtheliterature.Inparticular,wemakethefollowingcontributions:1.Inspiredbyiterativematrix-vectormultiplication(MVM)-basedinferencemethods[9,13,43,50,51],weprovideamodi\ufb01edbatchedversionoflinearconjugategradients(mBCG)thatprovidesallcomputationsnecessaryforboththemarginallikelihoodanditsderivatives.Moreover,mBCGuseslargematrix-matrixmultiplicationsthatmoreef\ufb01cientlyutilizemodernhardwarethanbothexistingCholeskyandMVMbasedinferencestrategies.Ourapproachalsocircumventsseveralcriticalspacecomplexityandnumericalstabilityissuespresentinexistinginferencemethods.Mostnotably,BBMMreducesthetimecomplexityofexactGPinferencefromO(n3)toO(n2).2.Weintroduceamethodforpreconditioningthismodi\ufb01edconjugategradientsalgorithmbasedonthepivotedCholeskydecomposition[4,19].Allrequiredoperationswiththispreconditionerareef\ufb01cient,andinpracticerequirenegligibletime.Wedemonstratebothempiricallyandtheoreticallythatthispreconditionersigni\ufb01cantlyacceleratesinference.3.WeintroduceGPyTorch,anewsoftwareplatformusingBBMMinferenceforscalableGaussianprocesses,whichisbuiltontopofPyTorch:https://gpytorch.ai.Ondatasetsaslargeas3000datapoints(untilwe\ufb01llGPUmemory)wedemonstratethatexactGPswithBBMMareupto20\u00d7fasterthanGPsusingCholesky-basedapproaches.Moreover,thepopularSKI[50]andSGPR[45]frameworkswithBBMMachieveupto15\u00d7and4\u00d7speedups(respectively)ondatasetsaslargeas500,000datapoints.Additionally,SKI,SGPRandotherscalableapproximationsareimplementedinlessthan50linesofcode,requiringonlyanef\ufb01cientmatrix-matrixmultiplicationroutine.2RelatedWorkConjugategradients,theLanczostridiagonalizationalgorithm,andtheirrelativesaremethodsfromnumericallinearalgebraforcomputinglinearsolvesandsolvingeigenvalueproblemswithoutexplicitlycomputingamatrix.Thesetechniqueshavebeenaroundfordecades,andarecoveredinpopularbooksandpapers[11,12,17,32,36,37,42].ThesealgorithmsbelongtoabroadclassofiterativemethodsknownasKrylovsubspacemethods,whichaccessmatricesonlythroughmatrix-vectormultiplies(MVMs).Historically,thesemethodshavebeenappliedtosolvinglargenumericallinearalgebraproblems,particularlythoseinvolvingsparsematricesthataffordfastMVMs.Recently,anumberofpapershaveusedtheseMVMmethodsforpartsofGPinference[9,13,15,34,39,43,49,50].OnekeyadvantageisthatMVMapproachescanexploitalgebraicstructureforincreasedcomputationalef\ufb01ciencies.Notably,thestructuredkernelinterpolation(SKI)method[50]usesstructuredkernelmatriceswithfastMVMstoachievearemarkableasymptoticcomplexity.Dongetal.[13]proposeMVMmethodsforcomputingstochasticestimatesoflogdeterminantsandtheirderivativesusingatechniquebasedonLanczostridiagonalization[16,46].WeutilizethesamelogdeterminantestimatorasDongetal.[13],exceptweavoidexplicitlyusingtheLanczostridiagonalizationalgorithmwhichhasstorageandnumericalstabilityissues[17].Preconditioningisaneffectivetoolforacceleratingtheconvergenceofconjugategradients.Thesetechniquesarefartoonumeroustoreviewadequatelyhere;however,Saad[42]containstwochaptersdiscussingavarietyofpreconditioningtechniques.Cutajaretal.[10]exploresusingpreconditionedconjugategradientsforexactGPinference,wheretheyusevarioussparseGPmethods(aswellassomeclassicalmethods)aspreconditioners.However,themethodsinCutajaretal.[10]donotprovidegeneralpurposepreconditioners.Forexample,methodslikeJacobipreconditioninghavenoeffectwhenusingastationarykernel[10,51],andmanyotherpreconditionershave\u2126(n2)complexity,whichdominatesthecomplexityofmostscalableGPmethods.ThePivotedCholeskydecompositionisanef\ufb01cientalgorithmforcomputingalow-rankdecompo-sitionofapositivede\ufb01nitematrix[4,19],whichweuseinthecontextofpreconditioning.Harbrechtetal.[19]explorestheuseofthepivotedCholeskydecompositionasalowrankapproximation,althoughprimarilyinascienti\ufb01ccomputingcontext.Inprovingconvergenceboundsforourpre-conditionerweexplicitlymakeuseofsometheoreticalresultsfrom[19](seeAppendixD).Bach[4]considersusingrandomcolumnsamplingaswellasthepivotedCholeskydecompositionasalow-rankapproximationtokernelmatrices.However,Bach[4]treatsthisdecompositionasan2\fapproximatetrainingmethod,whereasweusethepivotedCholeskydecompositionprimarilyasapreconditioner,whichavoidsanylossofaccuracyfromthelowrankapproximationaswellasthecomplexityofcomputingderivatives.3BackgroundNotation.Xwilldenoteasetofntrainingexamplesinddimensions,orequivalentlyann\u00d7dmatrixwheretheithrow(denotedxi)istheithtrainingexample.ydenotesthetraininglabels.k(x,x0)denotesakernelfunction,andKXXdenotesthematrixcontainingallpairsofkernelentries,i.e.[KXX]ij=k(xi,xj).kXx\u2217denoteskernelvaluesbetweentrainingexamplesandatestpointx\u2217,e.g.[kXx\u2217]i=k(xi,x\u2217).Ahatdenotesanaddeddiagonal:bKXX=KXX+\u03c32I.AGaussianprocess(GP)isakernelmethodthatde\ufb01nesafulldistributionoverthefunctionbeingmodeled,f(x)\u223cGP(\u00b5(x),k(x,x0)).PopularkernelsincludetheRBFkernel,k(x,x0)=sexp(cid:0)\u2212(kx\u2212x0k)/(2\u20182)(cid:1)andtheMat\u00e9rnfamilyofkernels[41].PredictionswithaGaussianprocess.PredictionswithaGParemadeutilizingthepredictiveposteriordistribution,p(f(x\u2217)|X,y).Giventwotestinputsx\u2217andx\u22170,thepredictivemeanforx\u2217andthepredictivecovariancebetweenx\u2217andx\u22170aregivenby:\u00b5f|D(x\u2217)=\u00b5(x\u2217)+k>Xx\u2217bK\u22121XXy,kf|D(x\u2217,x\u22170)=kx\u2217x\u22170\u2212k>Xx\u2217bK\u22121XXkXx\u22170,(1)TrainingaGaussianprocess.Gaussianprocessesdependonanumberofhyperparameters\u03b8.Hyperparametersmayincludethelikelihoodnoise,kernellengthscale,inducingpointlocations[45],orneuralnetworkparametersfordeepkernellearning[52].Theseparametersarecommonlylearnedbyminimizationorsamplingviathenegativelogmarginallikelihood,given(withderivative)byL(\u03b8|X,y)\u221dlog(cid:12)(cid:12)(cid:12)bKXX(cid:12)(cid:12)(cid:12)\u2212y>bK\u22121XXy,dLd\u03b8=y>bK\u22121XXdbKXXd\u03b8bK\u22121XXy+Tr bK\u22121XXdbKXXd\u03b8!.(2)4GaussianprocessinferencethroughblackboxmatrixmultiplicationThegoalofourpaperistoreplaceexistinginferencestrategieswithauni\ufb01edframeworkthatutilizesmodernhardwareef\ufb01ciently.WeadditionallydesirethatcomplexGPmodelscanbeusedinablackboxmannerwithoutadditionalinferencerules.Tothisend,ourmethodreducesthebulkofGPinferencetooneofthemostef\ufb01ciently-parallelizedcomputations:matrix-matrixmultiplication.WecallourmethodBlackboxMatrix-Matrixinference(BBMM)becauseitonlyrequiresausertospecifyamatrixmultiplyroutineforthekernelbKXXManditsderivativedbKXXd\u03b8M.Requiredoperations.Aninferenceengineisaschemeforcomputingalltheequationsdiscussedabove:thepredictivedistribution(1),theloss,anditsderivative(2).Theseequationshavethreeoperationsincommonthatdominateitstimecomplexity:1)thelinearsolvebK\u22121XXy,2)thelogdeterminantlog|bKXX|,and3)atracetermTr(bK\u22121XXdbKXXd\u03b8).Inmanyimplementations,thesethreequantitiesarecomputedusingtheCholeskydecompositionofbKXX,whichiscomputationallyexpensive,requiringO(n3)operations,anddoesnoteffectivelyutilizeparallelhardware.Recently,thereisagrowinglineofresearchthatcomputestheseoperationswithiterativeroutinesbasedonmatrix-vectormultiplications(MVMs).bK\u22121XXycanbecomputedusingconjugategradients(CG)[9,10,43,50],andtheothertwoquantitiescanbecomputedusingcallstotheiterativeLanczostridiagonalizationalgorithm[13,46].MVM-basedmethodsareasymptoticallyfasterandmorespaceef\ufb01cientthanCholeskybasedmethods[13,50].Additionally,thesemethodsareabletoexploitalgebraicstructureinthedataforfurtheref\ufb01ciencies[9,43,50].However,theyalsohavedisadvantages.ThequantitiesarecomputedviaseveralindependentcallstotheCGandstochasticLanczosquadraturesubroutines,whichareinherentlysequentialandthereforedonotfullyutilizeparallelhardware.Additionally,theLanczostridiagonalizationalgorithmrequiresO(np)spaceforpiterationsandsuffersfromnumericalstabilityissuesduetolossoforthogonality[17].Modi\ufb01edCG.OurgoalistocapitalizeontheadvantagesofMVM-basedmethods(space-ef\ufb01ciency,abilitytoexploitstructure,etc.)butwithef\ufb01cientroutinesthatareoptimizedformodernparallel3\fcomputehardware.Forthispurpose,ourmethodmakesuseofamodi\ufb01edBatchedConjugateGradientsAlgorithm(mBCG)algorithm.StandardconjugategradientstakesasinputavectoryandaroutineforcomputingamatrixvectorproductbKXXy,and,afterpiterations,outputsanapproximatesolveup\u2248bK\u22121XXy(withexactequalitywhenp=n).Wemodifyconjugategradientsto(1)performlinearsolveswithmultiplerighthandsidessimultaneously,and(2)returntridiagonalmatricescorrespondingtopartialLanczostridiagonalizationsofbKXXwithrespecttoeachrighthandside.2Speci\ufb01cally,mBCGtakesasinputamatrix[yz1\u00b7\u00b7\u00b7zt],andoutputs:[u0u1\u00b7\u00b7\u00b7ut]=bK\u22121XX[yz1\u00b7\u00b7\u00b7zt]and\u02dcT1,...,\u02dcTt(3)where\u02dcT1,...,\u02dcTtarepartialLanczostridiagonalizationsofbKXXwithrespecttothevectorsz1,...,zt,whichwedescribeshortly.Inwhatfollows,weshowhowtouseasinglecalltomBCGtocomputethethreeGPinferenceterms:bK\u22121XXy,Tr(bK\u22121XX\u2202bKXX\u2202\u03b8),andlog|bKXX|.bK\u22121XXyisequaltou0in(3),directlyreturnedfrommBCG.Wedescribetheothertwotermsbelow.EstimatingTr(bK\u22121XX\u2202bKXX\u2202\u03b8)fromCGreliesonstochastictraceestimation[3,14,25],whichallowsustotreatthistermasasumoflinearsolves.Giveni.i.d.randomvariablesz1,...,ztsothatE[zi]=0andE(cid:2)ziz>i(cid:3)=I,(e.g.,zi\u223cN(0,I))thematrixtraceTr(A)canbewrittenasTr(A)=E(cid:2)z>iAzi(cid:3),suchthatTr bK\u22121XXdbKXXd\u03b8!=E\"z>ibK\u22121XXdbKXXd\u03b8zi#\u22481ttXi=1(cid:16)z>ibK\u22121XX(cid:17) dbKXXd\u03b8zi!(4)isanunbiasedestimatorofthederivative.Thiscomputationmotivatesthez1,...,zttermsin(3):themBCGcallreturnsthesolvesbK\u22121XX[z1...zt],whichyieldsui=z>ibK\u22121XX.AsinglematrixmultiplywiththederivativedbKXXd\u03b8[z1...zt]yieldstheremainingtermsontheRHS.Thefulltracecanthenbeestimatedbyelementwisemultiplyingthesetermstogetherandsumming,asin(4).Estimatinglog|bKXX|canbeaccomplishedusingtheT1,...,TtmatricesfrommBCG.IfbKXX=QTQ>,withQorthonormal,thenbecausebKXXandThavethesameeigenvalues:log|bKXX|=Tr(logT)=E(cid:2)z>i(logT)zi(cid:3)\u2248tXi=1z>i(logT)zi(5)wherelogTheredenotesthematrixlogarithm,andtheapproximationcomesfromthesamestochastictraceestimationtechniqueusedfor(4).OneapproachtoobtainadecompositionbKXX=QTQ>istousetheLanczostridiagonalizationalgorithm.ThisalgorithmtakesthematrixbKXXandaprobevectorzandoutputsthedecompositionQTQ>(wherezisthe\ufb01rstcolumnofQ).However,ratherthanrunningthefullalgorithm,wecaninsteadrunpiterationsofthealgorithmttimes,eachwithavectorz1,...,zttoobtaintdecompositions\u02dcQ1\u02dcT1\u02dcQ>1,...,\u02dcQt\u02dcTt\u02dcQ>twith\u02dcQi\u2208Rn\u00d7pand\u02dcTi\u2208Rp\u00d7p.Wecanusethesepartialdecompositionstoestimate(5):E(cid:2)z>i(logT)zi(cid:3)=Ehz>i\u02dcQi(log\u02dcTi)\u02dcQ>izii\u22481ttXi=1z>i\u02dcQi(log\u02dcTi)\u02dcQ>izi=1ttXi=1e>1(log\u02dcTi)e1,(6)wheree1isthe\ufb01rstrowoftheidentitymatrix.RunningLanczoswithastartingvectorziensuresthatallcolumnsof\u02dcQiareorthogonaltoziexceptthe\ufb01rst,so\u02dcQizi=e1[13,16,46].InmBCG,weadaptatechniquefromSaad[42]whichallowsustocompute\u02dcT1,...,\u02dcTtcorrespondingtotheinputvectorsz1,...,zttomBCGfromthecoef\ufb01cientsofCGinO(1)additionalworkperiteration.Thisapproachallowsustocomputealogdeterminantestimateidenticalto(6)withoutrun-ningtheLanczosalgorithm.Thusweavoidtheextracomputation,storage,andnumericalinstabilityassociatedwithLanczositerations.WedescribethedetailsofthisadaptationinAppendixA.Runtimeandspace.Asshownabove,weareabletoapproximateallinferencetermsfromasinglecalltomBCG.TheseapproximationsimprovewiththenumberofmBCGiterations.Eachiteration2mBCGdifferesfromBlockCGalgorithms[35]inthatmBCGreturnsLanczostridiagonalizationterms.4\frequiresonematrix-matrixmultiplywithbKXX,andthesubsequentworktoderivetheseinferencetermstakesnegligibleadditionaltime(AppendixB).Therefore,piterationsofmBCGrequiresO(nt)space(seeAppendixB)andO(p\u039e(bKXX))time,where\u039e(bKXX)isthetimetomultiplybKXXbyan\u00d7tmatrix.ThismultiplicationtakesO(n2t)timewithastandardmatrix.ItisworthnotingthatthisisalowerasymptoticcomplexitythatstandardCholesky-basedinference,whichisO(n3).Therefore,BBMMoffersacomputationalspeedupforexactGPinference.AswewillshowinSection5,thistimecomplexitycanbefurtherreducedwithstructureddataorsparseGPapproximations.4.1PreconditioningWhileeachiterationofmBCGperformslargeparallelmatrix-matrixoperationsthatutilizehardwareef\ufb01ciently,theiterationsthemselvesaresequential.Anaturalgoalforbetterutilizinghardwareistotradeofffewersequentialstepsforslightlymoreeffortperstep.Weaccomplishthisgoalusingpreconditioning[12,17,42,47],whichintroducesamatrixPtosolvetherelatedlinearsystemP\u22121bKXXu=P\u22121yinsteadofbK\u22121XXy.Bothsystemsareguaranteedtohavethesamesolution,butthepreconditionedsystem\u2019sconvergencedependsontheconditioningofP\u22121bKXXratherthanthatofbKXX.WeobservetworequirementsofapreconditionertobeusedingeneralforGPinference.First,inordertoensurethatpreconditioningoperationsdonotdominaterunningtimewhenusingscalableGPmethods,thepreconditionershouldaffordroughlylineartimesolvesandspace.Second,weshouldbeabletoef\ufb01cientlycomputethelogdeterminantofthepreconditionermatrix,log|P|.ThisisbecausethemBCGalgorithmappliedtothepreconditionedsystemestimateslog|P\u22121bKXX|ratherthanlog|bKXX|.Wemustthereforecomputelog|bKXX|=log|P\u22121bKXX|+log|P|.ThePivotedCholeskyDecomposition.Foronepossiblepreconditioner,weturntothepivotedCholeskydecomposition.ThepivotedCholeskyalgorithmallowsustocomputealow-rankap-proximationofapositivede\ufb01nitematrix,KXX\u2248LkL>k[19].WepreconditionmBCGwith(LkL>k+\u03c32I)\u22121,where\u03c32istheGaussianlikelihood\u2019snoiseterm.Intuitively,ifPk=LkL>kisagoodapproximationofKXX,then(Pk+\u03c32I)\u22121bKXX\u2248I.WhilewereviewthepivotedCholeskyalgorithmfullyinAppendixC,wewouldliketoemphasizethreekeyproperties.First,itcanbecomputedinO(\u03c1(KXX)k2)time,where\u03c1(KXX)isthetimetoaccessarow(nominallythisisO(n)).Second,linearsolveswithbP=LkL>k+\u03c32IcanbeperformedinO(nk2)time.Finally,thelogdeterminantofbPcanbecomputedinO(nk2)time.InFigure6weempiricallyshowthatthispreconditionerdramaticallyacceleratesCGconvergence.Further,inAppendixD,weprovethefollowinglemmaandtheoremforunivariateRBFkernels:Lemma1.LetKXX\u2208Rn\u00d7nbeaunivariateRBFkernelmatrix.LetLkL>kbetherankkpivotedCholeskydecompositionofKXX,andletbPk=LkL>k+\u03c32I.Thenthereexistsaconstantb>0sothattheconditionnumber\u03ba(bP\u22121bKXX)satis\ufb01esthefollowinginequality:\u03ba(cid:16)bP\u22121kbKXX(cid:17),(cid:13)(cid:13)(cid:13)bP\u22121kbKXX(cid:13)(cid:13)(cid:13)2(cid:13)(cid:13)(cid:13)bK\u22121XXbPk(cid:13)(cid:13)(cid:13)2\u2264(1+O(nexp(\u2212bk)))2.(7)Theorem1(ConvergenceofpivotedCholesky-preconditionedCG).LetKXX\u2208Rn\u00d7nbean\u00d7nunivariateRBFkernel,andletLkL>kbeitsrankkpivotedCholeskydecomposition.AssumeweareusingpreconditionedCGtosolvethesystembK\u22121XXy=(KXX+\u03c32I)\u22121ywithpreconditionerbP=(LkL>k+\u03c32I).LetupbethepthsolutionofCG,andletu\u2217=bK\u22121XXybetheexactsolution.Thenthereexistssomeb>0suchthat:ku\u2217\u2212upkbKXX\u22642(1/(1+O(exp(kb)/n))pku\u2217\u2212u0kbKXX.(8)Theorem1impliesthatweshouldexpecttheconvergenceofconjugategradientstoimproveexponen-tiallywiththerankofthepivotedCholeskydecompositionusedforRBFkernels.Inourexperimentsweobservesigni\ufb01cantlyimprovedconvergenceforotherkernelsaswell(Figure6).Furthermore,wecanleverageLemma1andexistingtheoryfrom[46]toarguethatpreconditioningimprovesourlogdeterminantestimate.Inparticular,werestateTheorem4.1ofUbaruetal.[46]here:5\fTheorem2(Theorem4.1ofUbaruetal.[46]).LetKXX\u2208Rn\u00d7n,andletLkL>kbeitsrankkpivotedCholeskydecomposition.Supposewerunp\u226514r\u03ba(cid:16)bP\u22121kbKXX(cid:17)logD\u0001iterationsofmBCG,whereDisaterminvolvingthissameconditionnumberthatvanishesask\u2192n(see[46]),andweuset\u226524\u00012log(2/\u03b4)vectorsziforthesolves.Let\u0393bethelogdeterminantestimatefrom(6).Then:Prh|log|bP\u22121bKXX|\u2212\u0393|\u2264\u0001|log|bP\u22121bKXX||i\u22651\u2212\u03b4.(9)BecauseLemma1statesthattheconditionnumber\u03ba(cid:16)bP\u22121kbKXX(cid:17)decaysexponentiallywiththerankofLk,Theorem2impliesthatweshouldexpectthatthenumberofCGiterationsrequiredtoaccuratelyestimatelog|bP\u22121bKXX|decreasesquicklyaskincreases.Inaddition,inthelimitask\u2192nwehavethatlog|bKXX|=log|bP|.Thisisbecauselog|bP\u22121bKXX|\u21920(sincebP\u22121bKXXconvergestoI)andwehavethatlog|bKXX|=log|bP\u22121bKXX|+log|bP|.Sinceourcalculationoflog|bP|isexact,our\ufb01nalestimateoflog|bKXX|becomesmoreexactaskincreases.Infutureworkwehopetoderiveamoregeneralresultthatcoversmultivariatesettingsandotherkernels.5ProgrammabilitywithBBMMWehavediscussedhowtheBBMMframeworkismorehardwareef\ufb01cientthanexistinginferenceengines,andavoidsnumericalinstabilitieswithLanczos.AnotherkeyadvantageofBBMMisthatitcaneasilybeadaptedtocomplexGPmodelsorstructuredGPapproximations.IndeedBBMMisblackboxbynature,onlyrequiringaroutinetoperformmatrix-multiplicationswiththekernelmatrixanditsderivative.HereweprovideexamplesofhowexistingGPmodelsandscalableapproximationscanbeeasilyimplementedinthisframework.Thematrix-multiplicationroutinesforthemodelsrequireatmost50linesofPythoncode.Alloursoftware,includingthefollowingGPimplementationswithBBMM,areavailablethroughourGPyTorchlibrary:https://gpytorch.ai.BayesianlinearregressioncanbeviewedasGPregressionwiththespecialkernelmatrixbKXX=XX>+\u03c32I.Amatrixmultiplywiththiskernelagainstann\u00d7tmatrixV,(XX>+\u03c32I)VrequiresO(tnd)time.Therefore,BBMMrequiresO(ptnd)time,andisexactinO(tnd2)time.Thisrunningtimecomplexitymatchesexistingef\ufb01cientalgorithmsforBayesianlinearregression,withnoadditionalderivation.Multi-taskGaussianprocesses[5]canbeadaptedinthesamefashion[15].SparseGaussianProcessRegression(SGPR)[45]andmanyothersparseGPtechniques[21,40,44]usethesubsetofregressors(SoR)approximationforthekernel:bKXX\u2248(KXUK\u22121UUKUX+\u03c32I).Performingamatrix-matrixmultiplywiththismatrixrequiresO(tnm+tm3)timebydistribut-ingthevectormultiplyandgroupingtermscorrectly.ThiscomputationisasymptoticallyfasterthantheO(nm2+m3)timerequiredbyCholeskybasedinference.AugmentingtheSoRapproximationwithadiagonalcorrection,e.g.asinFITC[44],issimilarlystraightforward.StructuredKernelInterpolation(SKI)[50],alsoknownasKISS-GP,isaninducingpointmethoddesignedtoprovidefastmatrixvectormultiplies(MVMs)forusewithKrylovsubspacemethods.SKIisthusanaturalcandidateforBBMMandcanbene\ufb01tgreatlyfromhardwareacceleration.SKIisageneralizationofSoR,whichspeci\ufb01esKXU\u2248WKUU,whereWisasparsematrix.ForexampleWcancorrespondtothecoef\ufb01cientsofsparselocalcubicconvolutioninterpolation.TheSKIapproximationappliedtothetrainingcovariancematrixgivesusbKXX\u2248(WKUUW>+\u03c32I).AssumingnostructureinKUUamatrixmultiplyrequiresO(tn+tm2)time.InKISS-GP[50,51],thematrixKUUisalsochosentohavealgebraicstructure,suchasKroneckerorToeplitzstructure,whichfurtheracceleratesMVMs.Forexample,MVMswithaToeplitzKUUonlyrequireO(mlogm)time.ThusKISS-GPprovidesO(tn+tmlogm)matrix-matrixmultiplies[50].Compositionsofkernelscanoftenbehandledautomatically.Forexample,givenaBBMMroutineforK1,K2,K3,wecanautomaticallyperform(K1K2+K3)M=K1(K2M)+K3M.SGPRandKISS-GPareimplementedinthisfashion.Givensomepre-de\ufb01nedbasiccompositionalitystrategies,thekernelmatrixmultiplicationKMinSGPRreducestode\ufb01ninghowtoperformK\u22121UUM,and6\fSkillcraftn=3338Gasn=2565Winen=1599Airfoiln=1503Autompgn=3920102030Speedup over Cholesky CPU (Exact GP)1x SpeedupCholesky (GPU)GPyTorch (GPU)KEGGn=49kProteinn=45kKin40kn=40kElevatorsn=17kPolTelen=15k0246810Speed up over Cholesky CPU (SGPR)1x SpeedupCholesky (GPU)GPyTorch (GPU)Songn=515kBuzzn=583kKEGGn=49kProteinn=45kKin40kn=40k0102030Speed up Over Dong et al. [13] CPU (DKL+SKI)1x SpeedupDong et al. [13] (GPU)GPyTorch (GPU)Speedup FactorSpeedup FactorSpeedup FactorFigure1:SpeedupofGPU-acceleratedinferenceengines.BBMMisinblue,andcompetingGPUmethodsareingray.Left:ExactGPs.Middle:SGPR[21,45]\u2013speedupoverCPUCholesky-basedinferenceengines.Right:SKI+DKL[50,52]\u2013speedupoverCPUinferenceofDongetal.[13].similarlyforKISS-GPitreducestoperformingmultiplicationwithaToeplitzmatrixKUUM.ForproductkernelsonecanfollowGardneretal.[15].6ResultsWeevaluatetheBBMMframework,demonstrating:(1)theBBMMinferenceengineprovidesasubstantialspeedbene\ufb01toverCholeskybasedinferenceandstandardMVM-basedCGinference,especiallyforGPUcomputing;(2)BBMMachievescomparableorbetter\ufb01naltesterrorcomparedtoCholeskyinference,evenwithnokernelapproximations;and(3)preconditioningprovidesasubstantialimprovementintheef\ufb01ciencyofourapproach.Baselinemethods.WetestBBMMonthreetypesofGPs:1.ExactGPmodels,2.SGPRinducingpointmodels[21,45],and3.SKImodelswithToeplitzKUUanddeepkernels[50,52].ForExactandSGPR,wecompareBBMMagainstCholesky-basedinferenceenginesimplementedinGPFlow[33].GPFlowispresentlythefastestimplementationofthesemodelswithaCholeskyinferenceengine.SinceSKIisnotintendedforCholeskyinference,wecompareBBMMtotheinferenceprocedureofDongetal.[13],implementedinourGPyTorchpackage.ThisproceduredifferersfromBBMMinthatitcomputesbK\u22121XXywithoutapreconditionerandcomputeslog|bKXX|anditsderivativewiththeLanczosalgorithm.Datasets.WetestExactmodelson\ufb01vedatasetsfromtheUCIdatasetrepository[2]withupto3500trainingexamples(thelargestpossiblebeforeallimplementationsexhaustedGPUmemory):Skillcraft,Gas,Airfoil,Autompg,andWine.WetestSGPRonlargerdatasets(nupto50000):KEGG,Protein,Elevators,Kin40k,andPoleTele.ForSKIwetest\ufb01veofthelargestUCIdatasets(nupto515000):Song,Buzz,Protein,Kin40k,andKEGG.Experimentdetails.Allmethodsusethesameoptimizer(Adam)withidenticalhyperparameters.InBBMMexperimentsweuserankk=5pivotedCholeskypreconditionersunlessotherwisestated.Weuseamaximumofp=20iterationsofCGforeachsolve,andweuset=10probevectors\ufb01lledwithRademacherrandomvariablestoestimatethelogdeterminantandtraceterms.SGPRmodelsuse300inducingpoints.SKImodelsuse10,000inducingpointsandthedeepkernelsdescribedin[52].TheBBMMinferenceengineisimplementedinourGPyTorchpackage.152025300.00.10.20.30.40.5mBCG (\ufb02oat)Cholesky (\ufb02oat)Cholesky (double)CG Iterations PerformedResidual ErrorFigure2:SolveerrorformBCGandCholesky.AllspeedexperimentsarerunonanIntelXeonE5-2650CPUandanNVIDIATitanXpGPU.Speedcomparison.Figure1showsthespeedupobtainedbyGPU-acceleratedBBMMovertheleadingCPU-basedinferenceengines(CholeskyforExact/SGPR,Dongetal.[13]forSKI).Aswouldbeexpected,GPU-acceleratedBBMMisfasterthanCPU-basedinference.OnExactandSKI,BBMMisupto32timesfasterthanCPUinference,andupto10timesfasteronSGPR.Thelargestspeedupsoccuronthelargestdatasets,sincesmallerdatasetsexperi-encelargerGPUoverhead.Notably,BBMMachievesamuchlargerspeedupthanGPUac-7\fSkillcraftn=3338Gasn=2565Winen=1599Airfoiln=1503Autompgn=39201234Test MAEExact GP Errors (RBF)CholeskyGPyTorchSkillcraftn=3338Gasn=2565Winen=1599Airfoiln=1503Autompgn=39201234Test MAEExact GP Errors (Matern 5/2)CholeskyGPyTorchKEGGn=49kProteinn=45kKin40kn=40kElevatorsn=17kPolTelen=15k0.00.20.40.6Test MAESGPR Errors (Matern 5/2)CholeskyGPyTorchFigure3:Comparing\ufb01nalTestMAEwhenusingBBMMversusCholeskybasedinference.ThelefttwoplotscompareerrorsusingExactGPswithRBFandMatern-5/2kernels,andthe\ufb01nalplotcompareserrorusingSGPRwithaMatern-5/2kernelonsigni\ufb01cantlylargerdatasets.20406080100CGIterations10\u2212310\u22121SolveErrorProtein-DeepRBFKernelRank0(NoPrecond.)Rank2Rank5Rank920406080100CGIterations10\u2212410\u22122100SolveErrorKEGG-DeepMatern-5/2KernelNoPreconditionerRank2Rank5Rank90255075100125150TestTime(s)02TestMAEProtein-WallclocktimeNoPreconditionerRank5Preconditioner0255075100125TestTime(s)01020TestMAEKEGG-WallclocktimeNoPreconditionerRank5PreconditionerFigure4:TheeffectofpreconditioningonsolveerrorskKx\u2217\u2212yk/kykachievedbylinearconjugategradientsusingnopreconditionerversusrank2,5,and9pivotedCholeskypreconditionerson2UCIbenchmarkdatasetsusingdeepRBFanddeepMaternkernels.ThehyperparametersofKwerelearnedbymaximizingthemarginalloglikelihoodoneachdataset.celeratedCholeskymethods(Exact,SGPR),whichonlyachievearoughly4\u00d7speedup.ThisresultunderscoresthefactthatCholeskymethodsarenotaswellsuitedforGPUacceleration.Additionally,BBMMperformsbetterthantheGPU-acceleratedversionof[13]onSKI.ThisspeedupisbecauseBBMMisabletocalculateallinferencetermsinparallel,while[13]computesthetermsinseries.Errorcomparison.InFigure3wereporttestmeanaverageerror(MAE)forExactandSGPRmodels.3WedemonstrateresultsusingboththeRBFkernelandaMatern-5/2kernel.Acrossalldatasets,ourmethodisatleastasaccurateintermsof\ufb01naltestMAE.Onafewdatasets(e.g.Gas,Airfoil,andWinewithExactGPs)BBMMevenimproves\ufb01naltesterror.CGhasaregularizingeffectswhichmayimprovemethodsinvolvingtheexactkernelovertheCholeskydecomposition,wherenumericalissuesresultingfromextremelysmalleigenvaluesofthekernelmatrixareignored.Forexample,Choleskymethodsfrequentlyaddnoise(or\u201cjitter\u201d)tothediagonalofthekernelmatrixfornumericalstability.Itispossibletoreducethenumericalinstabilitieswithdoubleprecision(seeFigure2);however,thisrequiresanincreasedamountofcomputation.BBMMontheotherhandavoidsaddingthisnoise,withoutrequiringdoubleprecision.Preconditioning.Todemonstratetheeffectivenessofpreconditioningatacceleratingtheconvergenceofconjugategradients,we\ufb01rsttrainadeepRBFkernelmodelontwodatasets,ProteinandKEGG,andevaluatethesolveerrorofperformingbK\u22121XXyintermsoftherelativeresidualkbKXXu\u2212yk/kykasafunctionofthenumberofCGiterationsperformed.Welookatthiserrorwhenusingnopreconditioner,aswellasarank2,5,and9preconditioner.TodemonstratethatthepreconditionerisnotrestrictedtousewithanRBFkernel,weevaluateusingadeepRBFkernelonProteinanda3SKImodelsareexcludedfromFigure3.ThisisbecausetheBBMMinferenceengineandtheinferenceengineofDongetal.[13]returnidenticaloutputs(seeAppendixA)eventhoughBBMMisfaster.8\fdeepMatern-5/2kernelonKEGG.TheresultsareinthetopofFigure4.Asexpectedbasedonourtheoreticalintuitionsforthispreconditioner,increasingtherankofthepreconditionersubstantiallyreducesthenumberofCGiterationsrequiredtoachieveconvergence.InthebottomofFigure4,wecon\ufb01rmthatthesemoreaccuratesolvesindeedhaveaneffectonthe\ufb01naltestMAE.Weplot,asafunctionofthetotalwallclocktimerequiredtocomputepredictions,thetestMAEresultingfromusingnopreconditionerandfromusingarank5preconditioner.ThewallclocktimeisvariedbychangingthenumberofCGiterationsusedtocomputethepredictivemean.Weobservethat,becausesuchalowrankpreconditionerissuf\ufb01cient,usingpreconditioningresultsinsigni\ufb01cantlymoreaccuratesolveswhilehavingvirtuallynoimpactontherunningtimeofeachCGiteration.Consequentially,werecommendalwaysusingthepivotedCholeskypreconditionerwithBBMMsinceithasvirtuallynowall-clockoverheadandrapidlyacceleratesconvergence.7DiscussionInthispaper,wediscussanovelframeworkforGaussianprocessinference(BBMM)basedonblackboxmatrix-matrixmultiplicationroutineswithkernelmatrices.Wehaveimplementedthisframeworkandseveralstate-of-the-artGPmodelsinournewpubliclyavailableGPyTorchpackage.Non-Gaussianlikelihoods.Althoughthispaperprimarilyfocusesontheregressionsetting,BBMMisfullycompatiblewithvariationaltechniquessuchas[22,53],whicharealsosupportedinGPyTorch.Theseapproachesrequirecomputingthevariationallowerbound(orELBO)ratherthantheGPmarginalloglikelihood(2).WeleavetheexactdetailsoftheELBOderivationtootherpapers(e.g.[22]).However,wenotethatasinglecalltomBCGcanbeusedtocomputetheKLdivergencebetweentwomultivariateGaussians,whichisthemostcomputationallyintensivetermoftheELBO.AvoidingtheCholeskydecomposition.Asurprisingandimportanttake-awayofthispaperisthatitisbene\ufb01cialtoavoidtheCholeskydecompositionforGPinference,evenintheexactGPsetting.ThebasicalgorithmfortheCholeskydecomposition(describedinAppendixC)involvesadivide-andconquerapproachthatcanproveill-suitedforparallelhardware.Additionally,theCholeskydecompositionperformsalargeamountofcomputationtogetalinearsolvewhenfastapproximatemethodssuf\ufb01ce.Ultimately,theCholeskydecompositionofafullmatrixtakesO(n3)timewhileCGtakesO(n2)time.Indeed,asshowninFigure2,CGmayevenprovidebetterlinearsolvesthantheCholeskydecomposition.Whileweuseapivotedversionofthisalgorithmforpreconditioning,weonlycomputethe\ufb01rst\ufb01verowsofthisdecomposition.Byterminatingthealgorithmveryearly,weavoidthecomputationalbottleneckandmanyofthenumericalinstabilities.ItisourhopethatthisworkdramaticallyreducesthecomplexityofimplementingnewGaussianprocessmodels,whileallowingforinferencetobeperformedasef\ufb01cientlyaspossible.AcknowledgementsJRGandAGWaresupportedbyNSFIIS-1563887andbyFacebookResearch.GPandKQWaresupportedinpartbytheIII-1618134,III-1526012,IIS-1149882,IIS-1724282,andTRIPODS-1740822grantsfromtheNationalScienceFoundation.Inaddition,theyaresupportedbytheBillandMelindaGatesFoundation,theOf\ufb01ceofNavalResearch,andSAPAmericaInc.References[1]M.Abadi,P.Barham,J.Chen,Z.Chen,A.Davis,J.Dean,M.Devin,S.Ghemawat,G.Irving,M.Isard,etal.Tensor\ufb02ow:Asystemforlarge-scalemachinelearning.InOSDI,volume16,pages265\u2013283,2016.[2]A.AsuncionandD.Newman.Ucimachinelearningrepository.https://archive.ics.uci.edu/ml/,2007.Lastaccessed:2018-05-18.[3]H.AvronandS.Toledo.Randomizedalgorithmsforestimatingthetraceofanimplicitsymmetricpositivesemi-de\ufb01nitematrix.JournaloftheACM(JACM),58(2):8,2011.[4]F.Bach.Sharpanalysisoflow-rankkernelmatrixapproximations.InCOLT,2013.[5]E.V.Bonilla,K.M.Chai,andC.Williams.Multi-taskGaussianprocessprediction.InNIPS,2008.9\f[6]L.Bottou.Large-scalemachinelearningwithstochasticgradientdescent.InCOMPSTAT,pages177\u2013186.Springer,2010.[7]P.Chaudhari,A.Choromanska,S.Soatto,Y.LeCun,C.Baldassi,C.Borgs,J.Chayes,L.Sagun,andR.Zecchina.Entropy-sgd:Biasinggradientdescentintowidevalleys.arXivpreprintarXiv:1611.01838,2016.[8]T.Chen,M.Li,Y.Li,M.Lin,N.Wang,M.Wang,T.Xiao,B.Xu,C.Zhang,andZ.Zhang.Mxnet:A\ufb02exibleandef\ufb01cientmachinelearninglibraryforheterogeneousdistributedsystems.arXivpreprintarXiv:1512.01274,2015.[9]J.P.Cunningham,K.V.Shenoy,andM.Sahani.FastGaussianprocessmethodsforpointprocessintensityestimation.InICML,2008.[10]K.Cutajar,M.Osborne,J.Cunningham,andM.Filippone.Preconditioningkernelmatrices.InICML,2016.[11]B.N.Datta.Numericallinearalgebraandapplications,volume116.Siam,2010.[12]J.W.Demmel.Appliednumericallinearalgebra,volume56.Siam,1997.[13]K.Dong,D.Eriksson,H.Nickisch,D.Bindel,andA.G.Wilson.ScalablelogdeterminantsforGaussianprocesskernellearning.InNIPS,2017.[14]J.K.Fitzsimons,M.A.Osborne,S.J.Roberts,andJ.F.Fitzsimons.Improvedstochastictraceestimationusingmutuallyunbiasedbases.arXivpreprintarXiv:1608.00117,2016.[15]J.R.Gardner,G.Pleiss,R.Wu,K.Q.Weinberger,andA.G.Wilson.ProductkernelinterpolationforscalableGaussianprocesses.InAISTATS,2018.[16]G.H.GolubandG.Meurant.Matrices,momentsandquadraturewithapplications.PrincetonUniversityPress,2009.[17]G.H.GolubandC.F.VanLoan.Matrixcomputations,volume3.JHUPress,2012.[18]R.H.Hahnloser,R.Sarpeshkar,M.A.Mahowald,R.J.Douglas,andH.S.Seung.Digitalselectionandanalogueampli\ufb01cationcoexistinacortex-inspiredsiliconcircuit.Nature,405(6789):947,2000.[19]H.Harbrecht,M.Peters,andR.Schneider.Onthelow-rankapproximationbythepivotedcholeskydecomposition.Appliednumericalmathematics,62(4):428\u2013440,2012.[20]K.He,X.Zhang,S.Ren,andJ.Sun.Deepresiduallearningforimagerecognition.InCVPR,2016.[21]J.Hensman,N.Fusi,andN.D.Lawrence.Gaussianprocessesforbigdata.InUAI,2013.[22]J.Hensman,A.G.d.G.Matthews,andZ.Ghahramani.ScalablevariationalGaussianprocessclassi\ufb01cation.InICML,2015.[23]S.HochreiterandJ.Schmidhuber.Flatminima.NeuralComputation,9(1):1\u201342,1997.[24]G.Huang,Z.Liu,K.Q.Weinberger,andL.vanderMaaten.Denselyconnectedconvolutionalnetworks.InCVPR,2017.[25]M.F.Hutchinson.Astochasticestimatorofthetraceofthein\ufb02uencematrixforlaplaciansmoothingsplines.CommunicationsinStatistics-SimulationandComputation,19(2):433\u2013450,1990.[26]S.IoffeandC.Szegedy.Batchnormalization:Acceleratingdeepnetworktrainingbyreducinginternalcovariateshift.InICML,2015.[27]P.Izmailov,D.Podoprikhin,T.Garipov,D.Vetrov,andA.G.Wilson.Averagingweightsleadstowideroptimaandbettergeneralization.InUncertaintyinArti\ufb01cialIntelligence(UAI),2018.[28]Y.Jia,E.Shelhamer,J.Donahue,S.Karayev,J.Long,R.Girshick,S.Guadarrama,andT.Darrell.Caffe:Convolutionalarchitectureforfastfeatureembedding.InACMMM,pages675\u2013678.ACM,2014.[29]N.S.Keskar,D.Mudigere,J.Nocedal,M.Smelyanskiy,andP.T.P.Tang.Onlarge-batchtrainingfordeeplearning:Generalizationgapandsharpminima.arXivpreprintarXiv:1609.04836,2016.[30]A.Krizhevsky,I.Sutskever,andG.E.Hinton.Imagenetclassi\ufb01cationwithdeepconvolutionalneuralnetworks.InNIPS,2012.10\f[31]A.Krizhevsky,I.Sutskever,andG.E.Hinton.Imagenetclassi\ufb01cationwithdeepconvolutionalneuralnetworks.InAdvancesinneuralinformationprocessingsystems,pages1097\u20131105,2012.[32]C.Lanczos.Aniterationmethodforthesolutionoftheeigenvalueproblemoflineardifferentialandintegraloperators.UnitedStatesGovernm.PressOf\ufb01ceLosAngeles,CA,1950.[33]A.G.d.G.Matthews,M.vanderWilk,T.Nickson,K.Fujii,A.Boukouvalas,P.Le\u00f3n-Villagr\u00e1,Z.Ghahra-mani,andJ.Hensman.Gp\ufb02ow:AGaussianprocesslibraryusingTensorFlow.JournalofMachineLearningResearch,18(40):1\u20136,2017.[34]I.Murray.Gaussianprocessesandfastmatrix-vectormultiplies.InICMLWorkshoponNumericalMathematicsinMachineLearning,2009.[35]D.P.O\u2019Leary.Theblockconjugategradientalgorithmandrelatedmethods.Linearalgebraanditsapplications,29:293\u2013322,1980.[36]C.Paige.PracticaluseofthesymmetricLanczosprocesswithre-orthogonalization.BITNumericalMathematics,10(2):183\u2013195,1970.[37]B.N.Parlett.AnewlookattheLanczosalgorithmforsolvingsymmetricsystemsoflinearequations.Linearalgebraanditsapplications,29:323\u2013346,1980.[38]A.Paszke,S.Gross,S.Chintala,G.Chanan,E.Yang,Z.DeVito,Z.Lin,A.Desmaison,L.Antiga,andA.Lerer.AutomaticdifferentiationinPyTorch.2017.[39]G.Pleiss,J.R.Gardner,K.Q.Weinberger,andA.G.Wilson.Constant-timepredictivedistributionsforGaussianprocesses.InICML,2018.[40]J.Qui\u00f1onero-CandelaandC.E.Rasmussen.AunifyingviewofsparseapproximateGaussianprocessregression.JournalofMachineLearningResearch,6(Dec):1939\u20131959,2005.[41]C.E.RasmussenandC.K.Williams.Gaussianprocessesformachinelearning,volume1.MITpressCambridge,2006.[42]Y.Saad.Iterativemethodsforsparselinearsystems,volume82.siam,2003.[43]Y.Saat\u00e7i.ScalableinferenceforstructuredGaussianprocessmodels.PhDthesis,UniversityofCambridge,2012.[44]E.SnelsonandZ.Ghahramani.SparseGaussianprocessesusingpseudo-inputs.InNIPS,2006.[45]M.K.Titsias.VariationallearningofinducingvariablesinsparseGaussianprocesses.InAISTATS,pages567\u2013574,2009.[46]S.Ubaru,J.Chen,andY.Saad.Fastestimationoftr(f(a))viastochasticLanczosquadrature.SIAMJournalonMatrixAnalysisandApplications,38(4):1075\u20131099,2017.[47]H.A.VanderVorst.IterativeKrylovmethodsforlargelinearsystems,volume13.CambridgeUniversityPress,2003.[48]A.J.WathenandS.Zhu.Onspectraldistributionofkernelmatricesrelatedtoradialbasisfunctions.NumericalAlgorithms,70(4):709\u2013726,2015.[49]A.G.Wilson.CovariancekernelsforfastautomaticpatterndiscoveryandextrapolationwithGaussianprocesses.PhDthesis,UniversityofCambridge,2014.[50]A.G.WilsonandH.Nickisch.KernelinterpolationforscalablestructuredGaussianprocesses(KISS-GP).InICML,2015.[51]A.G.Wilson,C.Dann,andH.Nickisch.ThoughtsonmassivelyscalableGaussianprocesses.arXivpreprintarXiv:1511.01870,2015.[52]A.G.Wilson,Z.Hu,R.Salakhutdinov,andE.P.Xing.Deepkernellearning.InAISTATS,2016.[53]A.G.Wilson,Z.Hu,R.R.Salakhutdinov,andE.P.Xing.Stochasticvariationaldeepkernellearning.InNIPS,2016.11\f", "award": [], "sourceid": 3760, "authors": [{"given_name": "Jacob", "family_name": "Gardner", "institution": "Cornell University"}, {"given_name": "Geoff", "family_name": "Pleiss", "institution": "Cornell University"}, {"given_name": "Kilian", "family_name": "Weinberger", "institution": "Cornell University"}, {"given_name": "David", "family_name": "Bindel", "institution": "Cornell University"}, {"given_name": "Andrew", "family_name": "Wilson", "institution": "Cornell University"}]}