{"title": "Synthesizing the preferred inputs for neurons in neural networks via deep generator networks", "book": "Advances in Neural Information Processing Systems", "page_first": 3387, "page_last": 3395, "abstract": "Deep neural networks (DNNs) have demonstrated state-of-the-art results on many pattern recognition tasks, especially vision classification problems. Understanding the inner workings of such computational brains is both fascinating basic science that is interesting in its own right---similar to why we study the human brain---and will enable researchers to further improve DNNs. One path to understanding how a neural network functions internally is to study what each of its neurons has learned to detect. One such method is called activation maximization, which synthesizes an input (e.g. an image) that highly activates a neuron. Here we dramatically improve the qualitative state of the art of activation maximization by harnessing a powerful, learned prior: a deep generator network. The algorithm (1) generates qualitatively state-of-the-art synthetic images that look almost real, (2) reveals the features learned by each neuron in an interpretable way, (3) generalizes well to new datasets and somewhat well to different network architectures without requiring the prior to be relearned, and (4) can be considered as a high-quality generative method (in this case, by generating novel, creative, interesting, recognizable images).", "full_text": "SynthesizingthepreferredinputsforneuronsinneuralnetworksviadeepgeneratornetworksAnhNguyenanguyen8@uwyo.eduAlexeyDosovitskiydosovits@cs.uni-freiburg.deJasonYosinskijason@geometric.aiThomasBroxbrox@cs.uni-freiburg.deJeffClunejeffclune@uwyo.eduAbstractDeepneuralnetworks(DNNs)havedemonstratedstate-of-the-artresultsonmanypatternrecognitiontasks,especiallyvisionclassi\ufb01cationproblems.Understandingtheinnerworkingsofsuchcomputationalbrainsisbothfascinatingbasicsciencethatisinterestinginitsownright\u2014similartowhywestudythehumanbrain\u2014andwillenableresearcherstofurtherimproveDNNs.Onepathtounderstandinghowaneuralnetworkfunctionsinternallyistostudywhateachofitsneuronshaslearnedtodetect.Onesuchmethodiscalledactivationmaximization(AM),whichsynthesizesaninput(e.g.animage)thathighlyactivatesaneuron.Herewedramaticallyimprovethequalitativestateoftheartofactivationmaximizationbyharnessingapowerful,learnedprior:adeepgeneratornetwork(DGN).Thealgorithm(1)generatesqualitativelystate-of-the-artsyntheticimagesthatlookalmostreal,(2)revealsthefeatureslearnedbyeachneuroninaninterpretableway,(3)generalizeswelltonewdatasetsandsomewhatwelltodifferentnetworkarchitectureswithoutrequiringthepriortoberelearned,and(4)canbeconsideredasahigh-qualitygenerativemethod(inthiscase,bygeneratingnovel,creative,interesting,recognizableimages).1IntroductionandRelatedWorkUnderstandinghowthehumanbrainworkshasbeenalong-standingquestinhumanhistory.Neuro-scientistshavediscoveredneuronsinhumanbrainsthatselectively\ufb01reinresponsetospeci\ufb01c,abstractconceptssuchasHalleBerryorBillClinton,sheddinglightonthequestionofwhetherlearnedneuralcodesarelocalvs.distributed[1].Theseneuronswereidenti\ufb01edby\ufb01ndingthepreferredstimuli(here,images)thathighlyexciteaspeci\ufb01cneuron,whichwasaccomplishedbyshowingsubjectsmanydifferentimageswhilerecordingatargetneuron\u2019sactivation.Suchneuronsaremultifaceted:forexample,the\u201cHalleBerryneuron\u201drespondstoverydifferentstimulirelatedtotheactress\u2014frompicturesofherface,topicturesofherincostume,totheword\u201cHalleBerry\u201dprintedastext[1].Inspiredbysuchneuroscienceresearch,weareinterestedinsheddinglightintotheinnerworkingsofDNNsby\ufb01ndingthepreferredinputsforeachoftheirneurons.Astheneuroscientistsdid,onecouldsimplyshowthenetworkalargesetofimagesandrecordasetofimagesthathighlyactivateaneuron[2].However,thatmethodhasdisadvantagesvs.synthesizingpreferredstimuli:1)itrequiresadistributionofimagesthataresimilartothoseusedtotrainthenetwork,whichmaynotbeknown(e.g.whenprobingatrainednetworkwhenonedoesnotknowwhichdatawereusedtotrainit);2)eveninsuchadataset,manyinformativeimagesthatwouldactivatetheneuronmaynotexistbecausetheimagespaceisvast[3];3)withrealimages,itisunclearwhichoftheirfeaturesaneuronhaslearned:forexample,ifaneuronisactivatedbyapictureofalawnmowerongrass,itisunclearifit30thConferenceonNeuralInformationProcessingSystems(NIPS2016),Barcelona,Spain.\fFigure1:ImagessynthesizedfromscratchtohighlyactivateoutputneuronsintheCaffeNetdeepneuralnetwork,whichhaslearnedtoclassifydifferenttypesofImageNetimages.\u2018caresabout\u2019thegrass,butifanimagesynthesizedtohighlyactivatethelawnmowerneuroncontainsgrass(asinFig.1),wecanbemorecon\ufb01denttheneuronhaslearnedtopayattentiontothatcontext.Synthesizingpreferredstimuliiscalledactivationmaximization[4\u20138,3,9].Itstartsfromarandomimageanditerativelycalculatesviabackpropagationhowthecolorofeachpixelintheimageshouldbechangedtoincreasetheactivationofaneuron.Previousstudieshaveshownthatdoingsowithoutbiasingtheimagesproducedcreatesunrealistic,uninterpretableimages[5,3],becausethesetofallpossibleimagesissovastthatitispossibletoproduce\u2018fooling\u2019imagesthatexciteaneuron,butdonotresemblethenaturalimagesthatneuronhaslearnedtodetect.Instead,wemustconstrainoptimizationtogenerateonlysyntheticimagesthatresemblenaturalimages[6].Attemptingthatisaccomplishedbyincorporatingnaturalimagepriorsintotheobjectivefunction,whichhasbeenshowntosubstantiallyimprovetherecognizabilityoftheimagesgenerated[7,6,9].Manyhand-designednaturalimagepriorshavebeenexperimentallyshowntoimproveimagequalitysuchas:Gaussianblur[7],\u03b1-norm[5,7,8],totalvariation[6,9],jitter[10,6,9],data-drivenpatchpriors[8],center-biasregularization[9],andinitializingfrommeanimages[9].Insteadofhand-designingsuchpriors,inthispaper,weproposetouseasuperior,learnednaturalimageprior[11]akintoagenerativemodelofimages.Thispriorallowsustosynthesizehighlyhuman-interpretablepreferredstimuli,givingadditionalinsightintotheinnerfunctioningofnetworks.Whilethereisnowaytorigorouslymeasurehuman-interpretability,aproblemthatalsomakesquantitativelyassessinggenerativemodelsnear-impossible[12],weshouldnotceasescienti\ufb01cworkonimprovingqualitativeresultssimplybecausehumansmustsubjectivelyevaluatethem.Learninggenerativemodelsofnaturalimageshasbeenalong-standinggoalinmachinelearning[13].Manytypesofneuralnetworkmodelsexist,includingprobabilistic[13],auto-encoder[13],stochastic[14]andrecurrentnetworks[13].However,theyaretypicallylimitedtorelativelylow-dimensionalimagesandnarrowlyfocuseddatasets.Recently,advancesinnetworkarchitecturesandtrainingmethodsenabledthegenerationofhigh-dimensionalrealisticimages[15,16,11].MostoftheseworksarebasedonGenerativeAdversarialNetworks(GAN)[17],whichtrainstwomodelssimultaneously:agenerativemodelGtocapturethedatadistribution,andadiscriminativemodelDtoestimatestheprobabilitythatasamplecamefromthetrainingdataratherthanG.ThetrainingobjectiveforGistomaximizetheprobabilityofDmakingamistake.RecentlyDosovitskiyandBrox[11]trainednetworkscapableofgeneratingimagesfromhighlycompressedfeaturerepresentations,bycombininganauto-encoder-styleapproachwithGAN\u2019sadversarialtraining.Weharnesstheseimagegeneratornetworksaspriorstoproducesyntheticpreferredimages.Thesegeneratornetworksarecloseto,butnottrue,generativemodelsbecausetheyaretrainedwithoutimposinganyprioronthehiddendistributionasinvariationalauto-encoders[14]orGANs[17],andwithouttheadditionofnoiseasindenoisingauto-encoders[18].Thus,thereisnonaturalsamplingprocedurenoranimplicitdensityfunctionoverthedataspace.TheimagegeneratorDNNthatweuseasaprioristrainedtotakeinacode(e.g.vectorofscalars)andoutputasyntheticimagethatlooksasclosetorealimagesfromtheImageNetdataset[19]aspossible.ToproduceapreferredinputforaneuroninagivenDNNthatwewanttovisualize,weoptimizeintheinputcodespaceoftheimagegeneratorDNNsothatitoutputsanimagethatactivates2\f...Imagebananaconvertible.....Deep \u00a0generator \u00a0network(prior)DNN \u00a0being \u00a0visualizedcandleCodeForward \u00a0and \u00a0backward \u00a0passesu9u2u1c1c2fc6fc7fc8fc6c3c4c5...upconvolutionalconvolutionalFigure2:Tosynthesizeapreferredinputforatargetneuronh(e.g.the\u201ccandle\u201dclassoutputneuron),weoptimizethehiddencodeinput(redbar)ofadeepimagegeneratornetwork(DGN)toproduceanimagethathighlyactivatesh.Intheexampleshown,theDGNisanetworktrainedtoinvertthefeaturerepresentationsoflayerfc6ofCaffeNet.ThetargetDNNbeingvisualizedcanbeadifferentnetwork(withadifferentarchitectureandortrainedondifferentdata).Thegradientinformation(blue-dashedline)\ufb02owsfromthelayercontaininghinthetargetDNN(here,layerfc8)allthewaythroughtheimagebacktotheinputcodelayeroftheDGN.NotethatboththeDGNandtargetDNNbeingvisualizedhave\ufb01xedparameters,andoptimizationonlychangestheDGNinputcode(red).theneuronofinterest(Fig.2).Ourmethodrestrictsthesearchtoonlythesetofimagesthatcanbedrawnbytheprior,whichprovidesastrongbiasestowardrealisticvisualizations.Becauseouralgorithmusesadeepgeneratornetworktoperformactivationmaximization,wecallitDGN-AM.2MethodsNetworksthatwevisualize.Wedemonstrateourvisualizationmethodonavarietyofdifferentnetworks.Forreproducibility,weusepretrainedmodelsfreelyavailableinCaffeortheCaffeModelZoo[20]:CaffeNet[20],GoogleNet[21],andResNet[22].Theyrepresentdifferentconvnetarchitecturestrainedonthe\u223c1.3-million-image2012ImageNetdataset[23,19].OurdefaultDNNisCaffeNet[20],aminorvariantofthecommonAlexNetarchitecture[24]withsimilarperformance[20].Thelastthreefullyconnectedlayersofthe8-layerCaffeNetarecalledfc6,fc7andfc8(Fig.2).fc8isthelastlayer(presoftmax)andhas1000outputs,oneforeachImageNetclass.Imagegeneratornetwork.WedenotetheDNNwewanttovisualizeby\u03a6.Insteadofpreviousworks,whichdirectlyoptimizedanimagesothatithighlyactivatesaneuronhin\u03a6andoptionallysatis\ufb01eshand-designedpriorsembeddedinthecostfunction[5,7,9,6],hereweoptimizeintheinputcodeofanimagegeneratornetworkGsuchthatGoutputsanimagethathighlyactivatesh.ForGweusenetworksmadepubliclyavailableby[11]thathavebeentrainedwiththeprinciplesofGANs[17]toreconstructimagesfromhidden-layerfeaturerepresentationswithinCaffeNet[20].HowGistrainedincludesimportantdifferencesfromtheoriginalGANcon\ufb01guration[17].Herewecanonlybrie\ufb02ysummarizethetrainingprocedure;pleasesee[11]formoredetails.Thetrainingprocessinvolvesfourconvolutionalnetworks:1)a\ufb01xedencodernetworkEtobeinverted,2)ageneratornetworkG,3)a\ufb01xed\u201ccomparator\u201dnetworkCand4)adiscriminatorD.GistrainedtoinvertafeaturerepresentationextractedbythenetworkE,andhastosatisfythreeobjectives:1)forafeaturevectoryi=E(xi),thesynthesizedimageG(yi)hastobeclosetotheoriginalimagexi;2)thefeaturesoftheoutputimageC(G(yi))havetobeclosetothoseoftherealimageC(xi);3)DshouldbeunabletodistinguishG(yi)fromrealimages.TheobjectiveforDistodiscriminatebetweensyntheticimagesG(yi)andrealimagesxiasintheoriginalGAN[17].Inthispaper,theencoderEisCaffeNettruncatedatdifferentlayers.WedenoteCaffeNettruncatedatlayerlbyEl,andthenetworktrainedtoinvertElbyGl.The\u201ccomparator\u201dCisCaffeNetuptolayerpool5.Disaconvolutionalnetworkwith5convolutionaland2fullyconnectedlayers.Gisanupconvolutional(akadeconvolutional)architecture[15]with9upconvolutionaland3fullyconnectedlayers.Detailedarchitecturesareprovidedin[11].3\fSynthesizingthepreferredimagesforaneuron.Intuitively,wesearchintheinputcodespaceoftheimagegeneratormodelGto\ufb01ndacodeysuchthatG(y)isanimagethatproduceshighactivationofthetargetneuronhintheDNN\u03a6thatwewanttovisualize(i.e.optimizationmaximizes\u03a6h(G(y))).RecallthatGlisageneratornetworktrainedtoreconstructimagesfromthel-thlayerfeaturesofCaffeNet.Formally,andincludingaregularizationterm,wemayposetheactivationmaximizationproblemas\ufb01ndingacodebylsuchthat:byl=argmaxyl(\u03a6h(Gl(yl))\u2212\u03bbkylk)(1)Empirically,wefoundasmallamountofL2regularization(\u03bb=0.005)worksbest.Wealsocomputetheactivationrangeforeachneuroninthesetofcodes{yli}computedbyrunningvalidationsetimagesthroughEl.Wethenclipeachneuroninbyltobewithintheactivationrangeof[0,3\u03c3],where\u03c3isonestandarddeviationaroundthemeanactivation(theactivationislowerboundedat0duetotheReLUnonlinearitiesthatexistatthelayerswhosecodesweoptimize).Thisclippingactsasaprimitiveprioronthecodespaceandsubstantiallyimprovestheimagequality.Infuturework,weplantolearnthispriorviaaGANorothergenerativemodel.Becausethetruegoalofactivationmaximizationistogenerateinterpretablepreferredstimuliforeachneuron,weperformedrandomsearchinthehyperparameterspaceconsistingofL2weight\u03bb,numberofiterations,andlearningrate.Wechosethehyperparametersettingsthatproducedthehighestqualityimages.Wenotethatwefoundnocorrelationbetweentheactivationofaneuronandtherecognizabilityofitsvisualization.Ourcodeandparametersareavailableathttp://EvolvingAI.org/synthesizing.3Results3.1ComparisonbetweenpriorstrainedtoinvertfeaturesfromdifferentlayersSinceageneratormodelGlcouldbetrainedtoinvertfeaturerepresentationsofanarbitrarylayerlofE,wesampledl={3,5,6,7}toexploretheimpactonthischoiceandidentifyqualitativelywhichproducesthebestimages.Here,theDNNtovisualize\u03a6isthesameastheencoderE(CaffeNet),buttheycanbedifferent(asshownbelow).TheGlnetworksarefrom[11].ForeachGlnetworkwechosethehyperparametersettingsfromarandomsamplethatgavethebestqualitativeresults.Optimizingcodesfromtheconvolutionallayers(l=3,5)typicallyyieldshighlyrepeatedfragments,whereasoptimizingfully-connectedlayercodesproducesmuchmorecoherentglobalstructure(Fig.S13).Interestingly,previousstudieshaveshownthatGtrainedtoinvertlower-layercodes(smallerl)resultsinfarbetterreconstructionsthanhigher-layercodes[25,6].Thatcanbeexplainedbecausethoselow-levelcodescomefromnaturalimages,andcontainmoreinformationaboutimagedetailsthanmoreabstract,high-levelcodes.Foractivationmaximization,however,wearesynthesizinganentirelayercodefromscratch.WehypothesizethatthisprocessworksworseforGlpriorswithsmallerlbecauseeachfeatureinlow-levelcodeshasasmall,localreceptive\ufb01eld.Optimizationthushastoindependentlytunefeaturesthroughouttheimagewithoutknowingtheglobalstructure.Forexample,isitanimageofoneorfourrobins?Becausefully-connectedlayershaveinformationfromallareasoftheimage,theyrepresentinformationsuchasthenumber,location,size,etc.ofanobject,andthusallthepixelscanbeoptimizedtowardthisagreeduponstructure.Anorthogonal,non-mutually-exclusivehypothesisisthatthecodespaceataconvolutionallayerismuchmorehigh-dimensional,makingithardertooptimize.Wefoundthatoptimizinginthefc6codespaceproducesthebestvisualizations(Figs.1&S13).WethususethisG6DGNasthedefaultpriorfortheexperimentsintherestofthepaper.Inaddition,ourimagesqualitativelyappeartobethemostrealistic-lookingcomparedtovisualizationsfromallpreviousmethods(Fig.S17).Ourresultrevealsthatagreatamountof\ufb01nedetailandglobalstructurearecapturedbytheDNNevenatthelastoutputlayer.This\ufb01ndingisincontrasttoaprevioushypothesisthatDNNstrainedwithsupervisedlearningoftenignoreanobject\u2019sglobalstructure,andonlylearndiscriminativefeaturesperclass(e.g.colorortexture)[3].Section3.5providesevidencethatthisglobalstructuredoesnotcomefromtheprior.Totestwhetherourmethodmemorizesthetrainingsetimages,weretrievedtheclosestimagesfromthetrainingsetforeachofsamplesyntheticimages.Speci\ufb01cally,foreachsyntheticimageforanoutputneuronY(e.g.lipstick),we\ufb01ndanimageamongthesameclassYwiththelowestEuclideandistanceinpixelspace,asdoneinpreviousworks[17],butalsoineachofthe8codespacesofthe4\fencoderDNN.Whilethisisamuchhardertestthancomparingtoanearestneighborfoundamongtheentiredataset,wefoundnoevidencethatourmethodmemorizesthetrainingsetimages(Fig.S22).Webelieveevaluatingsimilarityinthespacesofdeeprepresentations,whichbettercapturesemanticaspectsofimages,isamoreinformativeapproachcomparedtoevaluatingonlyinthepixelspace.3.2DoesthelearnedpriortrainedonImageNetgeneralizetootherdatasets?WetestwhetherthesameDNNprior(G6)thatwastrainedoninvertingthefeaturerepresentationsofImageNetimagesgeneralizestoenablevisualizingDNNstrainedondifferentdatasets.Speci\ufb01cally,wetargettheoutputneuronsoftwoDNNsdownloadedfromCaffeModelZoo[20]):(1)AnAlexNetDNNthatwastrainedonthe2.5-million-imageMITPlacesdatasettoclassify205typesofplaceswith50.1%accuracy[26].(2)AhybridarchitectureofCaffeNetandthenetworkin[2]createdby[27]toclassifyactionsinvideosbyprocessingeachframeofthevideoseparately.Thedatasetconsistsof13,320videoscategorizedinto101humanactionclasses.ForDNN1,thepriortrainedonImageNetimagesgeneralizeswelltothecompletelydifferentMITPlacesdataset(Fig.3).ThisresultsuggeststhepriortrainedonImageNetwillgeneralizetoothernaturalimagedatasets,atleastifthearchitectureoftheDNNtobevisualized\u03a6isthesameasthearchitectureoftheencodernetworkEfromwhichthegeneratormodelGwastrainedtoinvertfeaturerepresentations.ForDNN2:thepriorgeneralizestoproducedecentresults;however,theimagesarenotqualitativelyassharpandclearasforDNN1(Fig.4).Wehavetwoorthogonalhypothesesforwhythishappens:1)\u03a6(theDNNfrom[27])isaheavilymodi\ufb01edversionofE(CaffeNet);2)thetwotypesofimagesaretoodifferent:theprimarilyobject-centricImageNetdatasetvs.theUCF-101dataset,whichfocusesonhumansperformingactions.Sec.3.3returnstothe\ufb01rsthypothesisregardinghowthesimilaritybetween\u03a6andEaffectstheimagequalityOverall,thepriortrainedwithaCaffeNetencodergeneralizeswelltovisualizingotherDNNsofthesameCaffeNetarchitecturetrainedondifferentdatasets.Figure3:PreferredstimuliforoutputunitsofanAlexNetDNNtrainedontheMITPlacesdataset[26],showingthattheImageNet-trainedpriorgeneralizeswelltoadatasetcomprisedofimagesofscenes.3.3Doesthelearnedpriorgeneralizetovisualizingdifferentarchitectures?WehaveshownthatwhentheDNNtobevisualized\u03a6isthesameastheencoderE,theresultantvisualizationsarequiterealisticandrecognizable(Sec.3.1).Tovisualizeadifferentnetworkarchitecture\u02c6\u03a6,onecouldtrainanew\u02c6Gtoinvert\u02c6\u03a6featurerepresentations.However,traininganewGDGNforeveryDNNwewanttovisualizeiscomputationallycostly.Here,wetestwhetherthesameDGNpriortrainedonCaffeNet(G6)canbeusedtovisualizetwostate-of-the-artDNNsthatarearchitecturallydifferentfromCaffeNet,butweretrainedonthesameImageNetdataset.BothweredownloadedfromCaffeModelZooandhavesimilaraccuracyscores:(a)GoogLeNetisa22-layernetworkandhasatop-5accuracyof88.9%[21];(b)ResNetisanewtypeofverydeeparchitecturewithskipconnections[22].Wevisualizea50-layerResNetthathasatop-5accuracyof93.3%.[22].DGN-AMproducesthebestimagequalitywhen\u03a6=E,andthevisualizationqualitytendstodegradeasthe\u03a6architecturebecomesmoredistantfromE(Fig.5,toprow;GoogleLeNetiscloser5\fFigure4:Preferredimagesforoutputunitsofaheavilymodi\ufb01edversionoftheAlexNetarchitecturetrainedtoclassifyvideosinto101classesofhumanactivities[27].Here,weoptimizeasinglepreferredimageperneuronbecausetheDNNonlyclassi\ufb01essingleframes(wholevideoclassi\ufb01cationisdonebyaveragingscoresacrossallvideoframes).inarchitecturetoCaffeNetthanResNet).Analternativehypothesisisthatthenetworkdepthimpairsgradientpropagationduringactivationmaximization.Inanycase,trainingageneralpriorforactivationmaximizationthatgeneralizeswelltodifferentnetworkarchitectures,whichwouldenablecomparativeanalysisbetweennetworks,remainsanimportant,openchallenge.Figure5:DGN-AMproducesthebestimagequalitywhentheDNNbeingvisualized\u03a6isthesameastheencoderE(here,CaffeNet),asinthetoprow,anddegradeswhen\u03a6isdifferentfromE.3.4Doesthelearnedpriorgeneralizetovisualizinghiddenneurons?VisualizingthehiddenneuronsinanImageNetDNN.Previousvisualizationtechniqueshaveshownthatlow-levelneuronsdetectsmall,simplepatternssuchascornersandtextures[2,9,7],mid-levelneuronsdetectsingleobjectslikefacesandchairs[9,2,28,7],butthatvisualizationsofhiddenneuronsinfully-connectedlayersarealienanddif\ufb01culttointerpret[9].SinceDGNwastrainedtoinvertthefeaturerepresentationsofreal,full-sizedImageNetimages,onepossibilityisthatthispriormaynotgeneralizetoproducingpreferredimagesforsuchhiddenneuronsbecausetheyareoftensmaller,differentintheme,andordonotresemblerealobjects.To\ufb01ndout,wesynthesizedpreferredimagesforthehiddenneuronsatalllayersandcomparethemtoimagesproducedbythemultifacetedfeaturevisualizationmethodfrom[9],whichharnesseshand-designedpriorsoftotalvariationandmeanimageinitialization.TheDNNbeingvisualizedisthesameasin[9](theCaffeNetarchitecturewithweightsfrom[7]).Theside-by-sidecomparison(Fig.S14)showsthatbothmethodsoftenagreeonthefeaturesthataneuronhaslearnedtodetect.However,overallDGN-AMproducesmorerealistic-lookingcolorandtexture,despitenotrequiringoptimizationtobeseededwithaveragesofrealimages,thusimprovingourabilitytolearnwhatfeatureeachhiddenneuronhaslearned.Anexceptionisforthefacesof6\fhumanandotheranimals,whichDGN-AMdoesnotvisualizewell(Fig.S14,3rdunitonlayer6;1stunitonlayer5;and6thunitonlayer4).VisualizingthehiddenneuronsinaDeepSceneDNN.Recently,Zhouetal.[28]foundthatobjectdetectorsautomaticallyemergeintheintermediatelayersofaDNNaswetrainittoclassifyscenecategories.Toidentifywhatahiddenneuroncaresaboutinagivenimage,theydenselyslideanoccludingpatchacrosstheimageandrecordwhenactivationdrops.Theactivationchangesarethenaggregatedtosegmentouttheexactregionthatleadstothehighneuralactivation(Fig.6,thehighlightedregionineachimage).Toidentifythesemanticsofthesesegmentations,humansarethenshownacollectionofsegmentedimagesforaspeci\ufb01cneuronandaskedtolabelwhattypesofimagefeaturesactivatethatneuron[28].Here,wecompareourmethodtotheirsonanAlexNetDNNtrainedtoclassify205categoriesofscenesfromtheMITPlacesdataset(describedinSec.3.2).ThepriorlearnedonImageNetgeneralizestovisualizingthehiddenneuronsofaDNNtrainedontheMITPlacesdataset(Fig.S15).Interestingly,ourvisualizationsproducesimilarresultstothemethodin[28]thatrequiresshowingeachneuronalarge,externaldatasetofimagestodiscoverwhatfeatureeachneuronhaslearnedtodetect(Fig.6).Sometimes,DGN-AMrevealsadditionalinformation:aunitthat\ufb01resforTVscreensalso\ufb01resforpeopleonTV(Fig.6,unit106).Overall,DGN-AMthusnotonlygeneralizeswelltoadifferentdataset,butalsoproducesvisualizationsthatqualitativelyfallwithinthehuman-providedcategoriesofwhattypeofimagefeatureseachneuronrespondsto[28].Figure6:Visualizationsofexamplehiddenneuronsatlayer5ofanAlexNetDNNtrainedtoclassifycategoriesofscenesfrom[28].Foreachunit:wecomparethetwovisualizationsproducedbyamethodfrom[28](left)totwovisualizationsproducedbyourmethod(right).Thelefttwoimagesarerealimages,eachhighlightingaregionthathighlyactivatestheneuron,andhumansprovidetextlabelsdescribingthecommonthemeinthehighlightedregions.Oursyntheticimagesenablethesameconclusionregardingwhatfeatureahiddenneuronhaslearned.Anextendedversionofthis\ufb01gurewithmoreunitsisinFig.S16.Bestviewedelectronicallywithzoom.3.5Dothesynthesizedimagesteachuswhattheneuronspreferorwhatthepriorprefers?Visualizingneuronstrainedonunseen,modi\ufb01edimages.WehaveshownthatDGN-AMcangeneratepreferredimagestimuliwithrealisticcolorsandcoherentglobalstructuresbyharnessingtheDGN\u2019sstrong,learned,naturalimageprior(Fig.1).Towhatextentdotheglobalstructure,naturalcolors,andsharptextures(e.g.ofthebramblingbird,Fig.1)re\ufb02ectthefeatureslearnedbythe\u201cbrambling\u201dneuronvs.thosepreferredbytheprior?Toinvestigatethat,wetrain3differentDNNs:oneonimagesthathavelessglobalstructure,oneonimagesofnon-realisticcolors,andoneonblurryimages.WetestwhetherDGN-AMwiththesamepriorproducesvisualizationsthatre\ufb02ectthesemodi\ufb01ed,unrealisticfeatures.Speci\ufb01cally,wetrain3differentDNNsfollowingCaffeNetarchitecturetodiscriminate2000classes.The\ufb01rst1000classescontainregularImageNetimages,andthe2nd1000classescontainmodi\ufb01edImageNetimages.Weperform3typesofmodi\ufb01cations:1)wecutupeachimageintoquartersandre-stitchthembackinarandomorder(Fig.S19);2)weconvertregularRGBintoBRGimages(Fig.S20);3)webluroutimageswithGaussianblurwithradiusof3(Fig.S21).Wevisualizebothgroupsofoutputneurons(thosetrainedon1000regularvs.1000modi\ufb01edclasses)ineachDNN(Figs.S19,S20,&S21).Thevisualizationsfortheneuronsthataretrainedonregularimagesoftenshowcoherentglobalstructures,realistic-lookingcolorsandsharpness.Incontrast,thevisualizationsforneuronsthataretrainedonmodi\ufb01edimagesindeedshowcut-upobjects(Fig.S19),imagesinBRGcolorspace(Fig.S20),andobjectswithwashedoutdetails(Fig.S21).TheresultsshowthatDGN-AMvisualizationsdocloselyre\ufb02ectthefeatureslearnedbyneuronsfromthedataandthatthesepropertiesarenotexclusivelyproducedbytheprior.7\fWhydovisualizationsofsomeneuronsnotshowcanonicalimages?WhilemanyDGN-AMvisualizationsshowglobalstructure(e.g.asingle,centeredtablelamp,Fig.1);someothersdonot(e.g.blobsoftexturesinsteadofadogwith4legs,Fig.S18)orotherwisearenon-canonical(e.g.aschoolbusofftothesideofanimage,Fig.S7).Sec.S5describesourexperimentsinvestigatingwhetherthisisashortcomingofourmethodorwhetherthesenon-canonicalvisualizationsre\ufb02ectsomepropertyoftheneurons.TheresultssuggestthatDGN-AMcanaccuratelyvisualizeaclassofimagesiftheimagesofthatsetaremostlycanonical,andthereasonwhythevisualizationsforsomeneuronslackglobalstructureorarenotcanonicalisthatthesetofimagesthatneuronhaslearnedtodetectareoftendiverse(multi-modal),insteadofhavingcanonicalpose.Moreresearchisneededintomultifacetedfeaturevisualizationalgorithmsthatseparatelyvisualizeeachtypeofimagethatactivatesaneuron[9].3.6OtherapplicationsofourproposedmethodDGN-AMcanalsobeusefulforavarietyofotherimportanttasks.Webrie\ufb02ydescribeourexperimentsforthesetasks,andreferthereadertothesupplementarysectionformoreinformation.1.Oneadvantageofsynthesizingpreferredimagesisthatwecanwatchhowfeaturesevolveduringtrainingtobetterunderstandwhatoccursduringdeeplearning.Doingsoalsotestswhetherthelearnedprior(trainedtoinvertfeaturesfromawell-trainedencoder)generalizestovisualizingunder\ufb01tandover\ufb01tnetworks.TheresultssuggestthatthevisualizationqualityisindicativeofaDNN\u2019svalidationaccuracytosomeextent,andthelearnedpriorisnotoverlyspecializedtothewell-trainedencoderDNN.SeeSec.S6formoredetails.2.OurmethodforsynthesizingpreferredimagescouldnaturallybeappliedtosynthesizepreferredvideosforanactivityrecognitionDNNtobetterunderstandhowitworks.Forexample,wefoundthatastate-of-the-artDNNclassi\ufb01esvideoswithoutpayingattentiontotemporalinformationacrossvideoframes(Sec.S7).3.Ourmethodcanbeextendedtoproducecreative,originalartbysynthesizingimagesthatactivatetwoneuronsatthesametime(Sec.S8).4DiscussionandConclusionWehaveshownthatactivationmaximization\u2014synthesizingthepreferredinputsforneuronsinneuralnetworks\u2014viaalearnedpriorintheformofadeepgeneratornetworkisafruitfulapproach.DGN-AMproducesthemostrealistic-looking,andthusinterpretable,preferredimagestodate,makingitqualitativelythestateoftheartinactivationmaximization.Thevisualizationsitsynthesizesfromscratchimproveourabilitytounderstandwhichfeaturesaneuronhaslearnedtodetect.Notonlydotheimagescloselyre\ufb02ectthefeatureslearnedbyaneuron,buttheyarevisuallyinteresting.WehaveexploredavarietyofwaysthatDGN-AMcanhelpusunderstandtrainedDNNs.Infuturework,DGN-AMoritslearnedpriorcoulddramaticallyimproveourabilitytosynthesizeanimagefromatextdescriptionofit(e.g.bysynthesizingtheimagethatactivatesacertaincaption)orcreatemorerealistic\u201cdeepdream\u201d[10]images.Additionally,thatthepriorusedinthispaperdoesnotgeneralizeequallywelltoDNNsofdifferentarchitecturesmotivatesresearchintohowtotrainsuchageneralprior.SuccessfullydoingsocouldenableinformativecomparativeanalysesbetweentheinformationtransformationsthatoccurwithindifferenttypesofDNNs.AcknowledgmentsTheauthorswouldliketothankYoshuaBengioforhelpfuldiscussionsandBoleiZhouforprovidingimagesforourstudy.JeffClunewassupportedbyanNSFCAREERaward(CAREER:1453549)andahardwaredonationfromtheNVIDIACorporation.JasonYosinskiwassupportedbytheNASASpaceTechnologyResearchFellowshipandNSFgrant1527232.AlexeyDosovitskiyandThomasBroxacknowledgefundingbytheERCStartingGrantVideoLearn(279401).References[1]R.Q.Quiroga,L.Reddy,G.Kreiman,C.Koch,andI.Fried.Invariantvisualrepresentationbysingleneuronsinthehumanbrain.Nature,435(7045):1102\u20131107,2005.8\f[2]M.D.ZeilerandR.Fergus.Visualizingandunderstandingconvolutionalnetworks.InComputerVision\u2013ECCV2014,pages818\u2013833.Springer,2014.[3]A.Nguyen,J.Yosinski,andJ.Clune.Deepneuralnetworksareeasilyfooled:Highcon\ufb01dencepredictionsforunrecognizableimages.InComputerVisionandPatternRecognition(CVPR),2015.[4]D.Erhan,Y.Bengio,A.Courville,andP.Vincent.Visualizinghigher-layerfeaturesofadeepnetwork.Dept.IRO,Universit\u00e9deMontr\u00e9al,Tech.Rep,4323,2009.[5]K.Simonyan,A.Vedaldi,andA.Zisserman.Deepinsideconvolutionalnetworks:Visualisingimageclassi\ufb01cationmodelsandsaliencymaps.ICLRworkshop,2014.[6]A.MahendranandA.Vedaldi.Visualizingdeepconvolutionalneuralnetworksusingnaturalpre-images.InComputerVisionandPatternRecognition(CVPR),2016.[7]J.Yosinski,J.Clune,A.Nguyen,T.Fuchs,andH.Lipson.Understandingneuralnetworksthroughdeepvisualization.InDeepLearningWorkshop,ICMLconference,2015.[8]D.Wei,B.Zhou,A.Torrabla,andW.Freeman.Understandingintra-classknowledgeinsidecnn.arXivpreprintarXiv:1507.02379,2015.[9]A.Nguyen,J.Yosinski,andJ.Clune.Multifacetedfeaturevisualization:Uncoveringthedifferenttypesoffeatureslearnedbyeachneuronindeepneuralnetworks.InVisualizationforDeepLearningWorkshop,ICMLconference,2016.[10]A.Mordvintsev,C.Olah,andM.Tyka.Inceptionism:Goingdeeperintoneuralnetworks.GoogleResearchBlog.RetrievedJune,20,2015.[11]A.DosovitskiyandT.Brox.Generatingimageswithperceptualsimilaritymetricsbasedondeepnetworks.InNIPS,2016.[12]L.Theis,A.vandenOord,andM.Bethge.Anoteontheevaluationofgenerativemodels.InICLR,2016.[13]Y.Bengio,I.J.Goodfellow,andA.Courville.Deeplearning.MITPress,2015.[14]D.P.KingmaandM.Welling.Auto-encodingvariationalbayes.InICLR,2014.[15]A.Dosovitskiy,J.TobiasSpringenberg,andT.Brox.Learningtogeneratechairswithconvolutionalneuralnetworks.InComputerVisionandPatternRecognition(CVPR),2015.[16]A.Radford,L.Metz,andS.Chintala.Unsupervisedrepresentationlearningwithdeepconvolutionalgenerativeadversarialnetworks.InICLR,2016.[17]I.Goodfellow,J.Pouget-Abadie,M.Mirza,B.Xu,D.Warde-Farley,S.Ozair,A.Courville,andY.Bengio.Generativeadversarialnets.InNIPS,2014.[18]G.AlainandY.Bengio.Whatregularizedauto-encoderslearnfromthedata-generatingdistribution.TheJournalofMachineLearningResearch,15(1):3563\u20133593,2014.[19]O.Russakovskyetal.Imagenetlargescalevisualrecognitionchallenge.IJCV,115(3):211\u2013252,2015.[20]Y.Jia,E.Shelhamer,J.Donahue,S.Karayev,J.Long,R.Girshick,S.Guadarrama,andT.Darrell.Caffe:Convolutionalarchitectureforfastfeatureembedding.arXivpreprintarXiv:1408.5093,2014.[21]C.Szegedy,W.Liu,Y.Jia,P.Sermanet,S.Reed,D.Anguelov,D.Erhan,V.Vanhoucke,andA.Rabinovich.Goingdeeperwithconvolutions.InComputerVisionandPatternRecognition(CVPR),2015.[22]K.He,X.Zhang,S.Ren,andJ.Sun.Deepresiduallearningforimagerecognition.InIEEEConferenceonComputerVisionandPatternRecognition(CVPR),2016.[23]J.Dengetal.Imagenet:Alarge-scalehierarchicalimagedatabase.InCVPR,2009.[24]A.Krizhevsky,I.Sutskever,andG.E.Hinton.Imagenetclassi\ufb01cationwithdeepconvolutionalneuralnetworks.InAdvancesinneuralinformationprocessingsystems,pages1097\u20131105,2012.[25]A.DosovitskiyandT.Brox.Invertingvisualrepresentationswithconvolutionalnetworks.InCVPR,2016.[26]B.Zhou,A.Lapedriza,J.Xiao,A.Torralba,andA.Oliva.Learningdeepfeaturesforscenerecognitionusingplacesdatabase.InAdvancesinneuralinformationprocessingsystems,2014.[27]J.Donahue,L.A.Hendricks,S.Guadarrama,M.Rohrbach,etal.Long-termrecurrentconvolutionalnetworksforvisualrecognitionanddescription.InComputerVisionandPatternRecognition,2015.[28]B.Zhou,A.Khosla,A.Lapedriza,A.Oliva,andA.Torralba.Objectdetectorsemergeindeepscenecnns.InInternationalConferenceonLearningRepresentations(ICLR),2015.9\f", "award": [], "sourceid": 1685, "authors": [{"given_name": "Anh", "family_name": "Nguyen", "institution": "University of Wyoming"}, {"given_name": "Alexey", "family_name": "Dosovitskiy", "institution": "University of Freiburg"}, {"given_name": "Jason", "family_name": "Yosinski", "institution": "Cornell"}, {"given_name": "Thomas", "family_name": "Brox", "institution": "University of Freiburg"}, {"given_name": "Jeff", "family_name": "Clune", "institution": "University of Wyoming"}]}