{"title": "SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Models", "book": "Advances in Neural Information Processing Systems", "page_first": 4027, "page_last": 4036, "abstract": "Remarkable achievements have been attained by deep neural networks in various applications. However, the increasing depth and width of such models also lead to explosive growth in both storage and computation, which has restricted the deployment of deep neural networks on resource-limited edge devices. To address this problem, we propose the so-called SCAN framework for networks training and inference, which is orthogonal and complementary to existing acceleration and compression methods. The proposed SCAN firstly divides neural networks into multiple sections according to their depth and constructs shallow classifiers upon the intermediate features of different sections. Moreover, attention modules and knowledge distillation are utilized to enhance the accuracy of shallow classifiers. Based on this architecture, we further propose a threshold controlled scalable inference mechanism to approach human-like sample-specific inference. Experimental results show that SCAN can be easily equipped on various neural networks without any adjustment on hyper-parameters or neural networks architectures, yielding significant performance gain on CIFAR100 and ImageNet. Codes will be released on github soon.", "full_text": "SCAN: A Scalable Neural Networks Framework\n\nTowards Compact and Ef\ufb01cient Models\n\nLinfeng Zhang1, Zhanhong Tan1, Jiebo Song2, Jingwei Chen3\n\nChenglong Bao\u22174, Kaisheng Ma\u22171\n\n1Institute for Interdisciplinary Information Sciences, Tsinghua University\n\n4Yau Mathematical Sciences Center, Tsinghua University\n\n2Institute for Interdisciplinary Information Core Technology, 3HiSilicon\n\n{zhang-lf19, tanzh19}@mails.tsinghua.edu.cn\n\n{kaisheng, clbao}@mail.tsinghua.edu.cn\n\nsongjb@iiisct.com, jean.chenjingwei@hisilicon.com\n\nAbstract\n\nRemarkable achievements have been attained by deep neural networks in vari-\nous applications. However, the increasing depth and width of such models also\nlead to explosive growth in both storage and computation, which has restricted\nthe deployment of deep neural networks on resource-limited edge devices. To\naddress this problem, we propose the so-called SCAN framework for networks\ntraining and inference, which is orthogonal and complementary to existing ac-\nceleration and compression methods. The proposed SCAN \ufb01rstly divides neural\nnetworks into multiple sections according to their depth and constructs shallow\nclassi\ufb01ers upon the intermediate features of different sections. Moreover, atten-\ntion modules and knowledge distillation are utilized to enhance the accuracy of\nshallow classi\ufb01ers. Based on this architecture, we further propose a threshold\ncontrolled scalable inference mechanism to approach human-like sample-speci\ufb01c\ninference. Experimental results show that SCAN can be easily equipped on various\nneural networks without any adjustment on hyper-parameters or neural networks\narchitectures, yielding signi\ufb01cant performance gain on CIFAR100 and ImageNet.\nCodes are be released on https://github.com/ArchipLab-LinfengZhang/\npytorch-scalable-neural-networks.\n\n1\n\nIntroduction\n\nRecently deep learning has evolved to become one of the dominant techniques in areas like natural lan-\nguage processing [6, 2] and computer vision [23, 22]. To achieve higher accuracy, over-parameterized\nmodels [26, 32] have been proposed at the expense of explosive growth in storage and compu-\ntation, which is not available for certain application scenes such as self-driving cars and mobile\nphones. Various techniques have been utilized to address this problem, including pruning [8, 9],\nquantization [4, 24], lightweight neural networks [12] design and knowledge distillation [11, 25, 1].\nAnother rising star in this domain named scalable neural networks has attracted increasing attention\ndue to its effectiveness and \ufb02exibility [37, 20, 17]. The scalability of neural networks refers to its\nability to adjust the trade-offs between response time and accuracy on the \ufb02y. As a result, scalable\nneural networks can always accomplish inference in budgeted and limited time, which is important\nfor real-world applications. Researchers have explored scalability through the lens of depth (layers)\nand width (channels). Built upon DenseNet [14], MSDNet [13] directly trains multiple classi\ufb01ers\n\n\u2217 Corresponding Authors.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\ffrom features at different levels according to their depths. Yu et al. proposes switchable batch\nnormalization which enables neural networks to work with arbitrary channels [34]. However, most\nexisting scalable neural networks still suffer from two drawbacks. Firstly, in MSDNet, multiple\nclassi\ufb01ers which share the same backbone neural network interfere with each other, leading to\naccuracy loss compared with training them individually. Secondly, in slimmable neural networks,\ncomputation of narrow classi\ufb01ers can\u2019t be reused by wide classi\ufb01ers, which means the inference of\nwide classi\ufb01ers has to predict from scratch, increasing the inference time.\nIn this paper, we propose SCAN, a scalable neural network framework to overcome aforementioned\ndif\ufb01culties. By dividing neural networks according to its own depth straightforwardly, the compu-\ntation of each classi\ufb01er can be shared. Through knowledge distillation and attention mechanism,\nmultiple classi\ufb01ers in the same backbone networks can bene\ufb01t from each other instead of creating\nnegative interaction. Substantial experimental results show that SCAN is a generic and effective\nneural networks framework which can be easily equipped with various neural networks without any\nadjustment in architectures or hyper-parameters.\nThe proposed SCAN framework is inspired by human vision systems. When a human being is asked\nto identify some images, most of easy images can be recognized instantly. Only fuzzy or easy-to-mix\nchallenging images require further consideration. In SCAN framework, images which are easy to\nbe classi\ufb01ed are predicted by shallow (shallow classi\ufb01er) classi\ufb01ers, consuming extremely little\ncomputation. Deep classi\ufb01ers only involve in the prediction of challenging samples. Compared to\ntraditional neural networks in which all the samples are treated with equal efforts, SCAN can obtain\na high ratio acceleration through human-like sample-speci\ufb01c dynamic inference.\nThe contributions of this paper are summarized as follows:\n\n\u2022 To the best of our knowledge, this is the \ufb01rst work combining model compression and\nacceleration with attention mechanism, which provides a novel choice for lightweight models\ndesign. Compared to existing lightweight design, the proposed mechanism is more hardware\nfriendly with features of input-speci\ufb01c adaptive networks and reuse of the backbone. To\nverify its effectiveness and generalization, SCAN is evaluated on various neural networks\nand datasets without adjustment in hyper-parameters and networks architectures.\n\n\u2022 Through this method, a two-fold improvement can be achieved on either accuracy or\nacceleration. Firstly, signi\ufb01cant accuracy gain can be observed, especially on the shallow\nclassi\ufb01ers. Secondly, a high ratio of acceleration can be obtained via scalable inference\nmechanism.\n\n2 Related work\n\nAdaptive computation graph: Adaptive computation graph is proposed to attain \ufb02exible and dy-\nnamic neural networks acceleration [13, 15, 29, 31]. Compared with constant and static computation\ngraph, it can meet various demands from diverse application scenes and inputs. SkipNet [29] targets\nto skip redundant layers in over-parameterized models like ResNet. A reinforcement learning based\non auxiliary gating module is proposed to decide whether to skip or not. BlockDrop [31] re\ufb01nes this\nmethod by producing a sample-speci\ufb01c dropping strategy, further enhancing acceleration.\nHowever, serious complication hides behind. Complex construction of skipping or dropping path\nseverely prevents hardware from better coordination, which makes aforementioned algorithm coun-\nterproductive. To address this problem, the proposed SCAN simply divides the neural networks into\n3 or 4 sections, which can be constructed as static graphs individually. The scalability and \ufb02exibility\nonly exist among them instead of inside them, which is not only harmonious with hardware but also\ngains a higher acceleration ratio.\nAttention mechanism: Attention mechanism of neural networks has been extensively utilized in\nvarious \ufb01elds of deep learning, yielding state-of-the-art results. It facilitates neural networks to\nfocus on valuable information of inputs, aiming to avoid interference from redundant messages.\nFirstly, proposed in machine translation, attention mechanism aims to align the words in the source\nlanguage and target language [2]. Then, it has been applied in other application of natural lan-\nguage processing [27, 30, 6], evolving to an indispensable module in neural networks architectures.\nMotivated by its success in NLP, attention mechanism has also been employed in computer vision\n\n2\n\n\ftasks. Conspicuous performance gain has also been observed in images recognition [28], images\ncaption [33], and \ufb01ne-grained classi\ufb01cation [7].\nIn this paper, a simpli\ufb01ed squeezing-expansion attention module has been proposed to facilitate the\ntraining of shallow classi\ufb01ers, improving the accuracy of shallow classi\ufb01ers in various networks.\nModel compression: The phenomenon that over-parameterized models can\u2019t be deployed on edge\ndevices for their excessive requirements of storage and computation has stimulated research of\nmodels compression and acceleration. The typical methods include pruning, quantization, compact\nmodels design and knowledge distillation. Pruning [8, 9] is to cut off the redundant connections or\nchannels in pre-trained neural networks. Quantization [24, 4] targets at replacing the 32 bits \ufb02oat\nnumbers with fewer bits. Knowledge distillation [11, 3, 39, 36, 1] aims to transfer the knowledge\nof over-parameterized models to a small model in order to approach higher accuracy with fewer\nparameters and computation. In addition, some researchers try to design compact models [12, 16]\nwhich has fewer parameters yet still a high accuracy.\nAforementioned work has thoroughly exploited the redundancy in weights and neural networks\narchitectures yet ignored the drawback that samples with diverse dif\ufb01culty are treated equally. The\nproposed SCAN framework is orthogonal and complementary to aforementioned work, targeting at\nexploring more acceleration possibility through unbalanced sample-speci\ufb01c disposal.\n\nFigure 1: The architecture of ResNet18 equipped with SCAN. (i) The whole neural networks can be\ndivided into three sections: backbone, attention modules and shallow classi\ufb01ers. (ii) The backbone\nsection is just identical to the origin model. (iii) Additional attention modules are attached after the\nintermediate features of backbone.(iv) Features re\ufb01ned by attention modules will be feed into the\nshallow classi\ufb01ers, which consist of a bottleneck layer and a fully connected layer.\n\n3 SCAN framework\n\nIn this section, we introduce the proposed SCAN architecture as 3 parts, as shown in Figure 1. Firstly,\nto obtain human-like scalable inference, classi\ufb01ers with varying response time are indispensable.\nAccording to self distillation [38], a bottleneck layer and a fully connected layer are organized as\nshallow classi\ufb01ers. In addition, knowledge distillation is utilized to facilitate the training of shallow\nclassi\ufb01ers, which will be further introduced in Section 3.1.\nSecondly, although shallow classi\ufb01ers permit instant prediction, it also leads to dramatic decline on\naccuracy. To address this problem, a simpli\ufb01ed attention module is proposed to compensate for the\naccuracy of shallow classi\ufb01ers, which will be further introduced in Section 3.2.\nWe further propose a threshold-based strategy to manage all the shallow classi\ufb01ers to corporate\ntogether. Moreover, a genetic algorithm is designed to search for proper thresholds, which will be\nbrought forth in Section 3.3.\n\n3\n\n\fFigure 2: The accuracy of classi\ufb01ers trained with different methods.\n\n3.1 Self distillation\n\nSelf distillation provides an effective method to construct and train shallow classi\ufb01ers which share\nthe same backbone neural network. It \ufb01rstly divides neural networks into several sections depending\non their depth. Then a bottleneck layer and fully connected layer are attached after the intermediate\nfeatures as shallow classi\ufb01ers, which are regarded as the student models in knowledge distillation. In\nthe training period, the knowledge of the deepest classi\ufb01er is distilled into each shallow classi\ufb01er,\nwhose function loss can be written as\n\nC(cid:88)\n\nC(cid:88)\n\n(cid:16)\n\n(cid:17)\n\nloss =\n\nlossi =\n\n(1\u2212 \u03b1)\u00b7 CrossEntropy(qi, y) + \u03b1\u00b7 KL(qi, qC) + \u03bb\u00b7||Fi \u2212 FC||2\n\n2\n\n(1)\n\ni=1\n\ni=1\n\nwhere C denotes the number of classi\ufb01ers. qi and qC represent the outputs of softmax in the ith\nclassi\ufb01er and the deepest classi\ufb01er respectively. y represents corresponding labels. Fi and FC signify\nthe feature maps in the ith classi\ufb01er and the deepest classi\ufb01er respectively. CrossEntropy, KL\ndenote the well-known cross entropy loss and Kullback\u2013Leibler divergence respectively. Experiments\nresults show that signi\ufb01cant performance gain can be observed on not only shallow classi\ufb01er but\nalso the deepest classi\ufb01ers. Motivated by its impressive achievement, self distillation is utilized to\nconstruct and train the shallow classi\ufb01ers in SCAN.\n\n3.2 Attention modules\n\nFigure 2 provides accuracy comparison of four methods training shallow classi\ufb01ers in ResNet50\non CIFAR100. The X axis is the depth of classi\ufb01ers, where x=5 indicates the ensemble of all the\nclassi\ufb01ers [19], and Y axis denotes accuracy. It\u2019s observed that evident accuracy decay can be\nobserved as the depth of classi\ufb01ers decreases. For example, 13% and 8% drop on accuracy exists\non the shallowest and second shallowest classi\ufb01er in self distillation. Moreover, as depicted in\nFigure 2, the 3th classi\ufb01er of self distillation and DSN [21] is lower than individual training, which\nmay be caused by the negative interaction among classi\ufb01ers in one backbone neural network [13].\nFeatures desired by different classi\ufb01ers are mixed up in the sharing backbone neural network. It\u2019s\nunattainable for each classi\ufb01er to detach its own features automatically. To address this problem\n\nAlgorithm 1 Scalable Inference\nInput: Samples X, Thresholds \u03a3 = {\u03c3i}N , Classi\ufb01ers C = {ci}N , Multi-classi\ufb01ers model M\nOutput: Predicted labels Y\n1: Y := None\n2: for i from 1 to N do\n3:\n4:\n5:\n6:\n7: if Y = None then\n8:\n9: return Prediction\n\nlogit = M.getSoftmaxOutputs(X, i)\nif max(logits) > \u03c3i then\nY := argmax(logits)\nBreak\n\nY := M.getEnsemblePrediction()\n\n4\n\n\fand further enhance the performance of shallow classi\ufb01ers, attention modules are utilized to obtain\nclassi\ufb01er-speci\ufb01c features from the sharing backbone neural network. Inspired by RAN [28], we\npropose a simpli\ufb01ed attention modules including one convolution layer for downsampling and one\ndeconvolution layer for upsampling. A sigmoid activation is attached after attention modules to obtain\nattention maps between 0 and 1. Then, the attention maps are involved in a dot product operation\nwith origin features, yielding classi\ufb01er-speci\ufb01c features. Its forward computation can be formulated\nas\n\nAttention M aps(Wconv, Wdeconv, F ) = \u03c3(\u03c6(\u03c8(F, Wconv)), Wdeconv)\n\n(2)\nwhere \u03c8 and \u03c6 denote convolution function and deconvolution function respectively. F represents the\ninput features and \u03c3 signi\ufb01es a sigmoid function. Notes that batch normalization and ReLU activation\nfunction after convolution and deconvolution layers are omitted here.\nExperiments results demonstrate that attention modules in SCAN lead to dramatic accuracy boost in\nshallow classi\ufb01ers. For instance, 5.46%, 4.13% ,and 5.16% accuracy gain can be observed on the\nshallow classi\ufb01ers in ResNet50 on CIFAR100, compared with self distillation [38].\n\n3.3 Scalable inference mechanism\n\nIt is generally acknowledged that the prediction of neural networks with a higher con\ufb01dence (softmax\nvalue) is more likely to be right. In this paper, we exploit this observation to determine whether a\nclassi\ufb01er gives a right or wrong prediction. As described in Algorithm 1, we set different thresholds\nfor shallow classi\ufb01ers. If the maximal output of softmax in shallow classi\ufb01er is larger than the\ncorresponding threshold, its results will be adopted as the \ufb01nal prediction. Otherwise, the neural\nnetworks will employ a deeper classi\ufb01er to predict, until the deepest one or the ensemble prediction.\nBecause most of the computation for shallow classi\ufb01ers is included by that for deep classi\ufb01ers, there\nis no much extra computation introduced.\nHowever, threshold controlled scalable inference causes another problem, that is the choice of\nthresholds for difference classi\ufb01ers. Thresholds matters: (i) A lower threshold for shallow classi\ufb01ers\nresults in that most samples will be predicted by shallow classi\ufb01ers, indicating more rapid response\nyet lower accuracy. (ii) Similarly, a higher threshold leads to a phenomenon that most samples will\nbe determined by deeper classi\ufb01ers, indicating precise prediction yet longer response time. (iii) By\nadjusting the value of thresholds, \ufb02exible accuracy-response time trade-offs can be approached on\nthe \ufb02y. Instead of designing thresholds manually, we propose a genetic algorithm based method to\nsearch the most optimal thresholds as is depicted in Algorithm 1.\nGenes Coding: Genes, a binary sequence is ought to be decoded into its corresponding threshold.\nTo guarantee the accuracy of the model, we empirically restrict the lower bound for thresholds as\n0.70. Its decode function can be formulated as\n\u03c3i = 1 \u2212 0.3\nN\n\n\u00b7 N(cid:88)\n\n(3)\n\nS(n) = 1\n\nn=1\n\nHere S(n) indicates the nth bit in the sequence of genes. \u03c3 denotes the threshold of ith gene. N is\nthe length of binary sequence utilized to express one threshold. The more bits \"1\" there are in this\nsequence, the lower the threshold is.\nFitness: Another crucial issue is to choose the metrics for computing the \ufb01tness for thresholds.\nTargeting at accelerating the models and improving the performance at the same time, these two\nelements are taken into consideration for the \ufb01tness metrics, which can be formulated as\n\nf itness = acceleration ratio + \u03b2 \u00b7 (accuracy \u2212 baseline)\n\n(4)\nwhere \u03b2 is a hyper-parameter to balance the impact of these two elements. Adjustment of \u03b2 leads to\ntrade-offs between accuracy and acceleration.\n\n4 Experiments\n\nWe evaluate SCAN on two benchmark datasets: CIFAR100 [18] and ImageNet (ILSVRC2012) [5]\nand three kinds of neural networks with difference depth and width: VGG [26], ResNet [10] and\nWide ResNet [35]. During training periods, common techniques like data argumentation (random\n\n5\n\n\fAlgorithm 2 Threshold Searching\nInput: Genes G, Multi-classi\ufb01ers model M, Dataset D , Generations g\nOutput: Optimal Thresholds \u03a3 = {\u03c3i}N\n1: RandomlyInitialize(G)\n2: for i from 1 to g do\n3:\n4:\n5:\n6:\n7: \u03a3 := decode(G)\n8: return \u03a3\n\n\ufb01tness = getFitness(G, D, M)\nG := weightedSelect(G, \ufb01tness)\nG := crossover(G)\nG := mutate(G)\n\n// Calculate \ufb01tness of each gene according to Equation 4.\n\n// Drop the genes with low \ufb01tness.\n// Each two genes cross over, generating new genes.\n// Each bit of genes may mutate with a low possibility.\n// Decode the genes into thresholds according to Equation 3.\n\nFigure 3: Accuracy and computation of\neach classi\ufb01er on CIFAR100.\n\nFigure 4: Accuracy and storage of each clas-\nsi\ufb01er on CIFAR100.\n\ncropping and horizon \ufb02ipping), learning rate decay, l2 regularization are equipped. To \ufb01t the size of\ntiny images in CIFAR, we slightly adjust the kernel size and strides of convolution and pooling layers.\nThe recommended value for hyper-parameters \u03bb and \u03b1 in Equation 1, and N in Equation 3 are 0.5,\n5e-7 and 30, respectively. Note that the reported ImageNet (ILSVRC2012) accuracy is evaluated on\nvalidation set. All the experiments are conducted by PyTorch1.0 on GPU devices.\n\n4.1 Results on CIFAR100\n\nExperiments results on CIFAR100 are depicted in Figure 3 and 4. The squares connected in the\nsame line from left to right denote the classi\ufb01er1/4 to classi\ufb01er4/4 and their ensemble. All of them\nshare the same backbone neural network. The triangle with the same color out of lines denotes the\ncorresponding baseline.\nIt is observed that (i) In all the situations, the classi\ufb01er2/4 equipped with SCAN outperforms its\nbaseline. (ii) 2.17X acceleration and 3.20X compression have been achieved on average with no\naccuracy drop. (iii) Compared with the corresponding baseline, 4.05% accuracy increment can be\nobtained with 4.4% relative computation increment on average. (iv) The ensemble of all classi\ufb01ers in\none neural network leads to 1.11% performance gain with almost no incremental computation and\nstorage. (v) Shallow classi\ufb01ers bene\ufb01t more from SCAN than deeper classi\ufb01ers. (vi) Deeper or wider\nneural networks bene\ufb01t more from SCAN than shallower or thinner neural networks.\n\n4.2 Results on ImageNet\n\nAs shown in Table 4.1, the same tendency of accuracy increment on all the classi\ufb01ers can also be\nobserved: (i) On average, 1.26% accuracy gain on ImageNet can be achieved, varying from 1.41% on\nResNet50 as maximum to 1.08% on ResNet101 as minimum. (ii) The depth of classi\ufb01ers impacts\ntheir accuracy more signi\ufb01cantly, which indicates there is less redundancy in ImageNet compared\nwith CIFAR100.\n\n6\n\n\fModels\n\nVGG16(BN)\nVGG19(BN)\n\nResNet18\nResNet50\nResNet101\nResNet152\nWRN20-8\nWRN44-8\n\nTable 1: Experiments results of accuracy (%) on CIFAR100.\nBaseline Classi\ufb01er1/4 Classi\ufb01er2/4 Classi\ufb01er3/4 Classi\ufb01er4/4 Ensemble\n72.46\n72.25\n77.09\n77.68\n77.98\n79.21\n74.61\n76.22\n\n75.18\n74.15\n78.62\n80.39\n80.95\n81.73\n79.25\n79.67\n\n76.80\n75.43\n80.46\n81.78\n82.06\n82.94\n80.04\n80.35\n\n71.29\n71.52\n71.84\n73.69\n72.26\n73.14\n74.52\n76.02\n\n74.92\n74.02\n77.74\n78.34\n79.26\n80.40\n78.17\n78.74\n\n75.29\n74.43\n79.13\n80.45\n81.12\n81.62\n\n/\n/\n\nTable 2: Experiments results of accuracy (%) on ImageNet.\n\nModels\nResNet18\nResNet50\nResNet101\n\nBaseline Classi\ufb01er1/4 Classi\ufb01er2/4 Classi\ufb01er3/4 Classi\ufb01er4/4\n68.02\n74.47\n75.24\n\n69.32\n75.88\n76.32\n\n65.32\n73.57\n74.51\n\n48.25\n53.86\n52.32\n\n58.00\n66.54\n65.33\n\n4.3 Results of scalable inference\n\nExperiments results for scalable inference are depicted in Figure 5. The X axis is the acceleration\nratio compared to its baseline. The Y axis is the top 1 accuracy evaluated on CIFAR100 and ImageNet.\nThe squares connected in the same line denotes the results of different thresholds for the same neural\nnetwork. The triangles on x=1 denote the baselines of different models.\nIt is observed that:(i) Compared with deploying one of the shallow classi\ufb01er individually, scalable\ninference leads to a higher acceleration ratio. (ii) Compared with baselines on CIFAR100, 4.41X\nacceleration can be achieved with no drop on accuracy on average, varying from 2.41X on ResNet18\nas minimum to 6.23X on ResNet152 as maximum. (iii) Compared with baselines on ImageNet, 1.99X\nacceleration can be achieved with no accuracy drop on average, varying from 1.54X on ResNet50 as\nminimum to 2.43X on ResNet101 as maximum. (iv) More accuracy gain can be observed on deeper\nneural networks which is in accordance with the general observation that over-parameterized models\nhave more potential to be compressed and accelerated.\n\n5 Discussion\n\nWhat have attention modules learned? In SCAN, attention modules are introduced to obtain\nclassi\ufb01er-speci\ufb01c features, leading to signi\ufb01cant performance gain on shallow classi\ufb01ers. We further\nvisualize the spatial attention maps as depicted in Figure 7. The heat maps indicate learned attention\nmaps, where the value of each pixel is computed as the mean value of pixels in the same position of\nall channels.\n\nFigure 5: Acceleration ratio and accuracy\nof different thresholds.\n\nFigure 6: Statistics of samples predicted in\neach classi\ufb01er on three kinds of datasets.\n\n7\n\n\fFigure 7: Visualization of attention maps in shallow classi\ufb01ers.\n\nAs is depicted in Figure 7, all the classi\ufb01ers pay their attention on the same spatial position - the\nbodies of a shark and a cat, while ignoring the backgrounds, which indicates that all of the attention\nmodules have learned to \ufb01nd the most informative pixels. The attention maps in classi\ufb01ers 1/4 seems to\nconcentrate on the details of shark\u2019s and cat\u2019s features such as their outlines. In contrast, the attention\nmaps in deeper classi\ufb01ers 3/4 focus more on the texture features, which indicates deep classi\ufb01ers that\nhave a larger receptive \ufb01led are more likely to predict based on global and low frequency information\nwhile shallow classi\ufb01ers incline to be dominated by local and high frequency information.\nHow many samples are predicted by shallow classi\ufb01ers? The determinants of the acceleration\nratio in SCAN is the number of samples predicted by shallow classi\ufb01ers, which varies from thresholds,\ndatasets and neural networks. Figure 6 shows the statistics of samples predicted by each classi\ufb01er of\nResNet18 on CIFAR10, CIFAR100 and ImageNet with the same thresholds. It\u2019s observed that:(i)\nMore than half samples in CIFAR10 and CIFAR100 can be classi\ufb01ed in the shallowest classi\ufb01er,\nwhich consumes the least computation compared with others. (ii) In ImageNet, more samples have to\nbe predicted in the last two classi\ufb01ers, which indicates the classi\ufb01cation of ImageNet data is beyond\nthe capacity of shallow classi\ufb01ers. Based on these observation, two possible usage are proposed as\nfollows:\nFirstly, the number of samples predicted in different classi\ufb01er can be utilized as a guidance of models\ncompression. For example, the classi\ufb01er4/4 in Figure 6 provides extremely little valid prediction in\nCIFAR10 and CIFAR100, indicating there is much more redundancy and compression potential.\nSecondly, it also can be utilized as a metric if datasets dif\ufb01culty. It\u2019s not rigorous to measure the\ndif\ufb01culty of datasets by prediction accuracy because different datasets consist of different numbers of\ncategories. SCAN provides a possible solution - the ratio of samples predicted by shallow classi\ufb01ers.\nFuture works: Although SCAN has achieved signi\ufb01cant acceleration and boost on accuracy, we\nstill believe it has more potential. Firstly, more creditable judgement of whether the prediction of\nshallow classi\ufb01ers should be adopted remains to be explored. This issue can be formulated as a binary\nclassi\ufb01cation problem which may be addressed by machine learning algorithms.\nSecondly, continued optimization on the structure of shallow classi\ufb01ers is necessary. The success of\nattention modules in SCAN proves that tiny adjustment on shallow classi\ufb01ers can lead to dramatic\naccuracy boost, indicating that more compact and ef\ufb01cient shallow classi\ufb01ers can be achieved by a\nwell designed structure.\n\n8\n\n\f6 Conclusion\n\nWe have proposed a novel neural networks training and inference framework named SCAN, whose\nbene\ufb01ts can be seen in three folds: Firstly, self distillation and attention modules are utilized to train\ncompact and ef\ufb01cient shallow classi\ufb01ers to achieve static acceleration and compression. Secondly,\nSCAN exploits the diversity in prediction dif\ufb01culty to accomplish human-like sample-speci\ufb01c\nconditional execution, yielding scalability and a high acceleration ratio. Thirdly, compared with self\ndistillation and its corresponding baseline, more signi\ufb01cant accuracy gain can be achieved on the\nensemble of all classi\ufb01ers with a negligible growth on computation and storage.\n\n7 Acknowledgement\n\nThis paper is supported by Institute for Interdisciplinary Information Core Technology, Beijing\nAcademy of Arti\ufb01cial Intelligence and Zhongguancun Haihua Institute for Frontier Information\nTechnology.\n\nReferences\n[1] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NeurIPS, pages 2654\u20132662, 2014.\n\n[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning\n\nto align and translate. In ICLR, 2015.\n\n[3] Cristian Bucilu\u02c7a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In SIGKDD, pages\n\n535\u2013541, 2006.\n\n[4] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural\n\nnetworks with binary weights during propagations. In NeurIPS, pages 3123\u20133131, 2015.\n\n[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical\n\nimage database. In CVPR, pages 248\u2013255, 2009.\n\n[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec-\n\ntional transformers for language understanding. In NAACL, 2018.\n\n[7] Jianlong Fu, Heliang Zheng, and Tao Mei. Look closer to see better: Recurrent attention convolutional\n\nneural network for \ufb01ne-grained image recognition. In CVPR, volume 2, page 3, 2017.\n\n[8] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with\n\npruning, trained quantization and huffman coding. In ICLR, 2016.\n\n[9] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for ef\ufb01cient\n\nneural network. In NeurIPS, pages 1135\u20131143, 2015.\n\n[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n\nIn CVPR, pages 770\u2013778, 2016.\n\n[11] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS,\n\n2014.\n\n[12] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco\nAndreetto, and Hartwig Adam. Mobilenets: Ef\ufb01cient convolutional neural networks for mobile vision\napplications. In CVPR, 2017.\n\n[13] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger.\n\nMulti-scale dense networks for resource ef\ufb01cient image classi\ufb01cation. In ICLR, 2018.\n\n[14] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected\n\nconvolutional networks. In CVPR, pages 4700\u20134708, 2017.\n\n[15] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic\n\ndepth. In ECCV, pages 646\u2013661, 2016.\n\n[16] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.\nSqueezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. In ICLR, 2016.\n\n9\n\n\f[17] Eunwoo Kim, Chanho Ahn, and Songhwai Oh. Nestednet: Learning nested sparse structures in deep neural\n\nnetworks. In CVPR, pages 8669\u20138678, 2018.\n\n[18] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical\n\nreport, Citeseer, 2009.\n\n[19] Ashish Kumar, Saurabh Goyal, and Manik Varma. Resource-ef\ufb01cient machine learning in 2 kb ram for the\n\ninternet of things. In ICML, pages 1935\u20131944. JMLR. org, 2017.\n\n[20] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks\n\nwithout residuals. arXiv preprint arXiv:1605.07648, 2016.\n\n[21] Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets.\n\nIn Arti\ufb01cial Intelligence and Statistics, pages 562\u2013570, 2015.\n\n[22] Tsung-Yi Lin, Piotr Doll\u00e1r, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature\n\npyramid networks for object detection. In CVPR, volume 1, page 4, 2017.\n\n[23] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and\n\nAlexander C Berg. Ssd: Single shot multibox detector. In ECCV, pages 21\u201337, 2016.\n\n[24] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classi\ufb01ca-\n\ntion using binary convolutional neural networks. In ECCV, pages 525\u2013542, 2016.\n\n[25] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua\n\nBengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.\n\n[26] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-\n\ntion. In ICLR, 2015.\n\n[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz\n\nKaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998\u20136008, 2017.\n\n[28] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and\n\nXiaoou Tang. Residual attention network for image classi\ufb01cation. In CVPR, pages 3156\u20133164, 2017.\n\n[29] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic\n\nrouting in convolutional networks. In ECCV, pages 409\u2013424, 2018.\n\n[30] Xuejian Wang, Lantao Yu, Kan Ren, Guanyu Tao, Weinan Zhang, Yong Yu, and Jun Wang. Dynamic\nattention deep model for article recommendation by learning human editors\u2019 demonstration. In SIGKDD,\npages 2051\u20132059, 2017.\n\n[31] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and\nRogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, pages 8817\u20138826,\n2018.\n\n[32] Saining Xie, Ross Girshick, Piotr Doll\u00e1r, Zhuowen Tu, and Kaiming He. Aggregated residual transforma-\n\ntions for deep neural networks. In CVPR, pages 5987\u20135995, 2017.\n\n[33] BaJ XuK, CourvilleA KirosR, et al. Show, attend and tell: Neural image caption generation with visual\n\nattention. In ICML, pages 20148\u20132057, 2015.\n\n[34] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In\n\nICLR, 2019.\n\n[35] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.\n\n[36] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance\n\nof convolutional neural networks via attention transfer. In ICLR, 2017.\n\n[37] Amir R Zamir, Te-Lin Wu, Lin Sun, William B Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese.\n\nFeedback networks. In CVPR, pages 1308\u20131317, 2017.\n\n[38] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your\nown teacher: Improve the performance of convolutional neural networks via self distillation. In arXiv\npreprint:1905.08094, 2019.\n\n[39] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, pages\n\n4320\u20134328, 2018.\n\n10\n\n\f", "award": [], "sourceid": 2222, "authors": [{"given_name": "Linfeng", "family_name": "Zhang", "institution": "Tsinghua University"}, {"given_name": "Zhanhong", "family_name": "Tan", "institution": "Tsinghua University"}, {"given_name": "Jiebo", "family_name": "Song", "institution": "Institute for Interdisciplinary Information Core Technology"}, {"given_name": "Jingwei", "family_name": "Chen", "institution": "Tsinghua University"}, {"given_name": "Chenglong", "family_name": "Bao", "institution": "Tsinghua university"}, {"given_name": "Kaisheng", "family_name": "Ma", "institution": "Tsinghua University"}]}