{"title": "Focused Quantization for Sparse CNNs", "book": "Advances in Neural Information Processing Systems", "page_first": 5584, "page_last": 5593, "abstract": "Deep convolutional neural networks (CNNs) are powerful tools for a wide range of vision tasks, but the enormous amount of memory and compute resources required by CNNs poses a challenge in deploying them on constrained devices. Existing compression techniques, while excelling at reducing model sizes, struggle to be computationally friendly. In this paper, we attend to the statistical properties of sparse CNNs and present focused quantization, a novel quantization strategy based on power-of-two values, which exploits the weight distributions after fine-grained pruning. The proposed method dynamically discovers the most effective numerical representation for weights in layers with varying sparsities, significantly reducing model sizes. Multiplications in quantized CNNs are replaced with much cheaper bit-shift operations for efficient inference. Coupled with lossless encoding, we build a compression pipeline that provides CNNs with high compression ratios (CR), low computation cost and minimal loss in accuracies. In ResNet-50, we achieved a 18.08x CR with only 0.24% loss in top-5 accuracy, outperforming existing compression methods. We fully compress a ResNet-18 and found that it is not only higher in CR and top-5 accuracy, but also more hardware efficient as it requires fewer logic gates to implement when compared to other state-of-the-art quantization methods assuming the same throughput.", "full_text": "Focused Quantization for Sparse CNNs\n\nYiren Zhao\u22171\n\nXitong Gao\u22172\n\nDaniel Bates1\n\nRobert Mullins1\n\nCheng-Zhong Xu3\n\n1 University of Cambridge\n\n2 Shenzhen Institutes of Advanced Technology\n\n3 University of Macau\n\nAbstract\n\nDeep convolutional neural networks (CNNs) are powerful tools for a wide range of\nvision tasks, but the enormous amount of memory and compute resources required\nby CNNs pose a challenge in deploying them on constrained devices. Existing\ncompression techniques, while excelling at reducing model sizes, struggle to be\ncomputationally friendly. In this paper, we attend to the statistical properties of\nsparse CNNs and present focused quantization, a novel quantization strategy based\non power-of-two values, which exploits the weight distributions after \ufb01ne-grained\npruning. The proposed method dynamically discovers the most effective numerical\nrepresentation for weights in layers with varying sparsities, signi\ufb01cantly reducing\nmodel sizes. Multiplications in quantized CNNs are replaced with much cheaper\nbit-shift operations for ef\ufb01cient inference. Coupled with lossless encoding, we\nbuilt a compression pipeline that provides CNNs with high compression ratios\n(CR), low computation cost and minimal loss in accuracy. In ResNet-50, we\nachieved a 18.08\u00d7 CR with only 0.24% loss in top-5 accuracy, outperforming\nexisting compression methods. We fully compressed a ResNet-18 and found that it\nis not only higher in CR and top-5 accuracy, but also more hardware ef\ufb01cient as it\nrequires fewer logic gates to implement when compared to other state-of-the-art\nquantization methods assuming the same throughput.\n\n1\n\nIntroduction\n\nDespite deep convolutional neural networks (CNNs) demonstrating state-of-the-art performance in\nmany computer vision tasks, their parameter-rich and compute-intensive nature substantially hinders\nthe ef\ufb01cient use of them in bandwidth- and power-constrained devices. To this end, recent years have\nseen a surge of interest in minimizing the memory and compute costs of CNN inference.\nPruning algorithms compress CNNs by setting weights to zero, thus removing connections or neurons\nfrom the models.\nIn particular, \ufb01ne-grained pruning [16, 6] provides the best compression by\nremoving connections at the \ufb01nest granularity, i.e. individual weights. Quantization methods reduce\nthe number of bits required to represent each value, and thus further provide memory, bandwidth\nand compute savings. Shift quantization of weights, which quantizes weight values in a model\nto powers-of-two or zero, i.e. {0,\u00b11,\u00b12,\u00b14, . . .}, is of particular of interest, as multiplications\nin convolutions become much-simpler bit-shift operations. The computational cost in hardware\ncan thus be signi\ufb01cantly reduced without a detrimental impact on the model\u2019s task accuracy [26].\nFine-grained pruning, however, is often in con\ufb02ict with quantization, as pruning introduces various\ndegrees of sparsities to different layers [25, 19]. Linear quantization methods (integers) have uniform\nquantization levels and non-linear quantizations (logarithmic, \ufb02oating-point and shift) have \ufb01ne levels\n\u2217Xitong Gao and Yiren Zhao contributed equally to this work. Correspondence to Xitong Gao\n\n(xt.gao@siat.ac.cn) and Yiren Zhao (yiren.zhao@cl.cam.ac.uk).\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\faround zero but levels grow further apart as values get larger in magnitude. Both linear and nonlinear\nquantizations thus provide precision where it is not actually required in the case of a pruned CNN.\nIt is observed that empirically, very few non-zero weights concentrate around zero in some layers\nthat are sparsi\ufb01ed with \ufb01ne-grained pruning (see Figure 1c for an example). Shift quantization is\nhighly desirable as it can be implemented ef\ufb01ciently, but it becomes a poor choice for certain layers\nin sparse models, as most near-zero quantization levels are under-utilized (Figure 1d).\n\n(a) Dense layers\n\n(b) After shift quantization\n\n(c) Sparse layers\n\n(d) After shift quantization\n\nFigure 1: The weight distributions of the \ufb01rst 8 layers of ResNet-18 on ImageNet. (a) shows the weight\ndistributions of the layers, (c) similarly shows the distributions (excluding zeros) for a sparsi\ufb01ed variant. (b)\nand (d) respectively quantize the weight distributions on the left with 5-bit shift quantization. Note that in some\nsparse layers, greedy pruning encourages weights to avoid near zero values. Shift quantization on these layers\nthus results in poor utilization of the quantization levels.\n\nThis dichotomy prompts the question, how can we quantize sparse weights ef\ufb01ciently and effectively?\nHere, ef\ufb01ciency represents not only the reduced model size but also the minimized compute cost.\nEffectiveness means that the quantization levels are well-utilized. From an information theory\nperspective, it is desirable to design a quantization function Q such that the quantized values in\n\u02c6\u03b8 = Q(\u03b8) closely match the prior weight distribution. We address both issues by proposing a new\napproach to quantize parameters in CNNs which we call focused quantization (FQ) that mixes shift\nand recentralized quantization methods. Recentralized quantization uses a mixture of Gaussian\ndistributions to \ufb01nd the most concentrated probability masses in the weight distribution of sparse\nlayers (\ufb01rst block in Figure 2), and independently quantizes the probability masses (rightmost of\nFigure 2) to powers-of-2 values. Additionally, not all layers consist of two probability masses,\nand recentralized quantization may not be necessary (as shown in Figure 1c). In such cases, we\nuse the Wasserstein distance between the two Gaussian components to decide when to apply shift\nquantization.\nFor evaluation, we present a complete compression pipeline comprising \ufb01ne-grain pruning, FQ and\nHuffman encoding and estimate the resource utilization in custom hardware required for inference.\nWe show that the compressed models with FQ not only provide higher task accuracies, but also\nrequire less storage and lower logic usage when compared to other methods. This suggests the\nFQ-based compression is a more practical alternative design for future custom hardware accelerators\ndesigned for neural network inference [24].\nIn this paper, we make the following contributions:\n\ncomputation and model size with minimal loss of accuracy.\n\nprovide the most effective quantization on sparse CNNs.\n\n\u2022 The proposed method, focused quantization for sparse CNNs, signi\ufb01cantly reduces both\n\u2022 FQ is hybrid, it systematically mixes a recentralized quantization with shift quantization to\n\u2022 We built a complete compression pipeline based on FQ. We observed that FQ achieves the\n\u2022 We found that a hardware design based on FQ demonstrates the most ef\ufb01cient hardware\n\nhighest compression rates on a range of modern CNNs with the least accuracy losses.\n\nutilization compared to previous state-of-the-art quantization methods [15, 23].\n\nThe rest of the paper is structured as follows. Section 2 discusses related work in the \ufb01eld of\nmodel compression. Section 3 introduces focused quantization. Section 4 presents and evaluates the\nproposed compression pipeline and Section 5 concludes the paper.\n\n2 Related Work\n\nRecently, a wide range of techniques have been proposed and proven effective for reducing the\nmemory and computation requirements of CNNs. These proposed optimizations can provide direct\n\n2\n\n\freductions in memory footprints, bandwidth requirements, total number of arithmetic operations,\narithmetic complexities or a combination of these properties.\nPruning-based optimization methods directly reduce the number of parameters in a network. Fine-\ngrained pruning method [6] signi\ufb01cantly reduces the size of a model but introduces element-wise\nsparsity. Coarse-grained pruning [17, 4] shrinks model sizes and reduce computation at a higher gran-\nularity that is easier to accelerate on commodity hardware. Quantization methods allow parameters\nto be represented in more ef\ufb01cient data formats. Quantizing weights to powers-of-2 recently gained\nattention because it not only reduces the model size but also simpli\ufb01es computation [14, 26, 18, 24].\nPrevious research also focused on quantizing CNNs to extremely low bit-widths such as ternary [27]\nor binary [12] values. They however introduce large numerical errors and thus cause signi\ufb01cant\ndegradations in model accuracies. To minimize loss in accuracies, the proposed methods of [23] and\n[15] quantize weights to N binary values, compute N binary convolutions and scale the convolution\noutputs individually before summation. Lossy and lossless encodings are other popular methods to\nreduce the size of a DNN, typically used in conjunction with pruning and quantization [3, 7].\nSince many compression techniques are available and building a compression pipeline provides a\nmultiplying effect in compression ratios, researchers start to chain multiple compression techniques.\nHan et al. [7] proposed Deep Compression that combines pruning, quantization and Huffman\nencoding. Dubey et al. [3] built a compression pipeline using their coreset-based \ufb01lter representations.\nTung et al. [22] and Polino et al. [20] integrated multiple compression techniques, where [22]\ncombined pruning with quantization and [20] employed knowledge distillation on top of quantization.\nAlthough there are many attempts in building an ef\ufb01cient compression pipeline, the statistical\nrelationship between pruning and quantization lacked exploration. In this paper, we look at exactly\nthis problem and propose a new method that exploits the statistical properties of weights in pruned\nmodels to quantize them ef\ufb01ciently and effectively.\n\nFigure 2: The step-by-step process of recentralized quantization of unpruned weights on block3f/conv1\nin sparse ResNet-50. Each step shows how it changes a \ufb01lter and the distribution of weights. Higher peaks in\nthe histograms denote values found with higher frequency. Values in the \ufb01lter share a common denominator\n128, indicated by \u201c/128\u201d. The \ufb01rst estimates the high-probability regions with a Gaussian mixture, and assign\nweights to a Gaussian component. The second normalizes each weight. The third quantizes the normalized\nvalues with shift quantization and produces a representation of quantized weights used for inference. The \ufb01nal\nblock visualizes the actual numerical values after quantization.\n\n3 Method\n\n3.1 Preliminaries: Shift quantization\n\nShift quantization is a quantization scheme which constrains weight values to powers-of-two or zero\nvalues. A representable value in a (k + 2)-bit shift quantization is given by:\n\n(1)\nwhere s = {\u22121, 0, 1} denotes either zero or the sign of the value, e is an integer bounded by [0, 2k\u22121],\nand b is the bias, a layer-wise constant which scales the magnitudes of quantized values. We use\n\nv = s \u00b7 2e\u2212b,\n\n3\n\nQuantized(5-bit values)Before(32-bit \ufb02oat)0-3.811.505.630-5.694.54-3.132.44/ 128Normalized(32-bit \ufb02oat)00.19-2.501.630-1.690.540.87-1.56/ 128\u00b5=0.03125AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=\u00b5+=0.03125AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==00.25-220-20.51-2/ 128\u00b5=0.03125AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=AAAB+XicbZDLSsNAFIZP6q3WW9Slm8EidNOSVEU3QsGNywr2Am0Ik+mkHTqZhJlJoYS+iRsXirj1Tdz5Nk7bLLT1h4GP/5zDOfMHCWdKO863VdjY3NreKe6W9vYPDo/s45O2ilNJaIvEPJbdACvKmaAtzTSn3URSHAWcdoLx/bzemVCpWCye9DShXoSHgoWMYG0s37b7UepX0R2qOjXn0q1f+3bZ0EJoHdwcypCr6dtf/UFM0ogKTThWquc6ifYyLDUjnM5K/VTRBJMxHtKeQYEjqrxscfkMXRhngMJYmic0Wri/JzIcKTWNAtMZYT1Sq7W5+V+tl+rw1suYSFJNBVkuClOOdIzmMaABk5RoPjWAiWTmVkRGWGKiTVglE4K7+uV1aNdrrlNzH6/KjUoeRxHO4Bwq4MINNOABmtACAhN4hld4szLrxXq3PpatBSufOYU/sj5/AK72kP4=\u00b5+=0.03125AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==AAAB+HicbZDLSgMxFIYz9VbrpaMu3QSLUBDKTFV0IxTcuKxgL9AOQybNtKFJZshFqEOfxI0LRdz6KO58G9N2Ftr6Q+DjP+dwTv4oZVRpz/t2CmvrG5tbxe3Szu7eftk9OGyrxEhMWjhhiexGSBFGBWlpqhnpppIgHjHSica3s3rnkUhFE/GgJykJOBoKGlOMtLVCt9znJjyDN9Creed+/TJ0K5bmgqvg51ABuZqh+9UfJNhwIjRmSKme76U6yJDUFDMyLfWNIinCYzQkPYsCcaKCbH74FJ5aZwDjRNonNJy7vycyxJWa8Mh2cqRHark2M/+r9YyOr4OMitRoIvBiUWwY1AmcpQAHVBKs2cQCwpLaWyEeIYmwtlmVbAj+8pdXoV2v+V7Nv7+oNKp5HEVwDE5AFfjgCjTAHWiCFsDAgGfwCt6cJ+fFeXc+Fq0FJ585An/kfP4APwWQxQ==Represented values0-3.75260-64.5-32/ 128\f\u02c6\u03b8 = Qshift\nn,b [\u03b8] to denote a n-bit shift quantization with a bias b of a weight value \u03b8 to the nearest\nrepresentable value \u02c6\u03b8. As we have discussed earlier and illustrated in Figure 1, shift quantization on\nsparse layers makes poor use of the range of representable values, i.e. the resulting distribution after\nquantization qshift\nn,b (\u03b8) is a poor approximation of the original layer weight distribution p(\u03b8|D), where\nD is the training dataset.\n3.2 Designing the Recentralized Quantization Function\n\nIntuitively, it is desirable to concentrate quantization effort on the high probability regions in the\nweight distribution in sparse layers. By doing so, we can closely match the distribution of quantized\nweights with the original, and thus at the same time incur smaller round-off errors. Recentralized\nquantization Q[\u03b8] is designed speci\ufb01cally for this purpose, and applied in a layer-wise fashion.\nAssuming that \u03b8 \u2208 \u03b8 is a weight value of a convolutional layer, we can de\ufb01ne Q[\u03b8] as follows:\n\n\u03b4c,m\u03b8 Qrec\n\nc\n\n[\u03b8], where Qrec\n\nc\n\n[\u03b8] = Qshift\n\nn,b (cid:20) \u03b8 \u2212 \u00b5c\n\n\u03c3c (cid:21) \u03c3c + \u00b5c.\n\n(2)\n\nQ[\u03b8] = z\u03b8\u03b1(cid:88)c\u2208C\n\nHere z\u03b8 is a predetermined constant {0, 1} binary value to indicate if \u03b8 is pruned, and it is used to set\npruned weights to 0. The set of components c \u2208 C determines the locations to focus quantization\neffort, each speci\ufb01ed by the component\u2019s mean \u00b5c and standard deviation \u03c3c. The Kronecker delta\n\u03b4c,m\u03b8 evaluates to either 1 when c = m\u03b8, or 0 otherwise. In other words, the constant m\u03b8 \u2208 C\nchooses which component in C is used to quantize \u03b8. Finally, Qrec\n[\u03b8] locally quantizes the component\nc with shift quantization. Following [27] and [14], we additionally introduce a layer-wise learnable\nscaling factor \u03b1 initialized to 1, which empirically improves the task accuracy.\nBy adjusting the \u00b5c and \u03c3c of each component c, and \ufb01nding suitable assignments of weights to\nthe components, the quantized weight distribution q\u03c6(\u03b8) can thus match the original closely, where\nwe use \u03c6 as a shorthand to denote the relevant hyperparameters, e.g. \u00b5c, \u03c3c. The following section\nexplains how we can optimize them ef\ufb01ciently.\n\nc\n\n3.3 Optimizing Recentralized Quantization Q[\u03b8]\n\nHyperparameters \u00b5c and \u03c3c in recentralized quantization can be optimized by applying the following\ntwo-step process in a layer-wise manner, which \ufb01rst identi\ufb01es regions with high probabilities (\ufb01rst\nblock in Figure 2), then locally quantize them with shift quantization (second and third blocks in\nFigure 2). First, we notice that in general, the weight distribution resembles a mixture of Gaussian\ndistributions. It is thus more ef\ufb01cient to \ufb01nd a Gaussian mixture model qmix\n\u03c6 (\u03b8) that approximates the\noriginal distribution p(\u03b8|D) to closely optimize the above objective:\n\nqmix\n\n\u03c6 (\u03b8) =(cid:88)c\u2208C\n\n\u03bbcf (\u03b8|\u00b5c, \u03c3c),\n\n(3)\n\nwhere f (\u03b8|\u00b5c, \u03c3c) is the probability density function of the Gaussian distribution N (\u00b5c, \u03c3c), the non-\nnegative \u03bbc de\ufb01nes the mixing weight of the cth component and \u03a3c\u2208C \u03bbc = 1. Here, we \ufb01nd the set\nof hyperparameters \u00b5c, \u03c3c and \u03bbc contained in \u03c6 that maximizes qmix\n\u03c6 (\u03b8) given that \u03b8 \u223c p(\u03b8|D). This\nis known as the maximum likelihood estimate (MLE), and the MLE can be ef\ufb01ciently computed by the\nexpectation-maximization (EM) algorithm [1]. In practice, we found it suf\ufb01cient to use two Gaussian\ncomponents, C = {\u2212, +}, for identifying high-probability regions in the weight distribution. For\nfaster EM convergence, we initialize \u00b5\u2212, \u03c3\u2212 and \u00b5+, \u03c3+ respectively with the means and standard\ndeviations of negative and positive values in the layer weights respectively, and \u03bb\u2212, \u03bb+ with 1\n2.\nWe then generate m\u03b8 from the mixture model, which individually selects the component to use for\neach weight. For this, m\u03b8 is evaluated for each \u03b8 by sampling a categorical distribution where the\nprobability of assigning a component c to m\u03b8, i.e. p(m\u03b8 = c), is \u03bbcf (\u03b8|\u00b5c, \u03c3c)/ qmix\nn,b [\u00b7] allows at most\nFinally, we set the constant b to a powers-of-two value, chosen to ensure that qshift\n2n+1 values to over\ufb02ow and clips them to the maximum representable magnitude. In\na proportion of\npractice, this heuristic choice makes better use of the quantization levels provided by shift quantization\nthan disallowing over\ufb02ows. After determining all of the relevant hyperparameters with the method\ndescribed above, \u02c6\u03b8 = Q[\u03b8] can be evaluated to quantize the layer weights.\n\n\u03c6 (\u03b8).\n\n1\n\n4\n\n\f3.4 Choosing the Appropriate Quantization\n\nAs we have discussed earlier, the weight distribution of sparse layers may not always have multiple\nhigh-probability regions. For example, \ufb01tting a mixture model of two Gaussian components on the\nlayer in Figure 3a gives highly overlapped components. It is therefore of little consequence which\ncomponent we use to quantize a particular weight value. Under this scenario, we can simply use\nn-bit shift quantization Qshift\nn,b [\u00b7] instead of a n-bit Q[\u00b7] which internally uses a (n \u2212 1)-bit signed shift\nquantization. By moving the 1 bit used to represent the now absent m to shift quantization, we further\nincrease its precision.\n\n(a) Weight distribution.\n\n(b) Overlapping components.\n\nFigure 3: The weight distribution of the layer block22/conv1 in a sparse ResNet-18 trained on ImageNet,\nas shown by the histograms. It shows that when the two Gaussian components have a large overlap, quantizing\nwith either one of them results in almost the same quantization levels.\n\nTo decide whether to use shift or recentralized quantization, it is necessary to introduce a metric to\ncompare the similarity between the pair of components. While the KL-divergence provides a measure\nfor similarity, it is however non-symmetric, making it unsuitable for this purpose. To address this,\nwe propose to \ufb01rst normalize the distribution of the mixture, then to use the 2-Wasserstein metric\nbetween the two Gaussian components after normalization as a decision criterion, which we call the\nWasserstein separation:\n\n1\n\nW(c1, c2) =\n\n\u03c32(cid:16)(\u00b5c1 \u2212 \u00b5c2 )2 + (\u03c3c1 \u2212 \u03c3c2 )2(cid:17) ,\n\n(4)\nwhere \u00b5c and \u03c3c are respectively the mean and standard deviation of the component c \u2208 {c1, c2},\nand \u03c32 denotes the variance of the entire weight distribution. FQ can then adaptively pick to use\nrecentralized quantization for all sparse layers except when W(c1, c2) < wsep, and shift quantization\nis used instead. In our experiments, we found wsep = 2.0 usually provides a good decision criterion.\nIn Section 4.3, we additionally study the impact of quantizing a model with different wsep values.\n\n3.5 Model Optimization\n\nTo optimize the quantized sparse model, we integrate the quantization process described above into\nthe gradient-based training of model parameters. Initially, we compute the hyperparameters \u00b5c, \u03c3c, \u03bbc\nfor each layer, and generate the component selection mask m\u03b8 for each weight \u03b8 with the method in\nSection 3.3. The resulting model is then \ufb01ne-tuned where the forward pass uses quantized weights\n\u02c6\u03b8 = Q[\u03b8], and the backward pass updates the \ufb02oating-point weight parameters \u03b8 by treating the\nquantization as an identity function. During the \ufb01ne-tuning process, the hyperparameters used by\nQ[\u03b8] are updated using the current weight distribution at every k epochs. We also found that in our\nexperiments, exponentially increasing the interval k between consecutive hyperparameter updates\nhelps to reduce the variance introduced by sampling and improves training quality.\n\n3.6 The MDL Perspective\n\nTheoretically, the model optimization can be formulated as a minimum description length (MDL)\noptimization [10, 5]. Given that we approximate the posterior p(\u03b8|D) with a distribution of quantized\nweights q\u03c6(\u03b8), where \u03c6 contains the hyperparameters used by the quantization function Q[\u03b8], the\nMDL problem minimizes the variational free energy [5], L(\u03b8, \u03b1, \u03c6) = LE + LC, where:\n\nLE = E \u02c6\u03b8\u223cq\u03c6(\u03b8)(cid:104)\u2212 log p(y|x, \u03b1, \u02c6\u03b8)(cid:105) , LC = KL(cid:0)q\u03c6(\u03b8)(cid:107)p(\u03b8|D)(cid:1) .\n\n(5)\n\n5\n\n0.10.00.10.10.00.1\fThe error cost LE re\ufb02ects the cross-entropy loss of the quantized model, with quantized weights \u02c6\u03b8 and\nlayer-wise scalings \u03b1, trained on the dataset D = (x, y), which is optimized by stochastic gradient\ndescent. The complexity cost LC is the Kullback-Leibler (KL) divergence from the quantized weight\ndistribution to the original. Intuitively, minimizing LC reduces the discrepancies between the weight\ndistributions before and after quantization. As this is intractable, we replace q\u03c6(\u03b8) with a close\n\u03c6 (\u03b8). It turns out that the process of \ufb01nding the MLE discussed\nsurrogate, a Gaussian mixture qmix\n\u03c6 (\u03b8)(cid:107)p(\u03b8|D)(cid:1), a close proxy for LC. Section 3.5\nin Section 3.3 is equivalent to minimizing KL(cid:0)qmix\nthen interleaves the optimization of LE and LC to minimize the MDL objective L(\u03b8, \u03b1, \u03c6).\n4 Evaluation\n\nWe applied focused compression (FC), a compression \ufb02ow which consists of pruning, FQ and\nHuffman encoding, on a wide range of popular vision models including MobileNets [11, 21] and\nResNets [8, 9] on the ImageNet dataset [2]. For all of these models, FC produced models with high\ncompression ratios (CRs) and permitted a multiplication-free hardware implementation of convolution\nwhile having minimal impact on the task accuracy. In our experiments, models are initially sparsi\ufb01ed\nusing Dynamic Network Surgery [6]. FQ is subsequently applied to restrict weights to low-precision\nvalues. During \ufb01ne-tuning, we additionally employed Incremental Network Quantization (INQ) [26]\nand gradually increased the proportion of weights being quantized to 25%, 50%, 75%, 87.5% and\n100%. At each step, the models were \ufb01ne-tuned for 3 epochs at a learning rate of 0.001, except for the\n\ufb01nal step at 100% we ran for 10 epochs, and decay the learning rate every 3 epochs. Finally, Huffman\nencoding was applied to model weights which further reduced model sizes. To simplify inference\ncomputation in custom hardware (Section 4.2), in our experiments \u00b5\u2212 and \u00b5+ are quantized to the\nnearest powers-of-two values, and \u03c3\u2212 and \u03c3+ are constrained to be equal.\n\n4.1 Model Size Reduction\n\nTable 1 compares the accuracies and compression rates before and after applying the compression\npipeline under different quantization bit-widths. It demonstrates the effectiveness of FC on the\nmodels. We found that sparsi\ufb01ed ResNets with 7-bit weights are at least 16\u00d7 smaller than the original\ndense model with marginal degradations (\u22640.24%) in top-5 accuracies. MobileNets, which are much\nless redundant and more compute-ef\ufb01cient models to begin with, achieved a smaller CR at around 8\u00d7\nand slightly larger accuracy degradations (\u22640.89%). Yet when compared to the ResNet-18 models, it\nis not only more accurate, but also has a signi\ufb01cantly smaller memory footprint at 1.71 MB.\nIn Table 2 we compare FC with many state-of-the-art model compression schemes. It shows that FC\nsimultaneously achieves the best accuracies and the highest CR on both ResNets. Trained Ternary\nQuantization (TTQ) [27] quantizes weights to ternary values, while INQ [26] and extremely low bit\nneural network (denoted as ADMM) [14] quantize weights to ternary or powers-of-two values using\nshift quantization. Distillation and Quantization (D&Q) [20] quantize parameters to integers via\ndistillation. Note that D&Q\u2019s results used a larger model as baseline, hence the compressed model has\nhigh accuracies and low CR. We also compared against Coreset-Based Compression [3] comprising\npruning, \ufb01lter approximation, quantization and Huffman encoding. For ResNet-50, we additionally\ncompare against ThiNet [17], a \ufb01lter pruning method, and Clip-Q [22], which interleaves training\nsteps with pruning, weight sharing and quantization. FC again achieves the highest CR (18.08\u00d7) and\naccuracy (74.86%).\n\n4.2 Computation Reduction\n\nQuantizing weights using FC can signi\ufb01cantly reduce computation complexities in models. By further\nquantizing activations and BN parameters to integers, the expensive \ufb02oating-point multiplications and\nadditions in convolutions can be replaced with simple bit-shift operations and integer additions. This\ncan be realized with much faster software or hardware implementations, which directly translates to\nenergy saving and much lower latencies in low-power devices. In Table 3, we evaluate the impact on\naccuracies by progressively applying FQ on weights, and integer quantizations on activations and\nbatch normalization (BN) parameters. It is notable that the \ufb01nal fully quantized model achieve similar\naccuracies to LQ-Net.\n\n6\n\n\fTable 1: The accuracies (%), sparsities (%) and CRs of focused compression on ImageNet models. The baseline\nmodels are dense models before compression and use 32-bit \ufb02oating-point weights, and 5 bits and 7 bits denote\nthe number of bits used by individual weights of the quantized models before Huffman encoding.\n\nModel\nResNet-18\nPruned\n5 bits\n7 bits\nResNet-50\nPruned\n5 bits\n7 bits\nMobileNet-V1\nPruned\n7 bits\nMobileNet-V2\nPruned\n7 bits\n\n0.30\n-0.58\n-0.37\n\n-0.48\n-0.72\n-0.59\n\nTop-1\n\u2206\n68.94 \u2014\n69.24\n68.36\n68.57\n75.58 \u2014\n75.10\n74.86\n74.99\n70.77 \u2014\n70.03\n69.13\n71.65 \u2014\n71.24\n70.05\n\n-0.74\n-1.64\n\n-0.41\n-1.60\n\n0.38\n-0.22\n-0.14\n\n-0.25\n-0.24\n-0.24\n\nTop-5\n\u2206\n88.67 \u2014\n89.05\n88.45\n88.53\n92.83 \u2014\n92.58\n92.59\n92.59\n89.48 \u2014\n89.13\n88.61\n90.44 \u2014\n90.31\n89.55\n\n-0.35\n-0.87\n\n-0.13\n-0.89\n\nSparsity\n0.00\n74.86\n74.86\n74.86\n0.00\n82.70\n82.70\n82.70\n0.00\n33.80\n33.80\n0.00\n31.74\n31.74\n\nSize (MB) CR (\u00d7)\n\u2014\n5.69\n16.33\n15.92\n\u2014\n7.98\n18.08\n17.98\n\u2014\n2.44\n7.90\n\u2014\n2.46\n8.14\n\n46.76\n8.31\n2.86\n2.94\n93.82\n11.76\n5.19\n5.22\n16.84\n6.89\n2.13\n13.88\n5.64\n1.71\n\nTable 2: Comparisons of top-1 and top-5 accuracies (%) and CRs with various compression methods. Numbers\nwith (cid:63) indicate results not originally reported and calculated by us. Note that D&Q used a much larger ResNet-\n18, the 5 bases used by ABC-Net denote 5 separate binary convolutions. LQ-Net used a \u201cpre-activation\u201d\nResNet-18 [9] with a 1.4% higher accuracy baseline than ours.\n\nResNet-18\n\nTTQ [27]\nINQ (2 bits) [26]\nINQ (3 bits) [26]\nADMM (2 bits) [14]\nADMM (3 bits) [14]\nABC-Net (5 bases, or 5 bits) [15]\nLQ-Net (preact, 2 bits) [23]\nD&Q (large) [20]\nCoreset [3]\nFocused compression (5 bits, sparse)\n\nResNet-50\n\nINQ (5 bits) [26]\nADMM (3 bits) [14]\nThiNet [17]\nClip-Q [22]\nCoreset [3]\nFocused compression (5 bits, sparse)\n\nTop-1 Top-5\n87.10\n66.00\n87.20\n66.60\n88.36\n68.08\n87.5\n67.0\n68.0\n88.3\n87.90\n67.30\n88.00\n68.00\n73.10\n91.17\n68.00 \u2014\n68.36\n88.45\nTop-1 Top-5\n92.45\n74.81\n91.6\n74.0\n72.04\n90.67\n73.70 \u2014\n74.00 \u2014\n74.86\n\n92.59\n\n2.92(cid:63)\n2.92(cid:63)\n4.38(cid:63)\n2.92(cid:63)\n4.38(cid:63)\n7.30(cid:63)\n2.92(cid:63)\n21.98\n3.11(cid:63)\n2.86\n\nSize (MB) CR (\u00d7)\n16.00(cid:63)\n16.00(cid:63)\n10.67(cid:63)\n16.00(cid:63)\n10.67(cid:63)\n6.4 (cid:63)\n16.00(cid:63)\n2.13(cid:63)\n15.00\n16.33\nSize (MB) CR (\u00d7)\n6.40(cid:63)\n10.67(cid:63)\n5.53(cid:63)\n14.00(cid:63)\n15.80\n18.08\n\n14.64(cid:63)\n8.78(cid:63)\n16.94\n6.70\n5.93(cid:63)\n5.19\n\nFigure 4 shows an accelerator design of the dot-products used in the convolutional layers with\nrecentralized quantization for inference. Using this, in Table 4 we provide the logic usage required by\nthe implementation to compute a convolution layer with 3 \u00d7 3 \ufb01lters with a padding size of 1, which\ntakes as input a 8 \u00d7 8 \u00d7 100 activation and produce a 8 \u00d7 8 \u00d7 100 tensor output. Additionally, we\ncompare FQ to shift quantization, ABC-Net [15] and LQ-Net [23]. The #Gates indicates the lower\nbound on the number of two-input logic gates required to implement the custom hardware accelerators\nfor the convolution, assuming an unrolled architecture and the same throughput. Internally, a 5-bit\nFQ-based inference uses 3-bit unsigned shift quantized weights, with a minimal overhead for the\nadded logic. Scaling constants \u03c3\u2212 and \u03c3+ are equal and thus can be fused into \u03b1l. Perhaps most\nsurprisingly, a 5-bit FQ has more quantization levels yet uses fewer logic gates, when compared to\nABC-Net and LQ-Net implementing the same convolution but with different quantizations. Both\nABC-Net and LQ-Net quantize each weight to N binary values, and compute N parallel binary\nconvolutions for each binary weight. The N outputs are then accumulated for each pixel in the output\nfeature map. In Table 4, they use N = 5 and 2 respectively. Even with the optimal compute pattern\nproposed by the two methods, there are at least O(M N ) additional high-precision multiply-adds,\nwhere M is the number of parallel binary convolutions, and N is the number of output pixels. This\noverhead is much more signi\ufb01cant when compared to other parts of compute in the convolution. As\nshown in Table 4, both have higher logic usage because of the substantial amount of high-precision\nmultiply-adds. In contrast, FQ has only one \ufb01nal learnable layer-wise scaling multiplication that can\nbe further optimized out as it can be fused into BN for inference. Despite having more quantization\nlevels and a higher CR, and being more ef\ufb01cient in hardware resources, the fully quantized ResNet-18\nin Table 3 can still match the accuracy of a LQ-Net ResNet-18.2\n\n2It is also notable that LQ-Net used \u201cpre-activation\u201d ResNet-18 which has a 1.4% advantage in baseline\n\naccuracy compared to ours.\n\n7\n\n\fTable 3: Comparison of the original ResNet-18 with\nsuccessive quantizations applied on weights, activa-\ntions and BN parameters. Each row denotes added\nquantization on new components.\n\nQuantized\n\nBaseline\n+ Weights (5-bit FQ)\n+ Activations (8-bit integer)\n+ BN (16-bit integer)\n\nTop-1\n\u2206\n68.94 \u2014\n68.36\n67.89\n67.95\n\n-0.58\n-1.05\n-0.99\n\nTop-5\n\u2206\n88.67 \u2014\n88.45\n88.08\n88.06\n\n-0.22\n-0.59\n-0.61\n\nTable 4: Computation resource estimates of custom\naccelerators for inference assuming the same compute\nthroughput.\n\nCon\ufb01guration\n\nABC-Net (5 bases, or 5 bits)\nLQ-Net (2 bits)\nShift quantization (3 bits, unsigned)\nFQ (5 bits)\nFQ (5 bits) + Huffman\n\n#Gates\nRatio\n806.1 M 2.93\u00d7\n314.4 M 1.14\u00d7\n275.2 M 1.00\u00d7\n275.6 M 1.00\u00d7\n276.4 M 1.00\u00d7\n\nFigure 4: An implementation of the dot-product used\nin convolution between an integer input and a \ufb01lter\nquantized by recentralized quantization. The notation\n/N means the \ufb01lter values share a common denomi-\nnator N.\n\n4.3 Exploring the Wasserstein Separation\n\nFigure 5: The effect of different threshold values on\nthe Wasserstein distance. The larger the threshold,\nthe fewer the number of layers using recentralized\nquantization instead of shift quantization.\n\nIn Section 3.4, we mentioned that some of the layers in a sparse model may not have multiple\nhigh-probability regions. For this reason, we use the Wasserstein distance W(c1, c2) between the two\ncomponents in the Gaussian mixture model as a metric to differentiate whether recentralized or shift\nquantization should be used. In our experiments, we speci\ufb01ed a threshold wsep = 2.0 such that for\neach layer, if W(c1, c2) \u2265 wsep then recentralized quantization is used, otherwise shift quantization\nis employed instead. Figure 5 shows the impact of choosing different wsep ranging from 1.0 to 3.5\nat 0.1 increments on the Top-1 accuracy. This model is a fast CIFAR-10 [13] classi\ufb01er with only 9\nconvolutional layers, so that it is possible to repeat training 100 times for each wsep value to produce\nhigh-con\ufb01dence results. Note that the average validation accuracy is minimized when the layer\nwith only one high-probability region uses shift quantization and the remaining 8 use recentralized\nquantization, which veri\ufb01es our intuition.\n\n5 Conclusion\n\nIn this paper, we exploit the statistical properties of sparse CNNs and propose focused quantization to\nef\ufb01ciently and effectively quantize model weights. The quantization strategy uses Gaussian mixture\nmodels to locate high-probability regions in the weight distributions and quantize them in \ufb01ne levels.\nCoupled with pruning and encoding, we build a complete compression pipeline and demonstrate high\ncompression ratios on a range of CNNs. In ResNet-18, we achieve 18.08\u00d7 CR with minimal loss in\naccuracies. We additionally show FQ allows a design that is more ef\ufb01cient in hardware resources.\nFurthermore, the proposed quantization uses only powers-of-2 values and thus provides an ef\ufb01cient\ncompute pattern. The signi\ufb01cant reductions in model sizes and compute complexities can translate to\ndirect savings in power ef\ufb01ciencies for future CNN accelerators on loT devices. Finally, FQ and the\noptimized models are fully open-source and released to the public3.\n\nAcknowledgments\n\nThis work is supported in part by the National Key R&D Program of China (No. 2018YFB1004804),\nthe National Natural Science Foundation of China (No. 61806192). We thank EPSRC for providing\nYiren Zhao his doctoral scholarship.\n\n3Available at: https://github.com/deep-fry/mayo.\n\n8\n\nShift values (3-bit)Quantized (5-bit)00.25-220-20.51-200.25-220-20.51-2/ 128/ 128Input (8-bit int)11212354263/ 8(cid:31)(cid:31)00.25-4240-816-6/ 1024XAAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==XAAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==\u21e5AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48k\u21e5AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48koutput14621223XAAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==XAAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==XAAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==XAAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==AAAB63icbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMeAF48RzAOSJcxOepMhM7PLzKwQQn7BiwdFvPpD3vwbZ5M9aGJBQ1HVTXdXlApurO9/e6WNza3tnfJuZW//4PCoenzSNkmmGbZYIhLdjahBwRW2LLcCu6lGKiOBnWhyl/udJ9SGJ+rRTlMMJR0pHnNGbS71TSYH1Zpf9xcg6yQoSA0KNAfVr/4wYZlEZZmgxvQCP7XhjGrLmcB5pZ8ZTCmb0BH2HFVUoglni1vn5MIpQxIn2pWyZKH+nphRacxURq5TUjs2q14u/uf1MhvfhjOu0syiYstFcSaITUj+OBlyjcyKqSOUae5uJWxMNWXWxVNxIQSrL6+T9lU98OvBw3WtQYo4ynAG53AJAdxAA+6hCS1gMIZneIU3T3ov3rv3sWwtecXMKfyB9/kDJ7GOMQ==\u21e5AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48k\u21e5AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48k\u21e5AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48k\u21e5AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48kAAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKewGQY8BLx4jmAckS5idzCZjZmeWmV4hhPyDFw+KePV/vPk3TpI9aGJBQ1HVTXdXlEph0fe/vcLG5tb2TnG3tLd/cHhUPj5pWZ0ZxptMS206EbVcCsWbKFDyTmo4TSLJ29H4du63n7ixQqsHnKQ8TOhQiVgwik5q9VAk3PbLFb/qL0DWSZCTCuRo9MtfvYFmWcIVMkmt7QZ+iuGUGhRM8lmpl1meUjamQ951VFG3JJwurp2RC6cMSKyNK4Vkof6emNLE2kkSuc6E4siuenPxP6+bYXwTToVKM+SKLRfFmSSoyfx1MhCGM5QTRygzwt1K2IgaytAFVHIhBKsvr5NWrRr41eD+qlKv5XEU4QzO4RICuIY63EEDmsDgEZ7hFd487b14797HsrXg5TOn8Afe5w+xW48k \u00b5AAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5cAAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5cAAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5cAAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5c\u00b5AAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5cAAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5cAAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5cAAAB7HicbVDLSgNBEOyNrxhfUY9eBoOQi2FXBD0GvHiM4CaBZAmzk9lkyMzsMg8hLPkGLx4U8eoHefNvnCR70MSChqKqm+6uOONMG9//9kobm1vbO+Xdyt7+weFR9fikrVOrCA1JylPVjbGmnEkaGmY47WaKYhFz2oknd3O/80SVZql8NNOMRgKPJEsYwcZJYV/YweWgWvMb/gJonQQFqUGB1qD61R+mxAoqDeFY617gZybKsTKMcDqr9K2mGSYTPKI9RyUWVEf54tgZunDKECWpciUNWqi/J3IstJ6K2HUKbMZ61ZuL/3k9a5LbKGcys4ZKslyUWI5MiuafoyFTlBg+dQQTxdytiIyxwsS4fCouhGD15XXSvmoEfiN4uK4160UcZTiDc6hDADfQhHtoQQgEGDzDK7x50nvx3r2PZWvJK2ZO4Q+8zx92eI5c \u00b5+AAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5aAAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5aAAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5aAAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5a\u00b5+AAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5aAAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5aAAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5aAAAB7HicbVDLSgNBEOyNrxhfUY9eBoMQEMKuCHoMePEYwU0CyRJmJ7PJkJnZZR5CWPINXjwo4tUP8ubfOEn2oIkFDUVVN91dccaZNr7/7ZU2Nre2d8q7lb39g8Oj6vFJW6dWERqSlKeqG2NNOZM0NMxw2s0UxSLmtBNP7uZ+54kqzVL5aKYZjQQeSZYwgo2Twr6wg8tBteY3/AXQOgkKUoMCrUH1qz9MiRVUGsKx1r3Az0yUY2UY4XRW6VtNM0wmeER7jkosqI7yxbEzdOGUIUpS5UoatFB/T+RYaD0VsesU2Iz1qjcX//N61iS3Uc5kZg2VZLkosRyZFM0/R0OmKDF86ggmirlbERljhYlx+VRcCMHqy+ukfdUI/EbwcF1r1os4ynAG51CHAG6gCffQghAIMHiGV3jzpPfivXsfy9aSV8ycwh94nz9zcI5aLearnablescaling factor\u21b5lAAAB73icbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeCF48V7Ae0oUy2m3bpZhN3N0IJ/RNePCji1b/jzX/jNs1BWx8MPN6bYWZekAiujet+O6WNza3tnfJuZW//4PCoenzS0XGqKGvTWMSqF6BmgkvWNtwI1ksUwygQrBtMbxd+94kpzWP5YGYJ8yMcSx5yisZKvQGKZIJDMazW3Iabg6wTryA1KNAaVr8Go5imEZOGCtS677mJ8TNUhlPB5pVBqlmCdIpj1rdUYsS0n+X3zsmFVUYkjJUtaUiu/p7IMNJ6FgW2M0Iz0aveQvzP66cmvPEzLpPUMEmXi8JUEBOTxfNkxBWjRswsQaq4vZXQCSqkxkZUsSF4qy+vk85lw3Mb3v1VrVkv4ijDGZxDHTy4hibcQQvaQEHAM7zCm/PovDjvzseyteQUM6fwB87nDwZ6j90=AAAB73icbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeCF48V7Ae0oUy2m3bpZhN3N0IJ/RNePCji1b/jzX/jNs1BWx8MPN6bYWZekAiujet+O6WNza3tnfJuZW//4PCoenzS0XGqKGvTWMSqF6BmgkvWNtwI1ksUwygQrBtMbxd+94kpzWP5YGYJ8yMcSx5yisZKvQGKZIJDMazW3Iabg6wTryA1KNAaVr8Go5imEZOGCtS677mJ8TNUhlPB5pVBqlmCdIpj1rdUYsS0n+X3zsmFVUYkjJUtaUiu/p7IMNJ6FgW2M0Iz0aveQvzP66cmvPEzLpPUMEmXi8JUEBOTxfNkxBWjRswsQaq4vZXQCSqkxkZUsSF4qy+vk85lw3Mb3v1VrVkv4ijDGZxDHTy4hibcQQvaQEHAM7zCm/PovDjvzseyteQUM6fwB87nDwZ6j90=AAAB73icbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeCF48V7Ae0oUy2m3bpZhN3N0IJ/RNePCji1b/jzX/jNs1BWx8MPN6bYWZekAiujet+O6WNza3tnfJuZW//4PCoenzS0XGqKGvTWMSqF6BmgkvWNtwI1ksUwygQrBtMbxd+94kpzWP5YGYJ8yMcSx5yisZKvQGKZIJDMazW3Iabg6wTryA1KNAaVr8Go5imEZOGCtS677mJ8TNUhlPB5pVBqlmCdIpj1rdUYsS0n+X3zsmFVUYkjJUtaUiu/p7IMNJ6FgW2M0Iz0aveQvzP66cmvPEzLpPUMEmXi8JUEBOTxfNkxBWjRswsQaq4vZXQCSqkxkZUsSF4qy+vk85lw3Mb3v1VrVkv4ijDGZxDHTy4hibcQQvaQEHAM7zCm/PovDjvzseyteQUM6fwB87nDwZ6j90=AAAB73icbVBNS8NAEJ3Ur1q/qh69LBahp5KIoMeCF48V7Ae0oUy2m3bpZhN3N0IJ/RNePCji1b/jzX/jNs1BWx8MPN6bYWZekAiujet+O6WNza3tnfJuZW//4PCoenzS0XGqKGvTWMSqF6BmgkvWNtwI1ksUwygQrBtMbxd+94kpzWP5YGYJ8yMcSx5yisZKvQGKZIJDMazW3Iabg6wTryA1KNAaVr8Go5imEZOGCtS677mJ8TNUhlPB5pVBqlmCdIpj1rdUYsS0n+X3zsmFVUYkjJUtaUiu/p7IMNJ6FgW2M0Iz0aveQvzP66cmvPEzLpPUMEmXi8JUEBOTxfNkxBWjRswsQaq4vZXQCSqkxkZUsSF4qy+vk85lw3Mb3v1VrVkv4ijDGZxDHTy4hibcQQvaQEHAM7zCm/PovDjvzseyteQUM6fwB87nDwZ6j90=Extract input by0123456789#layers with recentralized quantization1.01.52.02.53.03.5Wasserstein distance9.00%9.25%9.50%9.75%10.00%Top-1 Error\fReferences\n[1] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via\n\nthe EM algorithm. Journal of the Royal Statistical Society, Series B, 1977.\n\n[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-\nscale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern\nRecognition, pages 248\u2013255, June 2009.\n\n[3] Abhimanyu Dubey, Moitreya Chatterjee, and Narendra Ahuja. Coreset-based neural network\ncompression. In Proceedings of the European Conference on Computer Vision (ECCV), pages\n454\u2013470, 2018.\n\n[4] Xitong Gao, Yiren Zhao, \u0141ukasz Dudziak, Robert Mullins, and Cheng-Zhong Xu. Dynamic\nchannel pruning: Feature boosting and suppression. In International Conference on Learning\nRepresentations (ICLR), 2019.\n\n[5] Alex Graves. Practical variational inference for neural networks.\n\nInformation Processing Systems 24. 2011.\n\nIn Advances in Neural\n\n[6] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for ef\ufb01cient DNNs. In\n\nAdvances in Neural Information Processing Systems, 2016.\n\n[7] Song Han, Huizi Mao, and William J Dally. Deep Compression: Compressing deep neural\nnetworks with pruning, trained quantization and Huffman coding. International Conference on\nLearning Representations (ICLR), 2016.\n\n[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image\n\nrecognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.\n\n[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual\n\nnetworks. In European conference on computer vision, pages 630\u2013645. Springer, 2016.\n\n[10] Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing\nIn Proceedings of the Sixth Annual Conference on\n\nthe description length of the weights.\nComputational Learning Theory, COLT \u201993, 1993.\n\n[11] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias\nWeyand, Marco Andreetto, and Hartwig Adam. MobileNets: Ef\ufb01cient convolutional neural\nnetworks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.\n\n[12] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized\nneural networks. In Advances in neural information processing systems, pages 4107\u20134115,\n2016.\n\n[13] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 and CIFAR-100 datasets.\n\nhttp://www.cs.toronto.edu/ kriz/cifar.html, 2014.\n\n[14] Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural\nnetwork: Squeeze the last bit out with ADMM. In Thirty-Second AAAI Conference on Arti\ufb01cial\nIntelligence, 2018.\n\n[15] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network.\n\nIn Advances in Neural Information Processing Systems, pages 345\u2013353, 2017.\n\n[16] Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Penksy. Sparse con-\nvolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition,\n2015.\n\n[17] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A \ufb01lter level pruning method for deep\nneural network compression. In Proceedings of the IEEE International Conference on Computer\nVision, pages 5058\u20135066, 2017.\n\n[18] Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional neural networks using\n\nlogarithmic data representation. CoRR, 2016.\n\n9\n\n\f[19] Milos Nikolic, Mostafa Mahmoud, Andreas Moshovos, Yiren Zhao, and Robert Mullins.\nCharacterizing sources of ineffectual computations in deep learning networks. In 2019 IEEE\nInternational Symposium on Performance Analysis of Systems and Software (ISPASS), 2019.\n\n[20] Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and\n\nquantization. In International Conference on Learning Representations, 2018.\n\n[21] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.\nMobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference\non Computer Vision and Pattern Recognition, pages 4510\u20134520, 2018.\n\n[22] Frederick Tung, Greg Mori, and Simon Fraser. CLIP-Q: Deep network compression learning by\nin-parallel pruning-quantization. 2018 IEEE/CVF Conference on Computer Vision and Pattern\nRecognition, pages 7873\u20137882, 2018.\n\n[23] Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. LQ-Nets: Learned quan-\ntization for highly accurate and compact deep neural networks. In European Conference on\nComputer Vision (ECCV), 2018.\n\n[24] Yiren Zhao, Xitong Gao, Xuan Guo, Junyi Liu, Erwei Wang, Robert Mullins, Peter Cheung,\nGeorge A Constantinides, and Cheng-Zhong Xu. Automatic generation of multi-precision\nmulti-arithmetic CNN accelerators for FPGAs. In 2019 International Conference on Field-\nProgrammable Technology (ICFPT), 2019.\n\n[25] Yiren Zhao, Xitong Gao, Robert Mullins, and Cheng-Zhong Xu. Mayo: A framework for auto-\ngenerating hardware friendly deep neural networks. In Proceedings of the 2nd International\nWorkshop on Embedded and Mobile Deep Learning, EMDL\u201918, pages 25\u201330, 2018.\n\n[26] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen.\n\nIncremental network\nquantization: Towards lossless CNNs with low-precision weights. In International Conference\non Learning Representations, 2017.\n\n[27] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization.\n\nInternational Conference on Learning Representations (ICLR), 2017.\n\n10\n\n\f", "award": [], "sourceid": 2993, "authors": [{"given_name": "Yiren", "family_name": "Zhao", "institution": "University of Cambridge"}, {"given_name": "Xitong", "family_name": "Gao", "institution": "Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences"}, {"given_name": "Daniel", "family_name": "Bates", "institution": "University of Cambridge"}, {"given_name": "Robert", "family_name": "Mullins", "institution": "University of Cambridge"}, {"given_name": "Cheng-Zhong", "family_name": "Xu", "institution": "University of Macau"}]}