{"title": "See and Think: Disentangling Semantic Scene Completion", "book": "Advances in Neural Information Processing Systems", "page_first": 263, "page_last": 274, "abstract": "Semantic scene completion predicts volumetric occupancy and object category of a 3D scene, which helps intelligent agents to understand and interact with the surroundings. In this work, we propose a disentangled framework, sequentially carrying out 2D semantic segmentation, 2D-3D reprojection and 3D semantic scene completion. This three-stage framework has three advantages: (1) explicit semantic segmentation significantly boosts performance; (2) flexible fusion ways of sensor data bring good extensibility; (3) progress in any subtask will promote the holistic performance. Experimental results show that regardless of inputing a single depth or RGB-D, our framework can generate high-quality semantic scene completion, and outperforms state-of-the-art approaches on both synthetic and real datasets.", "full_text": "See and Think: Disentangling Semantic Scene\n\nCompletion\n\nShice Liu1, 2 Yu Hu1, 2 Yiming Zeng1, 2 Qiankun Tang1, 2 Beibei Jin1, 2\n\nYinhe Han1, 2 Xiaowei Li1, 2\n\n1State Key Laboratory of Computer Architecture\n\nInstitute of Computing Technology, Chinese Academy of Sciences\n\n{liushice,huyu,zengyiming,tangqiankun,jinbeibei,yinhes,lxw}@ict.ac.cn\n\n2 University of Chinese Academy of Sciences\n\nAbstract\n\nSemantic scene completion predicts volumetric occupancy and object category\nof a 3D scene, which helps intelligent agents to understand and interact with the\nsurroundings. In this work, we propose a disentangled framework, sequentially\ncarrying out 2D semantic segmentation, 2D-3D reprojection and 3D semantic scene\ncompletion. This three-stage framework has three advantages: (1) explicit semantic\nsegmentation signi\ufb01cantly boosts performance; (2) \ufb02exible fusion ways of sensor\ndata bring good extensibility; (3) progress in any subtask will promote the holistic\nperformance. Experimental results show that regardless of inputing a single depth\nor RGB-D, our framework can generate high-quality semantic scene completion,\nand outperforms state-of-the-art approaches on both synthetic and real datasets.\n\n1\n\nIntroduction\n\nHumans can understand unfamiliar circumstances quickly. For the scene shown in Figure 1, one\ncan recover the shape of the sofa though some parts of it are occluded by clothes. To endow agents\nwith this basic but important capability, semantic scene completion [1] is put forward, which predicts\nvolumetric occupancy and object category of a 3D scene.\nEarly works on semantic scene completion is limited in visible surface partition [2, 3] and shape\nrecovery without considering object category or environment context [4, 5]. Recently, [1, 6] agree\nscene completion and semantic labeling are tightly intertwined and simultaneously generate shapes\nand their categories. However, they not only exploit few semantic features, but are highly customized\nfor a speci\ufb01c sensor. For a more effective and general framework, we present a novel modeling by\nborrowing ideas from human perception. The Gestalt psychologists proposed that segmentation,\nshape assignment and recognition were ordered serially and hierarchically and lower-level cues form\nthe substrate for higher-level cues [7, 8]. Inspired by that, we treat 2D semantic segmentation as\nlower-level cues and leverage it to assist higher-level 3D semantic scene completion.\nWe propose a CNN-based framework to disentangle semantic scene completion. It sequentially\naccomplishes two subtasks, i.e., 2D semantic segmentation and 3D semantic scene completion. These\ntwo subtasks are connected by a 2D-3D reprojection layer. Considering that semantic segmentation\nis to acquire lower-level information as seeing and semantic scene completion is a higher-level task\nas thinking, we name it See And Think Network (SATNet). Figure 1 shows its three modules.\nBased on the framework, we provide various implementations. Note that all these implementations\nneed a depth image, for it is required by the 2D-3D reprojection layer. For a single depth or\nRGB-D input, SATNet can \ufb01nish semantic scene completion ef\ufb01ciently in a single-branch manner.\nMoreover, for the RGB-D input, two more effective double-branch implementations will be introduced.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: See And Think Network. It consists of (a) SNet, (b) a reprojection layer, and (c) TNet,\nsequentially carrying out 2D semantic segmentation, 2D-3D reprojection and 3D semantic scene\ncompletion. Given an RGB-D or HHA (a kind of depth encoding [9]) image, it generates the semantic\nvolume representation of the scene.\n\nWhichever input is given, SATNet outperforms the alternative approaches on both synthetic and real\ndatasets. The advantages of SATNet are summarized below.\nEffectiveness. We demonstrate that explicit semantic segmentation in SATNet boosts semantic\nscene completion, as is similar to human perception. Compared with state-of-the-art works, SATNet\nyields higher performance by 7% to 18% on the synthetic dataset and 3% to 4% on the real dataset.\nExtensibility. The disentangled framework allows us to \ufb02exibly fuse different sensors within\nSATNet. We can replicate subnetworks as multiple branches according to the number of and the type\nof sensors, and then concatenate the corresponding features in SNet or TNet by lazy fusion.\nEvolvability. This disentangled framework also taps the potential of evolving SATNet. As the\ncomplex task is partitioned into two easier subtasks and a simple transformation, it will be con\ufb01rmed\nthat progress in any subtask can heighten the effect of the entire framework.\n\n2 Related Work\n\nWe brie\ufb02y review related works on semantic segmentation and volume reconstruction, and then\ndetail semantic scene completion from two perspectives, model \ufb01tting based completion and voxel\nreasoning based completion.\nSemantic segmentation. Semantic segmentation is to acquire pixel-wise class labeling for an\nimage. Deep learning based semantic segmentation contains two major approaches, i.e., image based\nand volume based. The former leverages the dense pixels of images to reach high performance\n[10\u201313], while the latter attempts to tap the potential of geometric information [14\u201316]. However,\nthese methods consider pixels and voxels separately, thus not combining the advantages of images\nand volumes.\nVolume reconstruction. Volume reconstruction can be \ufb01nished by 3D convolution. With the help\nof large-scale 3D model repository like ShapeNet [17], a large number of works have been developed\nfor exploiting the object shape prior [18\u201323]. By using large-scale 3D scene repository like SUNCG\n[1], learning based scene volume reconstruction is gradually tapped [6, 24].\nModel \ufb01tting based completion. One effective approach to shape completion and semantic label-\ning is to \ufb01t 3D mesh models according to the observed surfaces [25\u201327, 2, 3]. Evidently, it is mainly\nlimited in the capacity of the 3D model library. A large library can provide numerous models, but at\nthe penalty of long time retrieval. A small library is very ef\ufb01cient, but may produce bad matching. To\naddress the issue, [28\u201330] simplify 3D model \ufb01tting to 3D bounding box representation. Nevertheless,\nthese methods sacri\ufb01ce the details of objects for the speed of reconstruction.\nVoxel reasoning based completion. Another popular approach is to complete and label voxels\ndirectly in the voxel space. By extracting features and integrating context, [31\u201333] use physical\n\n2\n\nRGBHHA(a) SNet: 2D Semantic Segmentation(b) Reprojection Layer(c) TNet: 3D Semantic Scene CompletionWallSofaObjectWallSofaObjectFloorFurnitureWindowFloorFurnitureWindowTableChairTableChairWallSofaObjectFloorFurnitureWindowTableChairImage Encoder and DecoderReprojectionVolumetric Convolutions\fFigure 2: Comparison of two upsamplings.\n(a) Use the value of the nearest position in the\ncorresponding feature map. (b) Shuf\ufb02e four\nfeature maps to a bigger one.\n\nFigure 3: 2D-3D reprojection layer. Given\nthe intrinsic camera matrix and the depth image,\nwe reproject the 2D semantic into 3D space and\ndiscretize it.\n\nstructure priors or conditional random \ufb01eld to reconstruct the unobserved part, but they are empirical\nor time-consuming. [34, 35] focus on multiview reconstruction and segmentation for outdoor scenes\nand [36, 37] manage to merge 2D semantic labels into 3D reconstructed surface via semantic visual\nSLAM. Obviously, multiview images are required in these approaches. Recently, [1] formulates\nthe joint task of volumetric completion and semantic labeling as scene semantic completion, and\nproposes SSCNet to accomplish the task end-to-end. However, it takes a single depth image as input\nand does not take advantage of rich features of RGB. Afterwards, [6, 24] attempt to add RGB features\ninto the network. Overall, in the aspect of task solving, these networks introduce a tight coupling\nbetween RGB and depth so that their extensibility and evolvability are constrained.\n\n3 See And Think Network\n\nSATNet is to attain higher-level semantic scene completion with the help of lower-level semantic\nsegmentation. It contains three modules: \ufb01rst, a 2D semantic segmentation subnetwork which\nestimates the semantic by a single RGB or depth image (Sec. 3.1); second, a 2D-3D reprojection\nlayer which transforms the 2D semantic into 3D space so as to complete the 3D scene (Sec. 3.2);\nthird, a 3D semantic scene completion subnetwork that processes the voxels for semantic completion\nof the whole scene (Sec. 3.3). Finally, the double-branch RGB-D fusion in SNet or TNet is discussed\n(Sec. 3.4). Our source code is available at https://github.com/ShiceLiu/SATNet.\n\n3.1 SNet: 2D Semantic Segmentation\n\nThe \ufb01rst issue we need to address is how to acquire semantic segmentation of the scene. Under the\ncondition of the same memory consumption, images have higher resolution than volumes. In addition,\n2D convolutions cost less time than 3D convolutions. Hense, we intend to tackle this subtask in 2D\nimage space. The network of 2D semantic segmentation is named SNet.\nFor a better semantic segmentation result, we utilize an encoder-decoder architecture with skip\nconnections to maintain the features under various receptive \ufb01elds. The encoder is ResNet-101 [38],\nresponsible for extracting multiscale features, and the decoder contains a series of dense upsampling\nconvolutions [39]. As shown in Figure 2, the dense upsampling convolution outputs a feature map\naccording to four feature maps of inputs, enabling details to be learned directly and avoiding the\ninaccuracy of interpolation.\nThe input of SNet is an H \u00d7 W image I, which can be an RGB image Irgb \u2208 R3\u00d7H\u00d7W or a depth\nimage Idepth \u2208 R1\u00d7H\u00d7W . Especially for the depth image input, it should be transformed to a three-\ndepth \u2208 R3\u00d7H\u00d7W mentioned in the method [9], for the purpose of keeping\nchannel HHA image I\u2217\nmore effective information. SNet outputs D-channel semantic feature maps SN et (I) \u2208 RD\u00d7H\u00d7W\nwhich will be fed into the next module, the 2D-3D reprojection layer.\n\n3\n\n(a) Nearest Neighbor UpsamplingC channelsC channelsC/4 channels(b) Dense Upsampling ConvolutionC channels331uxzvKyz\uf0b4\uf0e9\uf0f9\uf0e9\uf0f9\uf0ea\uf0fa\uf0ea\uf0fa=\uf0ea\uf0fa\uf0ea\uf0fa\uf0ea\uf0fa\uf0ea\uf0fa\uf0eb\uf0fb\uf0eb\uf0fb341XxYyRtZz\uf0b4\uf0e9\uf0f9\uf0e9\uf0f9\uf0ea\uf0fa\uf0ea\uf0fa\uf0ea\uf0fa=\uf0e9\uf0f9\uf0eb\uf0fb\uf0ea\uf0fa\uf0ea\uf0fa\uf0ea\uf0fa\uf0ea\uf0fa\uf0eb\uf0fb\uf0eb\uf0fbTransformationby the ExtrinsicCamera Matrix:Transformationby the IntrinsicCamera Matrix:(),uv(),,XYZ\fFigure 4: The overall TNet. The param-\neters of 3D convolution are kernel size,\npadding size and dilation size. The black\ndash lines denote adding while the blue\nsolid lines denote concatenation.\n\nFigure 5: Two double-branch RGB-D fusion ways. The\ndouble-branch fusion of RGB and depth for SATNet can\nbe implemented in SNet or TNet: (a) SNetFuse is to fuse\nat the end of the SNet. (b) TNetFuse is to fuse at the end\nof the TNet.\n\n3.2\n\n2D-3D Reprojection Layer\n\nAllowing for shape recovery in 3D space, the 2D-3D reprojection layer is a vital bond for mapping\nample 2D semantic features to their corresponding 3D spatial positions. Provided the depth image\nIdepth, the intrinsic camera matrix K \u2208 R3\u00d73 and the extrinsic camera matrix [R|t] \u2208 R3\u00d74, each\npixel pu,v (homogeneous coordinates are [u, v, 1]T ) in the image can be reprojected to an individual\n3D point px,y,z (homogeneous coordinates are [X, Y, Z, 1]T ) easily by camera projection equation\npu,v = K [R|t] px,y,z, as shown in Figure 3. After discretizing the 3D scene space into a volume\nwhose size is SX \u00d7 SY \u00d7 SZ, we can establish the mapping M between 2D pixels and 3D voxels,\nand transform the image to the volume of scene surface.\nIn the period of forward propagation, the semantic feature vector valueu,v \u2208 RD for each pixel is\nassigned to its corresponding voxel via the mapping M, while zero vectors are assigned to the empty\nforeground and the occluded areas in the scene. In this way, we obtain a volume \u03c0 (SN et (I)) \u2208\nRD\u00d7SX\u00d7SY \u00d7SZ , re\ufb02ecting volumetric occupancy and semantic labeling of the 3D scene surface.\nIn the period of backward propagation, only the gradients of voxels on the surface of the scene\nare propagated to SN et (I), whereas the gradients of the voxels \ufb01lled with zero vectors are not\npropagated. Overall, the propagation is so simple in either period that it can be regarded as reshaping\nfeature maps. Accordingly, it takes little time to accomplish.\nFurthermore, we rotate the scene volume to align with gravity based on the Manhattan assumption\n[40]. On one hand, it avoids tilted ground, walls or tables and provides better visual appearance. On\nthe other hand, the shapes of objects will be simple due to not considering the angle of pitch or yaw.\nIn practice, this rotation is implemented as an additional item in the extrinsic camera matrix.\n\n3.3 TNet: 3D Semantic Scene Completion\n\nFollowing the 2D-3D reprojection layer, TNet is proposed to accomplish higher-level semantic scene\ncompletion by the volume of lower-level semantic scene surface \u03c0 (SN et (I)) \u2208 RD\u00d7SX\u00d7SY \u00d7SZ .\nAs is shown in Figure 4, TNet completes a 3D scene step-by-step.\nTNet is composed of two residual blocks [38], two atrous spatial pyramid poolings (ASPP) [13] and\ntwo 1 \u00d7 1 convolutions. The \ufb01rst four components compute semantic features of each voxel step-by-\nstep. Then the four groups of feature maps are concatenated for the \ufb01nal category assignment by the\ntwo 1 \u00d7 1 convolutions. In the meantime, the residual block bene\ufb01ts quick convergence while ASPP\naggregates multiscale features to enhance the capability of shape recovery. Thus, we successfully\nobtain the semantic completed scene volume V = T N et (\u03c0 (SN et (I))) \u2208 RC\u00d7SX\u00d7SY \u00d7SZ , where\nC denotes the number of categories we predict.\n\n4\n\nConv3d(3,1,1)Conv3d(3,1,1)inputfeatures 1Conv3d(3,1,1)Conv3d(3,1,1)features 2ASPPASPPfeatures 3features 4ConcatConv3d(1,0,1)Conv3d(1,0,1)output(a) The outline of TNetinputConv3d(3,1,1)Conv3d(3,1,1)Conv3d(3,2,2)Conv3d(3,2,2)Conv3d(3,3,3)Conv3d(3,3,3)output(b) The detail of ASPPRGBHHASNet(Color Branch)SNet(Depth Branch)ConcatReprojectionReprojectionTNet(a) RGB-D Fusion in SNetRGBHHASNet(Depth Branch)SNet(Color Branch)ReprojectionReprojectionReprojectionReprojectionTNet(Color Branch)TNet(Depth Branch)ConcatConcat 1 \uf0b4 1 Conv 1 \uf0b4 1 Conv 1 \uf0b4 1 Conv 1 \uf0b4 1 Conv(b) RGB-D Fusion in TNet\fIn SSCNet [1], the scene surface without semantic information is encoded into a volume for com-\npletion and all the convolutions have \ufb01xed receptive \ufb01elds. Unlike SSCNet, TNet takes advantage\nof semantic scene surface and leverages multiscale feature extractors to yield better performance.\nAdditionally, we will demonstrate these two measures are effective indeed.\n\n3.4 Double-branch RGB-D Fusion\n\nNot only can the three modules in SATNet cooperate with each other for accurate completion results,\nbut also the modular framework of SATNet itself can make it easier to extend branches for fusing\nvarious sensors. As is shown in Figure 5, we introduce two double-branch fusion ways, SNetFuse\nand TNetFuse, to fuse RGB and depth.\nSNetFuse. One double-branch fusion way is to fuse RGB and depth at the end of the SNet. The\nfeature maps output by the color branch and by the depth branch are concatenated and then the\nconcatenated feature maps will be reprojected to 3D space. The branches of RGB and depth share the\nsame architecture but their parameters are not tied, enabling two branches to extract useful features\nfor semantic segmentation respectively. Furthermore, it will be con\ufb01rmed in the next section that\nbetter semantic segmentation is of great bene\ufb01t to the \ufb01nal semantic scene completion.\nTNetFuse. Another double-branch fusion way is to fuse RGB and depth at the end of the TNet,\nwhich integrates the two semantic scene completion results generated respectively by RGB and depth\nfor a better one. Such a fusion manner can be regarded as boosting in the ensemble learning. Each of\nthe two branches consists of SNet, 2D-3D reprojection layer, and TNet, but their parameters are not\ntied. Compared with SNetFuse, TNetFuse can achieve better completion results but at the expense of\nlarger memory and time consumption.\n\n3.5\n\nImplementation Details\n\nData Preprocessing. Data preprocessing contains three parts. (1) Generation of volumetric labels:\nthe size of the output volume is 4.8m (horizontally) \u00d72.88m (vertically) \u00d74.8m (in depth) and the\nvolume is discretized with grid size 0.08m, resulting in a 60 \u00d7 36 \u00d7 60 volume. The generation of\nscene volumetric labels is time consuming, so we prepare all the data of\ufb02ine. (2) Data balance: as\nthe number of non-occupied voxels is much more than the occupied voxels, we randomly sample\nN occupied voxels and 2N non-occupied voxels for training. (3) 2D-3D reprojection mapping\ncalculation: we calculate the mapping M between 2D pixels and 3D voxels in advance.\nTraining Paradigm. We implement our framework in PyTorch. The training procedure consists\nof two steps. We \ufb01rst train 2D semantic segmentation with supervision. Then we initialize the\nweights of SNet and train SATNet end-to-end. We use cross entropy loss and SGD to optimize with a\nmomentum of 0.9, a weight decay of 0.0001 and a batch size of 1. In addition, the learning rate of\nSNet and TNet is 0.001 and 0.01, respectively. It takes us around a week to accomplish the training\nperiod on GeForce GTX 1080Ti GPU.\n\n4 Evaluation\n\n4.1 Evaluation Basis\n\n4.1.1 Datasets\n\nWe evaluate our framework on two benchmark datasets, including the popular NYUv2 dataset [41]\nand the large-scale 3D scene repository SUNCG dataset [1].\nNYUv2. The NYUv2 dataset, a real dataset, is composed of 1449 RGB-D images and is standardly\npartitioned into 795 training samples and 654 testing samples, each associated with an RGB and\ndepth image. Due to no ground truth of volumetric occupancy and semantic labels in NYUv2 dataset,\nwe follow [42] and [43] to attain the ground truth. NYUv2 is a challenging dataset, for it unavoidably\nhas measurement errors and unmeasured areas in the depth images collected from Kinect. Inspired\nby [1], we pretrain the network on SUNCG before \ufb01netuning it on NYUv2 for better performance.\nSUNCG-D and SUNCG-RGBD. The SUNCG dataset, a synthetic dataset, is composed of 45622\nindoor scenes and one can acquire RGB-D images and semantic scene volumes by setting different\n\n5\n\n\fTable 1: Semantic scene completion results on NYUv2 dataset.\n\n\ufb02oor wall win.\n14.1\n11.7\n5.8\n62.5\n94.6\n10.8\n\n13.3\n19.1\n24.7\n\nSemantic scene completion\ntable\nchair\n6.0\n9.4\n7.0\n8.5\n17.3\n15.9\n\nbed\n29.0\n40.6\n53.2\n\nsofa\n24.0\n27.7\n45.9\n\nScene completion\nprec.\n58.5\n65.7\n59.3\n62.5\n69.5\n66.8\n69.2\n67.6\n67.3\n67.3\n64.4\n\nmethod\nLin [29]\nGeiger [26]\nSong [1]\nGuedes [6]1\n92.5\nGarbade [24]\n91.3\nDepth\n87.0\nRGBD\n91.0\nSNetFuse\n92.1\nTNetFuse\n94.5\nNoSNet\nGTSNet\n94.5\n1 Guedes et al. [6] did not provide class-wise results.\n\nrecall\n49.9\n58.0\n92.9\n82.3\n82.7\n86.6\n81.2\n85.9\n85.8\n84.4\n92.5\n\nIoU ceil.\n0.0\n36.4\n10.2\n44.4\n56.6\n15.1\n54.3\n60.7\n60.6\n59.5\n60.7\n60.6\n59.5\n61.3\n\n12.9\n20.6\n22.5\n22.2\n17.3\n19.8\n22.3\n\n-\n\n-\n\n-\n\n25.3\n27.0\n30.0\n28.6\n28.0\n28.1\n32.0\n\n-\n\n20.1\n9.2\n21.1\n18.2\n16.6\n0.7\n27.6\n\n-\n\n16.1\n19.5\n17.9\n19.2\n19.3\n15.4\n22.7\n\n-\n\n56.3\n56.9\n52.4\n56.2\n57.5\n50.8\n58.4\n\n-\n\n43.4\n54.7\n44.5\n51.2\n53.8\n43.2\n58.9\n\n-\n\n17.2\n16.9\n15.1\n16.2\n17.7\n15.7\n24.8\n\ntvs.\n7.0\n6.0\n13.9\n\n-\n\n10.4\n15.2\n19.5\n12.2\n18.5\n11.0\n28.0\n\nfurn.\n16.2\n22.6\n31.1\n\n-\n\n33.0\n37.1\n36.0\n37.0\n38.4\n31.9\n49.4\n\nobjs.\n1.1\n5.9\n12.6\n\n-\n\n14.3\n15.7\n17.3\n17.4\n18.9\n7.7\n29.8\n\navg.\n12.0\n19.6\n30.5\n27.5\n31.0\n33.1\n33.0\n33.6\n34.4\n29.0\n40.8\n\nFigure 6: Qualitative Results on NYUv2 Dataset. We show the qualitative results generated by\ndouble-branch TNetFuse on NYUv2 Dataset. Compared with the results of SSCNet [1], SATNet\ngives more accurate semantic scene completions such as the windows and pictures on the wall.\n\ncamera positions. Song et al. [1] has provided a subset of SUNCG, made up of depth images and\ntheir volumes, so we call it SUNCG-D. Besides, we generate another subset SUNCG-RGBD, made\nup of RGB-D images and their volumes. SUNCG-D consists of 139368 training samples and 470\ntesting samples, while SUNCG-RGBD consists of 13011 training samples and 499 testing samples.\nAlthough the SUNCG dataset is synthetic, the evaluation on the SUNCG dataset is quite signi\ufb01cant.\nConsidering that the limited number of models in the NYUv2 dataset might result in memorizing\nmodels, experiments in the large-scale 3D scene repository, i.e. SUNCG-D and SUNCG-RGBD,\ncould relieve this issue to some degree.\n\n4.1.2 Metric\n\nWe mainly evaluate our framework with intersection over union (IoU) between our predictions and\nground truth, and two tasks are considered: scene completion and semantic scene completion. For the\nformer, we treat all voxels as binary predictions, i.e., occupied or non-occupied. For the latter, we pay\nattention to IoUs of each class and average them to get the mean IoU of semantic scene completion.\n\n4.2 Experimental Results\n\n4.2.1 Comparison to Alternative Approaches\nComparison on NYUv2 dataset. Table 1 presents the semantic scene completion results on\nNYUv2 dataset with comparison to some alternative approaches. Lin et al. [29] uses 3D bounding\nboxes to approximate objects, while Geiger et al. [26] retrieves 3D models to \ufb01nd the best matching\nto represent the object. Song et al. [1] proposes SSCNet to predict volumetric occupancy and class\nlabels simultaneously by a single depth image. Guedes et al. [6] utilizes two SSCNets for RGB and\n\n6\n\nimageground truthSATNetSSCNetimageground truthSATNetSSCNetwallfloorwindowchairbedsofatabletelevisionfurnitureobject\fTable 2: Semantic scene completion results on SUNCG-D dataset.\n\nScene completion\nprec.\n76.3\n80.7\n77.7\n80.9\n79.2\n\nrecall\n95.2\n96.5\n95.1\n95.3\n94.5\n\nIoU ceil.\n96.3\n73.5\n97.9\n78.5\n97.2\n74.9\n78.0\n97.8\n97.5\n75.8\n\n\ufb02oor wall win.\n84.9\n28.2\n58.5\n82.5\n44.4\n80.9\n82.1\n57.4\n29.4\n80.8\n\n56.8\n57.7\n52.7\n56.8\n56.7\n\nSemantic scene completion\ntable\nchair\n33.7\n21.3\n47.3\n45.1\n34.0\n33.6\n41.1\n45.7\n34.4\n33.0\n\nsofa\n52.7\n72.3\n62.5\n71.3\n65.0\n\nbed\n56.0\n78.4\n69.6\n72.6\n61.5\n\ntvs.\n10.9\n45.7\n25.5\n44.0\n28.8\n\nfurn.\n44.3\n67.1\n49.0\n60.3\n50.2\n\nobjs.\n25.4\n55.2\n39.3\n51.9\n38.0\n\navg.\n46.4\n64.3\n53.5\n61.9\n52.3\n\nTable 3: Semantic scene completion results on SUNCG-RGBD dataset.\n\nScene completion\nprec.\n43.5\n52.3\n49.8\n56.7\n53.9\n49.2\n50.7\n50.9\n48.3\n50.8\n53.4\n\nrecall\n90.7\n92.7\n94.3\n91.7\n95.2\n93.8\n94.6\n92.8\n94.8\n92.8\n95.0\n\nIoU ceil.\n64.9\n41.5\n62.5\n50.2\n59.0\n48.3\n53.9\n65.5\n60.6\n52.6\n45.4\n47.6\n47.5\n49.4\n65.7\n49.0\n64.1\n47.1\n66.6\n48.8\n52.0\n63.7\n\n\ufb02oor wall win.\n25.2\n60.1\n58.5\n57.8\n50.6\n45.0\n60.7\n56.4\n52.7\n57.3\n38.4\n57.3\n41.3\n55.2\n58.5\n46.2\n48.8\n51.0\n30.6\n56.7\n55.6\n61.8\n\n57.6\n48.6\n46.0\n50.3\n53.2\n48.0\n50.1\n48.3\n45.1\n41.7\n48.0\n\nSemantic scene completion\ntable\nchair\n23.1\n25.5\n26.9\n24.4\n26.8\n24.9\n30.6\n26.1\n27.4\n28.6\n18.2\n20.3\n22.8\n22.5\n25.8\n25.4\n20.1\n23.7\n22.1\n22.7\n33.9\n38.0\n\nsofa\n37.9\n50.4\n49.0\n43.7\n53.3\n36.6\n44.4\n46.9\n42.5\n36.4\n57.4\n\nbed\n40.4\n46.5\n42.0\n47.3\n46.8\n36.9\n41.3\n47.0\n41.1\n47.1\n51.3\n\ntvs.\n29.8\n41.1\n40.8\n37.2\n41.1\n26.7\n33.7\n34.2\n41.7\n25.2\n43.6\n\nfurn.\n45.7\n40.7\n46.6\n44.9\n44.1\n36.6\n40.7\n38.8\n34.4\n30.7\n48.5\n\nobjs.\n4.7\n20.2\n22.4\n30.0\n29.0\n15.2\n16.1\n20.1\n24.3\n13.4\n28.8\n\navg.\n37.7\n43.4\n41.2\n44.8\n44.9\n34.5\n37.8\n41.5\n39.7\n35.7\n48.2\n\nmethod\nSong [1]\nDepth\nDepth (w/o i)\nDepth (w/o a)\nNoSNet\n\nmethod\nSong [1]\nDepth\nRGBD\nSNetFuse\nTNetFuse\nDepth (w/o i)\nRGBD (w/o i)\nDepth (w/o a)\nRGBD (w/o a)\nNoSNet\nGTSNet\n\ndepth respectively and combines the outputs of two SSCNet branches by concatenation. However,\n[6] directly processes low-resolution RGB scene surface in 3D space, resulting in losing lots of\ndetails. Garbade et al. [24] processes RGB by a series of 2D convolutions which is similar to our\nSNet, but directly follows SSCNet for depth. For SATNet, we experiment with four implementations:\nsingle-branch for depth input (Depth), single-branch for RGB-D input (RGBD), double-branch\nSNetFuse for RGB-D input (SNetFuse) and double-branch TNetFuse for RGB-D input (TNetFuse).\nBesides, we further do some ablation studies and discuss them in Sec. 4.2.2.\nCompared to the other approaches in Table 1, the SATNet variants have the overall highest accuracy.\nFor a single depth input, our approach produces more accurate predictions (Depth 33.1% vs. Song et\nal. [1] 30.5%). For RGB-D input, our approach gets even higher IoU by double-branch TNetFuse\n(TNetFuse 34.4% vs. Garbade et al. [24] 31.0%). In addition, for each category, we also acquire\ngenerally higher IoUs. Thus, 2D semantic segmentation for depth or RGB can promote the \ufb01nal\ncompletion. Figure 6 shows the qualitative results, where we complete six different scenes via\ndouble-branch TNetFuse and compare the results with SSCNet. Well as SSCNet works for many\ncases, it fails in the objects which are hard to distinguish from depth. For instance, SSCNet mistakes\nthe television as another object in the second row of Figure 6, and fails to complete the pictures on\nthe wall in the third row. In contrast, SATNet leverages RGB to overcome this dif\ufb01culty.\nComparison on SUNCG dataset. Table 2 and Table 3 presents the quantitative results on SUNCG-\nD and SUNCG-RGBD dataset respectively. We mainly compare our framework with the benchmark\napproach Song et al. [1], because other approaches do not provide source data on SUNCG or source\ncode. It can be seen that for the depth input, SATNet yields higher IoU for both SUNCG-D and\nSUNCG-RGBD datasets by 17.9% and 5.7% respectively, and the effect will be more prominent\nfor the RGB-D input. Besides, the less noisy measurement on SUNCG dataset accounts for more\naccurate scene completions and higher mean IoU of semantic scene completion on SUNCG dataset\nthan on NYUv2 dataset. We further do a lot of extra ablation studies which are detailed in Sec. 4.2.2.\n\n4.2.2 Ablation Study\n\nIs disentangling semantic segmentation helpful?\n[1] has proved that the semantic will help scene\ncompletion. Moreover, we intend to demonstrate that disentangling semantic segmentation is much\nmore helpful for semantic scene completion, by means of presenting several ablation studies. Firstly,\nto exclude the effect of different inputs, we compare with [1] for the single depth input and compare\nwith [6, 24] for the RGB-D input. Whatever input is provided, SATNet has better performance on both\nNYUv2 and SUNCG dataset. Secondly, to exclude the effect of the network structure, we maintain\nthe same structure with SATNet but do not initialize SNet (represented as (w/o i)) with the weights\ntrained on the semantic segmentation task. We examine in two single-branch situations, where SNet\n\n7\n\n\fFigure 7: The convergence speed w/ or w/o se-\nmantic segmentation initialization.\n\nFigure 8: The relationship of the average IoU\nand the segmentation accuracy.\n\nFigure 9: The corrections made by the TNet.\n\nis provided with a depth image (Depth (w/o i)) or an RGB image (RGBD (w/o i)). As shown in Table\n2 and Table 3, the network with initialization has higher performance than that without initialization.\nThirdly, we \ufb01nd the explicit semantic segmentation could accelerate the speed of convergence. Figure\n7 shows the loss of completion with 2D semantic segmentation initialization descends faster than\nthat without initialization. Finally, to demonstrate disentangling semantic segmentation is helpful,\nTable 1 and Table 3 show the groundtruth semantic segmentation (GTSNet) could generate better\ncompletions than RGB or depth semantic segmentation, and much better than those without semantic\nsegmentation (NoSNet).\nWhat is the effect of SNet? By reason that disentangling semantic segmentation is helpful, SNet\nplays an important role in generating semantic segmentation from various inputs. Moreover, in this\npart, we intend to probe to what extent the semantic segmentation bene\ufb01ts semantic scene completion.\nFigure 8 shows the relationship of the average IoU and the segmentation accuracy. While the accuracy\nof semantic segmentation is improved, the average IoU of semantic scene completion is bumped\nup. When the accuracy of segmentation reaches 60%, other parts instead of SNet would become\nthe bottleneck and the average IoU of completion suffers slow growth. Whatever, better semantic\nsegmentation always results in better semantic scene completion.\nWhat is the effect of TNet? Fed with semantic scene surfaces, TNet generates semantic scene\ncompletions. For the purpose of accurate completions, TNet is supposed to not only leverage semantic\nsegmentation to assist completion, but also correct the mistakes made in the segmentation. Table\n4 indicates the number and the ratio of correct/wrong completed voxels vary with correct/wrong\nsegmented pixels (the pixels correspond to the voxels on the surface of the scene by the 2D-3D\nreprojection layer). As Table 4 shows, when the segmentation is correct, the probability of correct\ncompletion is 79%. When the segmentation is wrong, TNet is able to correct 55% mistakes occurred\nin the segmentation. To make the process of correction clear, Figure 9 shows some sensible corrections\nmade by the TNet.\nIs aggregating multiscale features useful? As ASPP has attained great success in semantic seg-\nmentation, we intend to con\ufb01rm that there is also its place in semantic scene completion. Hence,\n\n8\n\n020406080100the accuracy of semantic segmentation (%)28303234363840the average IoU of SATNet (%)imageground truthSATNetimageSATNetwallfloorwindowchairbedsofatabletelevisionsegmentationsegmentationfurnitureobjectground truth\fThe result of semantic scene completion\n\nCorrect\n\nWrong\n\nCorrect\n\n445514 (79%)\n\n119445 (21%)\n\nWrong\n\n109656 (55%)\n\n89272 (45%)\n\nSNetFuse TNetFuse\n\ntime for training (s/iter)\ntime for testing (s/iter)\nmemory for training (GB)\nmemory for testing (GB)\nstorage of parameters (GB)\n\n0.9\n0.2\n7.3\n2.2\n1.1\n\n1.3\n1.3\n13.7\n3.5\n1.2\n\nThe result of\n\nsemantic\n\nsegmentation\n\nTable 4: The number and the ratio of cor-\nrect/wrong completed voxels vary with the cor-\nrect/wrong segmented pixels.\n\nTable 5: Time and memory consumption\nfor two double-branch fusions (SNetFuse and\nTNetFuse).\n\nwe compare the results with and without ASPP (represented as (w/o a)) in SATNet. Similarly, we\nexamine on both depth input (Depth (w/o a)) and RGB-D input (RGBD (w/o a)). On SUNCG-D\ndataset, ASPP boosts IoUs in every class overall and yields 2.4% performance improvement averagely.\nOn SUNCG-RGBD dataset, ASPP also brings 1.9% and 1.5% performance improvement for depth\ninput and RGB-D input, respectively.\nIs there any difference for different inputs? Although the mean IoU of single-branch RGB-D\ninput and depth input on NYUv2 dataset are approximately equal, we \ufb01nd SATNet has different\npreferences in terms of two types of inputs. For the real scene dataset NYUv2, the IoU of each\ncategory differs. RGB-D inputs promote completing objects with unique colors and textures, such as\nwindows and televisions; while depth inputs promote completing objects with clear shapes, such as\nbeds and sofas. For the synthetic dataset SUNCG, due to the inauthentic RGB images, objects with\nlarge contrast will be overwhelmingly better, such as furniture and objects. Furthermore, the result\nof double-branch SNetFuse is better than single-branch RGB-D or depth input on both datasets, on\naccount of more accurate semantic segmentation by RGB-D than by a single RGB or depth.\nIs there an absolutely best fusion way? To answer this question, we take some extra metrics into\naccount. Table 5 presents time and memory consumption for two fusions. They have approximately\nequal number of network parameters, but TNetFuse is much slower than SNetFuse. In fact, the\ntime-consuming 3D convolutions account for the slow speed of TNet and numerous 3D feature maps\naccount for its large memory consumption. Hence, we should choose the fusion approach that suits\nthe actual demands, even though TNetFuse achieves higher performance indeed.\nIs it possible to evolve SATNet? In order to demonstrate the evolvability of SATNet, we improve\nSATNet by individually enhancing the two subtasks. (1) Progress in SNet. Figure 8 shows what\nwill happen if SNet produces more accurate segmentation. And the improvement in semantic\nsegmentation will boost semantic scene completion to different extent. (2) Progress in TNet. In\nthe previous discussion, we have shown that ASPP bene\ufb01ts semantic scene completion, which is an\nexample that more effective components in TNet account for higher performance on completion.\nLimitations. We do not discuss how the volumetric resolution in\ufb02uences semantic scene com-\npletion, owing to much larger memory requirement for large 3D volume. Intuitively speaking,\nlow-resolution volumes might make some objects appear strange and restrict the effect. Moreover, to\nsome extent, disentangling semantic segmentation from semantic scene completion is a recommen-\ndation given by us. It might be more effective and valuable to allow the network to spontaneously\ndecide what subtasks it should disentangle.\n\n5 Conclusion\n\nIn this paper, we propose SATNet, a disentangled semantic scene completion framework. It consists\nof SNet for 2D semantic segmentation, a 2D-3D reprojection layer, and TNet for semantic scene com-\npletion. We provide single-branch and double-branch implementations within the SATNet framework,\nwhich demonstrates the SATNet framework is effective, extensible and evolvable. Experimental\nresults show SATNet outperforms the state-of-the-art approaches.\n\n9\n\n\fAcknowledgments\n\nWe thank Shuran Song for sharing the SSCNet results and the SUNCG development kits. This work\nis supported in part by National Natural Science Foundation of China under grant No. 61274030, and\nin part by Innovation Project of Institute of Computing Technology, Chinese Academy of Sciences\nunder grant No. 20186090. Yu Hu is the corresponding author.\n\nReferences\n[1] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic\nScene Completion from a Single Depth Image. In IEEE Conference on Computer Vision and Pattern\nRecognition (CVPR), pages 190\u2013198, 2017.\n\n[2] Saurabh Gupta, Pablo Arbelaez, and Jitendra Malik. Perceptual Organization and Recognition of Indoor\nScenes from RGB-D Images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\npages 564\u2013571, 2013.\n\n[3] Xiaofeng Ren, Liefeng Bo, and Dieter Fox. RGB-(D) Scene Labeling: Features and Algorithms. In IEEE\n\nConference on Computer Vision and Pattern Recognition (CVPR), pages 2759\u20132766, 2012.\n\n[4] Michael Firman, Oisin Mac Aodha, Simon Julier, and Gabriel J Brostow. Structured Prediction of\nUnobserved Voxels from a Single Depth Image. In IEEE Conference on Computer Vision and Pattern\nRecognition (CVPR), pages 5431\u20135440, 2016.\n\n[5] Duc Thanh Nguyen, Binh-Son Hua, Khoi Tran, Quang-Hieu Pham, and Sai-Kit Yeung. A Field Model for\nRepairing 3D Shapes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages\n5676\u20135684, 2016.\n\n[6] Andre Bernardes Soares Guedes, Teo\ufb01lo Emidio de Campos, and Adrian Hilton. Semantic Scene Comple-\n\ntion Combining Colour and Depth: Preliminary Experiments. arXiv preprint arXiv:1802.04735, 2018.\n\n[7] E Bruce Goldstein. The Blackwell Handbook of Sensation and Perception. John Wiley & Sons, 2008.\n\n[8] Allison B Sekuler and Stephen E Palmer. Perception of Partly Occluded Objects: A Microgenetic Analysis.\n\nJournal of Experimental Psychology: General, 121(1):95, 1992.\n\n[9] Saurabh Gupta, Ross Girshick, Pablo Arbel\u00e1ez, and Jitendra Malik. Learning Rich Features from RGB-D\nImages for Object Detection and Segmentation. In European Conference on Computer Vision (ECCV),\npages 345\u2013360, 2014.\n\n[10] Fisher Yu and Vladlen Koltun. Multi-Scale Context Aggregation by Dilated Convolutions. arXiv preprint\n\narXiv:1511.07122, 2015.\n\n[11] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Re\ufb01neNet: Multi-Path Re\ufb01nement Networks\nIn IEEE Conference on Computer Vision and Pattern\n\nfor High-Resolution Semantic Segmentation.\nRecognition (CVPR), pages 5168\u20135177, 2017.\n\n[12] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid Scene Parsing\nNetwork. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2881\u20132890,\n2017.\n\n[13] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab:\nSemantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected\nCRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834\u2013848, 2018.\n\n[14] Daniel Maturana and Sebastian Scherer. VoxNet: A 3D Convolutional Neural Network for Real-Time\nObject Recognition. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),\npages 922\u2013928, 2015.\n\n[15] Charles R Qi, Hao Su, Matthias Nie\u00dfner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric\nand Multi-View CNNs for Object Classi\ufb01cation on 3D Data. In IEEE Conference on Computer Vision and\nPattern Recognition (CVPR), pages 5648\u20135656, 2016.\n\n[16] R Qi Charles, Hao Su, Mo Kaichun, and Leonidas J Guibas. PointNet: Deep Learning on Point Sets for\n3D Classi\ufb01cation and Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), pages 77\u201385, 2017.\n\n10\n\n\f[17] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio\nSavarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An Information-Rich 3D Model\nRepository. arXiv preprint arXiv:1512.03012, 2015.\n\n[18] Jiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and\nWilliam T Freeman. Single Image 3D Interpreter Network. In European Conference on Computer Vision\n(ECCV), pages 365\u2013382, 2016.\n\n[19] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3D-R2N2: A Uni\ufb01ed\nApproach for Single and Multi-view 3D Object Reconstruction. In European Conference on Computer\nVision (ECCV), pages 628\u2013644, 2016.\n\n[20] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a Predictable and\nGenerative Vector Representation for Objects. In European Conference on Computer Vision (ECCV),\npages 484\u2013499, 2016.\n\n[21] Danilo Jimenez Rezende, SM Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas\nHeess. Unsupervised Learning of 3D Structure from Images. In Advances in Neural Information Processing\nSystems (NIPS), pages 4996\u20135004, 2016.\n\n[22] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. MarrNet: 3D\nShape Reconstruction via 2.5D Sketches. In Advances in Neural Information Processing Systems (NIPS),\npages 540\u2013550, 2017.\n\n[23] Abhishek Kar, Christian H\u00e4ne, and Jitendra Malik. Learning a Multi-View Stereo Machine. In Advances\n\nin Neural Information Processing Systems (NIPS), pages 364\u2013375, 2017.\n\n[24] Martin Garbade, Johann Sawatzky, Alexander Richard, and Juergen Gall. Two Stream 3D Semantic Scene\n\nCompletion. arXiv preprint arXiv:1804.03550, 2018.\n\n[25] Saurabh Gupta, Pablo Arbel\u00e1ez, Ross Girshick, and Jitendra Malik. Aligning 3D Models to RGB-D Images\nof Cluttered Scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages\n4731\u20134740, 2015.\n\n[26] Andreas Geiger and Chaohui Wang. Joint 3D Object and Layout Inference from a Single RGB-D Image.\n\nIn German Conference on Pattern Recognition, pages 183\u2013195, 2015.\n\n[27] Oliver Mattausch, Daniele Panozzo, Claudio Mura, Olga Sorkine-Hornung, and Renato Pajarola. Object\nDetection and Classi\ufb01cation from Large-Scale Cluttered Indoor Scans. In Computer Graphics Forum,\nvolume 33, pages 11\u201321, 2014.\n\n[28] Hao Jiang and Jianxiong Xiao. A Linear Approach to Matching Cuboids in RGBD Images. In IEEE\n\nConference on Computer Vision and Pattern Recognition (CVPR), pages 2171\u20132178, 2013.\n\n[29] Dahua Lin, Sanja Fidler, and Raquel Urtasun. Holistic Scene Understanding for 3D Object Detection with\nRGBD Cameras. In IEEE International Conference on Computer Vision (ICCV), pages 1417\u20131424, 2013.\n\n[30] Shuran Song and Jianxiong Xiao. Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images.\n\nIn IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 808\u2013816, 2016.\n\n[31] Bo Zheng, Yibiao Zhao, C Yu Joey, Katsushi Ikeuchi, and Song-Chun Zhu. Beyond Point Clouds: Scene\nUnderstanding by Reasoning Geometry and Physics. In IEEE Conference on Computer Vision and Pattern\nRecognition (CVPR), pages 3127\u20133134, 2013.\n\n[32] Byung-soo Kim, Pushmeet Kohli, and Silvio Savarese. 3D Scene Understanding by Voxel-CRF. In IEEE\n\nInternational Conference on Computer Vision (ICCV), pages 1425\u20131432, 2013.\n\n[33] Nathan Silberman, Lior Shapira, Ran Gal, and Pushmeet Kohli. A Contour Completion Model for\nIn European Conference on Computer Vision (ECCV), pages\n\nAugmenting Surface Reconstructions.\n488\u2013503, 2014.\n\n[34] Christian H\u00e4ne, Christopher Zach, Andrea Cohen, Roland Angst, and Marc Pollefeys. Joint 3D Scene\nReconstruction and Class Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), pages 97\u2013104, 2013.\n\n[35] Maros Blaha, Christoph Vogel, Audrey Richard, Jan D Wegner, Thomas Pock, and Konrad Schindler.\nLarge-Scale Semantic 3D Reconstruction: an Adaptive Multi-Resolution Model for Multi-Class Volumetric\nLabeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3176\u20133184,\n2016.\n\n11\n\n\f[36] Keisuke Tateno, Federico Tombari, and Nassir Navab. When 2.5D is Not Enough: Simultaneous Recon-\nstruction, Segmentation and Recognition on Dense SLAM. In IEEE International Conference on Robotics\nand Automation (ICRA), pages 2295\u20132302, 2016.\n\n[37] Keisuke Tateno, Federico Tombari, Iro Laina, and Nassir Navab. CNN-SLAM: Real-Time Dense Monocular\nSLAM with Learned Depth Prediction. In IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), pages 6565\u20136574, 2017.\n\n[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition.\n\nIn IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770\u2013778, 2016.\n\n[39] Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell.\n\nUnderstanding Convolution for Semantic Segmentation. arXiv preprint arXiv:1702.08502, 2017.\n\n[40] Bruno Caprile and Vincent Torre. Using Vanishing Points for Camera Calibration. International Journal\n\nof Computer Vision, 4(2):127\u2013139, 1990.\n\n[41] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor Segmentation and Support\nInference from RGBD Images. In European Conference on Computer Vision (ECCV), pages 746\u2013760,\n2012.\n\n[42] Ruiqi Guo, Chuhang Zou, and Derek Hoiem. Predicting Complete 3D Models of Indoor Scenes. arXiv\n\npreprint arXiv:1504.02437, 2015.\n\n[43] Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, and Roberto Cipolla. SceneNet:\n\nUnderstanding Real World Indoor Scenes with Synthetic Data. arXiv preprint arXiv:1511.07041, 2015.\n\n12\n\n\f", "award": [], "sourceid": 186, "authors": [{"given_name": "Shice", "family_name": "Liu", "institution": "Institute of Computing Technology, Chinese Academy of Sciences"}, {"given_name": "YU", "family_name": "HU", "institution": "Institute of Computing Technology, Chinese Academy of Sciences"}, {"given_name": "Yiming", "family_name": "Zeng", "institution": "Institute of Computing Technology Chinese Academy of Sciences"}, {"given_name": "Qiankun", "family_name": "Tang", "institution": "Institute of Computing Technology, Chinese Academy of Sciences"}, {"given_name": "Beibei", "family_name": "Jin", "institution": "Institute of Computing Technology, Chinese Academy of Sciences"}, {"given_name": "Yinhe", "family_name": "Han", "institution": "Institute of Computing Technology, Chinese Academy of Sciences"}, {"given_name": "Xiaowei", "family_name": "Li", "institution": "Institute of Computing Technology, Chinese Academy of Sciences"}]}