{"title": "Adversarial Scene Editing: Automatic Object Removal from Weak Supervision", "book": "Advances in Neural Information Processing Systems", "page_first": 7706, "page_last": 7716, "abstract": "While great progress has been made recently in automatic image manipulation, it has been limited to object centric images like faces or structured scene datasets.\nIn this work, we take a step towards general scene-level image editing by developing an automatic interaction-free object removal model. Our model learns to find and remove objects from general scene images using image-level labels and unpaired data in a generative adversarial network (GAN) framework. We achieve this with two key contributions: a two-stage editor architecture consisting of a mask generator and image in-painter that co-operate to remove objects, and a novel GAN based prior for the mask generator that allows us to flexibly incorporate knowledge about object shapes. We experimentally show on two datasets that our method effectively removes a wide variety of objects using weak supervision only.", "full_text": "Adversarial Scene Editing:\n\nAutomatic Object Removal from Weak Supervision\n\nRakshith Shetty1\n\nMario Fritz2\n\nBernt Schiele1\n\n1Max Planck Institute for Informatics, Saarland Informatics Campus\n\n2CISPA Helmholtz Center i.G., Saarland Informatics Campus\n\nSaarbr\u00fccken, Germany\n\n1firstname.lastname@mpi-inf.mpg.de\n2firstname.lastname@cispa.saarland\n\nAbstract\n\nWhile great progress has been made recently in automatic image manipulation, it\nhas been limited to object centric images like faces or structured scene datasets. In\nthis work, we take a step towards general scene-level image editing by developing\nan automatic interaction-free object removal model. Our model learns to \ufb01nd and\nremove objects from general scene images using image-level labels and unpaired\ndata in a generative adversarial network (GAN) framework. We achieve this\nwith two key contributions: a two-stage editor architecture consisting of a mask\ngenerator and image in-painter that co-operate to remove objects, and a novel GAN\nbased prior for the mask generator that allows us to \ufb02exibly incorporate knowledge\nabout object shapes. We experimentally show on two datasets that our method\neffectively removes a wide variety of objects using weak supervision only.\n\n1\n\nIntroduction\n\nAutomatic editing of scene-level images to add/remove objects and manipulate attributes of objects\nlike color/shape etc. is a challenging problem with a wide variety of applications. Such an editor can\nbe used for data augmentation [1], test case generation, automatic content \ufb01ltering and visual privacy\n\ufb01ltering [2]. To be scalable, the image manipulation should be free of human interaction and should\nlearn to perform the editing without needing strong supervision. In this work, we investigate such\nan automatic interaction free image manipulation approach that involves editing an input image to\nremove target objects, while leaving the rest of the image intact.\nThe advent of powerful generative models like generative adversarial networks (GAN) has led to\nsigni\ufb01cant progress in various image manipulation tasks. Recent works have demonstrated altering\nfacial attributes like hair color, orientation [3], gender [4] and expressions [5] and changing seasons\nin scenic photographs [6]. An encouraging aspect of these works is that the image manipulation is\nlearnt without ground truth supervision, but with using unpaired data from different attribute classes.\nWhile this progress is remarkable, it has been limited to single object centric images like faces or\nconstrained images like street scenes from a single point of view [7]. In this work we move beyond\nthese object-centric images and towards scene-level image editing on general images. We propose an\nautomatic object removal model that takes an input image and a target class and edits the image to\nremove the target object class. It learns to perform this task with only image-level labels and without\nground truth target images, i.e. using only unpaired images containing different object classes.\nOur model learns to remove objects primarily by trying to fool object classi\ufb01ers in a GAN framework.\nHowever, simply training a generator to re-synthesize the input image to fool object classi\ufb01ers leads\nto degenerate solutions where the generator uses adversarial patterns to fool the classi\ufb01ers. We\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\faddress this problem with two key contributions. First we propose a two-stage architecture for\nour generator, consisting of a mask generator, and an image in-painter which cooperate to achieve\nremoval. The mask generator learns to fool the object classi\ufb01er by masking some pixels, while the\nin-painter learns to make the masked image look realistic. The second part of our solution is a GAN\nbased framework to impose shape priors on the mask generator to encourage it to produce compact\nand coherent shapes. The \ufb02exible framework allows us to incorporate different shape priors, from\nrandomly sampled rectangles to unpaired segmentation masks from a different dataset. Furthermore,\nwe propose a novel locally supervised real/fake classi\ufb01er to improve the performance of our in-painter\nfor object removal. Our experiments show that our weakly supervised model achieves on par results\nwith a baseline model using a fully supervised Mask-RCNN [8] segmenter in a removal task on the\nCOCO [9] dataset.\nAn important use-case of our system would be in automatic content \ufb01ltering, e.g.\nfor privacy\nor parental control. This would involve automatic removal of objects and sensitive content from\nlarge databases or continuous streams of images. Content to be removed in these scenarios are\noften personalized and beyond the usually studied object categories in computer vision. Thus a\nsystem which can learn to remove these objects from cheap image-level labels would be useful. We\ndemonstrate the applicability of our object remover model to such content \ufb01ltering task, by training it\nto automatically remove brand logos from images with only image level labels.\n\n2 Related work\n\nGenerative adversarial networks. Generative adversarial networks (GAN) [10] are a framework\nwhere a generator learns by competing in an adversarial game against a discriminator network. The\ndiscriminator learns to distinguish between the real data samples and the \u201cfake\u201d generated samples.\nThe generator is optimized to fool the discriminator into classifying generated samples as real. The\ngenerator can be conditioned on additional information to learn conditional generative models [11].\nImage manipulation with unpaired data. A conditional GAN based image-to-image translation\nsystem was developed in [12] to manipulate images using paired supervision data. Li et al. [6]\nalleviated the need for paired supervision using cycle constraints and demonstrated translation\nbetween two different domains of unpaired images including (horse\u2194zebras) and (summer\u2194winter).\nSimilar cyclic reconstruction constraints were extended to multiple domains to achieve facial attributes\nmanipulation without paired data [5]. Nevertheless these image manipulation works have been limited\nto object centric images like faces [5] or constrained images like street scenes from one point of\nview [6]. In our work we take a step towards general scene-level manipulation by addressing the\nproblem of object removal from generic scenes. Prior works on scene-level images like the COCO\ndataset have focused on synthesizing entire images conditioned on text [13\u201315] and scene-graphs [16].\nHowever generated image quality on scene-level images [16] is still signi\ufb01cantly worse than on\nstructured data like faces [17]. In contrast we focus on the manipulation of parts of images rather\nthan full image synthesis and achieve better image quality and control.\nObject removal. We propose a two-staged editor with a mask-generator and image in-painter which\njointly learn to remove the target object class. Prior works on object removal focus on algorithmic\nimprovements to in-painting while assuming users provide the object mask [18\u201320]. One could argue\nthat object segmentation masks can be obtained by a stand alone segmenter like Mask-RCNN [8]\nand just in-paint this masked region to achieve removal. However, this needs expensive mask\nannotation to supervise the segmentation networks for every category of image entity one wishes to\nremove for example objects or brand logos.Additionally, as we show in our experiments, even perfect\nsegmentation masks are not suf\ufb01cient for perfect removal. They tend to trace the object shapes too\nclosely and leave object silhouettes giving away the object class. In contrast, our model learns to\nperform removal by jointly optimizing the mask generator and the in-painter for the removal task with\nonly weak supervision from image-level labels. This joint optimization allows the two components to\ncooperate to achieve removal performance on par with a fully supervised segmenter based removal.\n\n3 Learning to remove objects\n\nWe propose an end-to-end model which learns to \ufb01nd and remove objects automatically from images\nwithout any human interaction. It learns to perform this removal with only access to image-level\nlabels without needing expensive ground-truth location information like bounding boxes or masks.\n\n2\n\n\f(a) Our editor is composed of a mask-generator and an image in-painter\n\ninput\n\nclassic\n\ngan\n\nours\n\n(b) Two-stage generator avoids ad-\nversarial patterns\n\nFigure 1: Illustrating (a) the proposed two-staged architecture and (b) the motivation for this approach\n\nAdditionally, we do not have ground-truth target images showing the expected output image with the\ntarget object removed since it is infeasible to obtain such data in general.\nWe overcome the lack of ground-truth location and target image annotations by designing a generative\nadversarial framework (GAN) to train our model with only unpaired data. Here our editor model\nlearns from weak supervision from three different classi\ufb01ers. The model learns to locate and remove\nobjects by trying to fool an object classi\ufb01er. It learns to produce realistic output by trying to fool an\nadversarial real/fake classi\ufb01er. Finally, it learns to produce realistic looking object masks by trying to\nfool a mask shape classi\ufb01er. Let us examine these components in detail.\n\n3.1 Editor architecture: A two-staged approach\n\nRecent works [4, 5] on image manipulation utilize a generator network which takes the input image\nand synthesizes the output image to re\ufb02ect the target attributes. While this approach works well\nfor structured images of single faces, we found in own experiments that it does not scale well for\nremoving objects from general scene images. In general scenes with multiple objects, it is dif\ufb01cult\nfor the generator to remove only the desired object while re-synthesizing the rest of the image\nexactly. Instead, the generator \ufb01nds the easier solution to fool the object classi\ufb01er by producing\nadversarial patterns. This is also facilitated by the fact that the object classi\ufb01er in crowded scenes\nhas a much harder task than a classi\ufb01er determining hair-colors in object centric images and thus is\nmore susceptible to adversarial patterns. Figure 1b illustrates this observation, where a single stage\ngenerator from [5] trying to remove the person, fools the classi\ufb01er using adversarial noise. We can\nalso see that the colors of the entire image have changed even when removing a single local object.\nWe propose a two-staged generator architecture shown in Figure 1a to address this issue. The \ufb01rst\nstage is a mask generator, GM , which learns to locate the target object class, ct, in the input image x\nand masks it out by generating a binary mask m = GM (x, ct). The second stage is the in-painter, GI,\nwhich takes the generated mask and the masked-out image as input and learns to in-paint to produce\n\na realistic output. Given the inverted mask (cid:101)m = 1 \u2212 m, \ufb01nal output image y is computed as\n\ny = (cid:101)m \u00b7 x + m \u00b7 GI ((cid:101)m \u00b7 x)\n\n(1)\nThe mask generator is trained to fool the object classi\ufb01er for the target class whereas the in-painter is\ntrained to only fool the real/fake classi\ufb01er by minimizing the loss functions shown below.\n\nLcls(GM ) = \u2212 Ex [log(1 \u2212 Dcls(y, ct))]\nLrf(GI ) = \u2212 Ex [Drf(y)]\n\n(2)\n(3)\n\nwhere Dcls(y, ct) is the object classi\ufb01er score for class ct and Drf is the real/fake classi\ufb01er.\nHere Drf is adversarial, i.e. it is constantly updated to classify generated samples y as \u201cfake\u201d. The\nobject classi\ufb01er Dcls however is not adversarial, since it leads to the classi\ufb01er using the context to\npredict the object class even when the whole object is removed. Instead, to make the Dcls robust to\npartially removed objects, we train it on images randomly masked with rectangles. The multiplicative\ncon\ufb01guration in (1) makes it easy for GM to remove the objects by masking them out. Additionally,\nthe in-painter also does not produce adversarial patterns as it is not optimized to fool the object\nclassi\ufb01er but only to make the output image realistic. The ef\ufb01cacy of this approach is illustrated in\nthe image on the right on Figure 1b, where our two-staged model is able to cleanly remove the person\nwithout affecting the rest of the image.\n\n3\n\nImage\u00a0\u00a0In\u00adpainterMask\u00a0GeneratorObjectClassifierReal/Fake\u00a0?Is\u00a0there\u00a0\u00a0a\u00a0person\u00a0?EditorReal/\u00a0FakeClassifier\f3.2 Mask priors\n\nWhile the two-stage architecture avoids ad-\nversarial patterns and converge to desirable\nsolutions, it is not suf\ufb01cient. The mask gen-\nerator can still produce noisy masks or con-\nverge to bad solutions like masking most\nof the image to fool the object classi\ufb01er.\nA simple solution is to favor small sized\nmasks. We do this by simply minimizing\nthe exponential function of the mask size,\nexp(\u03a3ijmij). But this only penalizes large\nmasks but not noisy or incoherent masks.\nTo avoid these degenerate solutions, we\npropose a novel mechanism to regularize\nthe mask generator to produce masks close\nto a prior distribution. We do this by minimizing the Wasserstein distance between the generated\nmask distribution and the prior distribution P (m) using Wasserstein GAN (WGAN) [21] as shown\nin Figure 2. The WGAN framework allows \ufb02exibility while choosing the prior since we only need\nsamples from the prior and not a parametric form for the prior.\nThe prior can be chosen with varying complexity depending on the amount of information available,\nincluding knowledge about shapes of different object classes. For example we can use unpaired\nsegmentation masks from a different dataset as a shape prior to the generator. When this is not\navailable, we can impose the prior that objects are usually continuous coherent shapes by using\nsimple geometric shapes like randomly generated rectangles as the prior distribution.\nGiven a class speci\ufb01c prior mask distribution, P (mp|ct), we setup a discriminator, DM to assign\nhigh scores to samples from this prior distribution and the masks generated by GM (x, ct). The\nmask generator is then additionally optimized to fool the discriminator DM . The adversarial losses\nminimized by DM and GM are as below:\n\nFigure 2: Imposing mask priors with a GAN framework\n\nL(DM ) = Ex [DM (GM (x, ct), ct)] \u2212 Emp\u223cP (mp|ct) [DM (mp, ct)]\n\nLprior(GM ) = \u2212 Ex [DM (GM (x, ct), ct)]\n\n(4)\n(5)\n\n3.3 Optimizing the in-painting network for removal\n\nThe in-painter network GI is tasked with synthesizing a plausible image patch to \ufb01ll the region\nmasked-out by GM , to produce a realistic output image. Similar to prior works on in-painting [22\u2013\n24], we train GI with self-supervision by trying to reconstruct random image patches and weak\nsupervision from fooling an adversarial real/fake classi\ufb01er. The reconstruction loss encourages GI to\nkeep consistency with the image while the adversarial loss encourages it to produce sharper images.\nReconstruction losses. To obtain self-supervision to the in-painter we mask random rectangular\npatches mr from the input and ask GI to reconstruct these patches. We minimize the L1 loss and the\nperceptual loss [25] between the in-painted image and the input as follows:\n\nLrecon(GI ) = (cid:107)GI ((cid:101)mr \u00b7 x) \u2212 x(cid:107)1 + \u03a3k (cid:107)\u03c6k (GI ((cid:101)mr \u00b7 x)) \u2212 \u03c6k(x)(cid:107)1\n\n(6)\nMask buffer. The masks generated by GM (x, ct) can be of arbitrary shape and hence the in-painter\nshould be able to \ufb01ll in arbitrary holes in the image. We \ufb01nd that the in-painter trained only on\nrandom rectangular masks performs poorly on masks generated by GM . However, we cannot simply\ntrain the in-painter with reconstruction loss in (6) on masks generated by GM . Unlike random masks\nmr which are unlikely to align exactly with an object, generated masks GM (x, ct) overlap the objects\nwe intend to remove. Using reconstruction loss here would encourage the in-painter to regenerate\nthis object. We overcome this by storing generated masks from previous batches in a mask buffer and\nrandomly applying them on images from the current batch. These are not objects aligned anymore\ndue to random pairing and we train the in-painter GI with the reconstruction loss, allowing it to adapt\nto the changing mask distribution produced by the GM (x, ct).\nLocal real/fake loss. In recent works on in-painting using adversarial loss [22\u201324], in-painter is\ntrained adversarially against a classi\ufb01er Drf which learns to predict global \u201creal\u201d and \u201cfake\u201d labels for\ninput x and the generated images y respectively. A drawback with this formulation is that only a small\n\n4\n\nMask\u00a0GeneratorMaskDiscrim\u00ad\u00a0inatorto\u00a0In\u00adPainterClassLabelPrior\u00a0/GeneratedSamples\u00a0frommask\u00a0prior\u00a0\fpercentage of pixels in the output y is comprised of truly \u201cfake\u201d pixels generated by the in-painter, as\nseen in Equation (1). This is a hard task for the classi\ufb01er Drf hard since it has to \ufb01nd the few pixels\nthat contribute to the global \u201cfake\u201d label. We tackle this by providing local pixel-level real/fake labels\non the image to Drf instead of a global one. The pixel-level labels are available for free since the\n\ninverted mask (cid:101)m acts as the ground-truth \u201creal\u201d label for Drf. Note that this is different from the\n\npatch GAN [12] where the classi\ufb01er producing patch level real/fake predictions is still supervised\nwith a global image-level real/fake label. We use the least-square GAN loss [26] to train the Drf,\nsince we found the WGAN loss to be unstable with local real/fake prediction. This is because, Drf\ncan minimize the WGAN loss with assigning very high/low scores to one patch, without bothering\nwith the other parts of the image. However, least-squares GAN loss penalizes both very high and\nvery low predictions, thereby giving equal importance to different image regions.\n\nL(Drf) =\n\nmij \u00b7 (Drf(y)ij + 1)2\n\n(7)\n\n(cid:88)\n\nij\n\n1\n\n\u03a3ij(cid:101)mij\n\n(cid:101)mij \u00b7 (Drf(y)ij \u2212 1)2 +\n\n(cid:88)\n\n1\n\n\u03a3ijmij\n\nij\n\nPenalizing variations. We also incorporate the style-loss (Lsty) proposed in [24] to better match the\ntextures in the in-painting output with that of the input image and the total variation loss (Ltv) since it\nhelps produce smoother boundaries between the in-painted region and the original image.\nThe mask generator and the in-painter are optimized in alternate epochs using gradient descent. When\nthe GM is being optimized, parameters of GI are held \ufb01xed and vice-versa when GI is optimized.\nWe found that optimizing both the models at every step led to unstable training and many training\ninstances converged to degenerate solutions. Alternate optimization avoids this while still allowing\nthe mask generator and in-painter to co-adapt. The \ufb01nal loss function for GM and GI is given as:\n\nLtotal(GM ) = \u03bbcLcls + \u03bbpLprior + \u03bbsz exp(\u03a3ijmij)\nLtotal(GI ) = \u03bbrf Lrf + \u03bbrLrecon + \u03bbtvLtv + \u03bbstyLsty\n\n(8)\n(9)\n\n4 Experimental setup\nDatasets. Keeping with the goal of performing removal on general scene images, we train and test\nour model mainly on the COCO dataset [9] since it contains signi\ufb01cant diversity within object classes\nand in the contexts in which they appear. We test our proposed GAN framework to impose priors\non the mask generator with two different priors namely rotated boxes and unpaired segmentation\nmasks. We use the segmentation masks from Pascal-VOC 2012 dataset [27] (without the images) as\nthe unpaired mask priors. To facilitate this we restrict our experiments on 20 classes shared between\nthe COCO and Pascal datasets. To demonstrate that our editor model can generalize beyond objects\nand can learn to remove to different image entities, we test our model on the task of removing logos\nfrom natural images. We use the Flickr Logos dataset [28], which has a training set of 810 images\ncontaining 27 annotated logo classes and a test set of 270 images containing 5 images per class\nand 135 random images containing no logos. Further details about data pre-processing and network\narchitectures is presented in the supplementary material.\nEvaluation metrics. We evaluate our object removal for three aspects: removal performance to\nmeasure how effective is our model at removing target objects and image quality assessment to\nquantify how much of the original image is edited and \ufb01nally human evaluation to judge removal.\n\u2022 Removal performance: We quantify the removal performance by measuring the performance\nof an object classi\ufb01er on the edited images using two metrics. Removal success rate measures the\npercentage of instances where the editor successfully fools the object classi\ufb01er score below the\ndecision boundary for the target object class.False removal rate measures the percentage of cases\nwhere the editor removes the wrong objects while trying to remove the target class. This is again\nmeasured by monitoring if the object classi\ufb01er score drops below decision boundary for other classes.\n\u2022 Image quality assessment: To be useful, our editor should remove the target object class while\nleaving the rest of the image intact.Thus, we quantify the usefulness by measuring similarity between\nthe output and the input image using three metrics namely peak signal-to-noise ratio (pSNR), structural\nsimilarity index (ssim) [29] and perceptual loss [30]. The \ufb01rst two are standard metrics used in image\nin-painting literature, whereas the perceptual loss [30] was recently proposed as a learned metric to\ncompare two images. We use the squeezenet variant of this metric.\n\u2022 Human evaluation: We conduct a study to obtain human judgments of removal performance.\nWe show hundred randomly selected edited images to a human judge and asked if they see the\n\n5\n\n\fInput\nimage\n\nM-\nRCCN\nbased\n\nOurs\n\nperson\n\ndog\nFigure 3: Qualitative examples of removal of different object classes\n\nairplane\n\ntv\n\nperson\n\ncow\n\nperson\n\ntarget object class. To keep the number of annotations reasonable, we conduct the human evaluation\nonly on the person class (largest class). Each image is shown to three separate judges and removal\nis considered successful when all three humans agree that they do not see the object class. The\nparticipants in the study were not aware of the project and were just asked to determine if they see a\n\u2019person\u2019 (either full body or clear body parts/ silhouettes) in the images shown. The outputs from\ndifferent models were all shown in the same session to a human judge in a randomized order to\nprevent biasing the results against latter models. This human study evaluates the removal system\nholistically and helps verify that the removal performance measured by a classi\ufb01er is similar to as\nperceived by the humans, and thus validating the automatic evaluation protocol.\nBaselines with additional supervision. Since there is no prior work proposing a fully automatic\nobject removal solution, we compare our model against removal using a stand-alone fully supervised\nsegmentation model, Mask-RCNN [8]. We obtain segmentation mask predictions from Mask-RCNN\nand use our trained in-painter to achieve removal. Additionally we also compare our model to a\nweakly supervised segmentation method from [31] (referred to as SDI), which learns to segment\nobjects by using ground truth bounding boxes as supervision. Please note that both the above methods\nuse stronger supervision in terms of object bounding boxes (Mask-RCNN and SDI) and object\nsegmentation (Mask-RCNN) than our proposed method, which uses only image level labels.\n\n5 Results\nWe present qualitative and quantitative evaluations of our editor and comparisons to the Mask-RCNN\nbased removal. Qualitative results show that our editor model works well across diverse scene types\nand object classes. Quantitative analysis shows that our weakly supervised model performs on par\nwith the fully supervised Mask-RCNN in the removal task, in both automatic and human evaluation.\n\n5.1 Qualitative results\n\nFigure 3 shows the results of object removal performed by our model (last row) on the COCO dataset\ncompared to the Mask-RCNN baseline. We see that our model works across diverse scene types, with\nsingle objects (columns 1-4) or multiple instances of the same object class (col. 5-6) and even for a\nfairly large object (last column). Figure 3 also highlight the problems with simply using masks from\na segmentation model, Mask-RCNN, for removal. Mask-RCNN is trained to accurately segment\nthe objects and thus the masks it produces very closely trace the object boundary, too closely for\nremoval purposes. We can clearly see the silhouettes of objects in all the edited images on the second\nrow. These results justify our claim that segmentation annotations are not needed to learn to remove\nobjects and might not be the right annotations anyway.\nOur model is not tied to notion of objectness and can be easily extended to remove other image\nentities. The \ufb02exible GAN based mask priors allow us to use random rectangular boxes as priors\nwhen object shapes are not available. To demonstrate this we apply our model to the task of removing\nbrand logos automatically from images. The model is trained using image level labels and box prior.\nQualitative examples in Figure 4 shows that our model works well for this task, despite the fairly\n\n6\n\n\fInput\nimage\n\nOurs\n\nInput\nimage\n\nOurs\n\nmini\n\npepsi heineken\n\nnbc\n\nbmw\nFigure 4: Results of logo removal\n\ncitroen\n\nInput\nimage\n\nNo\nprior\n\nbox\nprior\n\npascal\nmask\nprior\n\nFigure 5: Effect of priors on generated masks\n\nsmall training set (800 images). It is able to \ufb01nd and remove logos in different contexts with only\nimage level labels. The image on the bottom left shows a failure case where the model fails to realize\nthat the text \u201cNBC\u201d belongs to the logo.\nFigure 5 shows the masks generated by our model with different mask priors on the COCO dataset.\nThese examples illustrate the importance of the proposed mask priors. The masks generated by the\nmodel using no prior (second row) are very noisy since the model has no information about object\nshapes and is trying to infer everything from the image level classi\ufb01er. Adding the box prior already\nmakes the masks much cleaner and more accurate. We can note that the generated masks are \u201cboxier\u201d\nwhile not strictly rectangles. Finally using unpaired segmentation masks from the pascal dataset as\nshape priors makes the generated masks more accurate and the model is able to recover the object\nshapes better. This particularly helps in object with diverse shapes, for example people and dogs.\n\n5.2 Quantitative evaluation of removal performance\n\nTo quantify the removal performance we run an object classi\ufb01er on the edited images and measure its\nperformance. We use a separately trained classi\ufb01er for this purpose, not the one used in our GAN\ntraining, to fairly compare our model and the Mask-RCNN based removal.\nSanity of object classi\ufb01er performance. The classi\ufb01er we use to evaluate our model achieves\nper-class average F1-score of 0.57, overall average F1-score of 0.67 and mAP of 0.58. This is close to\nthe results achieved by recent published work on multi-label classi\ufb01cation [32] on the COCO dataset,\nwhich achieves class average F1-score of 0.60, overall F1-score of 0.68 and mAP of 0.61. While\nthese numbers are not directly comparable (different image resolution, different number of classes),\nit shows that our object classi\ufb01er has good performance and can be relied upon. Furthermore, human\nevaluation shows similar results as our automatic evaluation.\nEffect of priors. Table 1 compares the different versions of our model using different priors. The\nbox prior uses randomly generated rectangles of different aspect ratios, area and rotations. The Pascal\n(n) prior uses n randomly chosen unpaired segmentation masks for each class from the Pascal dataset.\nThe table shows metrics measuring the removal performance, image quality and mask accuracy.\nThe arrows \u2191 and \u2193 indicate if higher or lower is better for the corresponding metric. Comparing\nremoval performance in Table 1 we see that while the model with no prior achieves very high removal\nrate (94%), but it does so with large masks (37 %) which causes low output image quality. As we\nadd priors, the generated masks become smaller and compact. We also see that mIou of the masks\nincrease with stronger priors (0.22-0.23 for pascal prior), indicating they are more accurate. Smaller\nand more accurate masks also improve the image quality metrics and false removal rates which drop\nmore than half from 36% to 16%. This is inline with the visual examples in Figure 5, where model\nwithout prior produces very noisy masks and quality of the masks improve with priors.\nAnother interesting observation from Table 1 is that using very few segmentation masks from pascal\ndataset leads to a drop in removal success rate, especially for the person class. This is because the\n\n7\n\n\fTable 1: Quantifying the effect of using more accurate mask priors\n\nRemoval Performance\n\nPrior\n\nNone\nboxes\npascal (10)\npascal (100)\npascal (all)\n\nperson\n\nremoval success \u2191\nall\n94\n83\n67\n70\n73\n\n96\n88\n59\n75\n81\n\nfalse \u2193\nremoval\n\n36\n23\n17\n16\n16\n\nImage quality metrics\n\nMask accuracy\n\npercep.\nloss \u2193\n\npSNR \u2191\n\nssim \u2191 mIou \u2191 % masked\narea \u2193\n\n0.13\n0.11\n0.07\n0.07\n0.08\n\n19.97\n20.41\n23.81\n23.02\n22.64\n\n0.743\n0.777\n0.833\n0.821\n0.803\n\n0.15\n0.18\n0.23\n0.22\n0.22\n\n37.7\n28.1\n16.7\n18.1\n20.2\n\nTable 3: Comparison to ground truth masks and Mask-RCNN baselines.\n\nModel\n\nSupervision\n\nGT masks\nMask RCNN\nMask RCNN (dil. 7x7)\n\nours-pascal\n\n-\nSeg. masks &\nbound boxes\nimage labels &\nunpaired masks\n\nRemoval Performance\n\nImage quality metrics\n\nperson\n\nremoval success \u2191\nall\n66\n68\n75\n\n72\n73\n77\n\n73\n\n81\n\nfalse \u2193\nremoval\n\npercep.\nloss \u2193\n\npSNR \u2191\n\nssim \u2191\n\n5\n6\n10\n\n16\n\n0.04\n0.05\n0.07\n\n0.08\n\n27.43\n25.59\n24.13\n\n22.64\n\n0.930\n0.900\n0.882\n\n0.803\n\nperson class has very diverse shapes due to varying poses and scales. Using only ten masks in the prior\nfails to capture this diversity and performs poorly (59%). As we increase the number of mask samples\nin the prior, removal performance jumps signi\ufb01cantly to 81% on the person class. Considering these\nresults, we note that the pascal all version offers the best trade-off between removal and image quality\ndue to more accurate masks and we will use this model in comparison to benchmarks.\nBenchmarking against GT and Mask-RCNN. Table 3 compares the performance of our model\nagainst baselines using ground-truth (GT) masks and Mask-RCNN segmentation masks for removal.\nThese benchmarks use the same in-painter as our-pascal model. We see that our model outperforms\nthe fully supervised Mask-RCNN masks and even the GT masks in terms of removal (66%& 68%\nvs 73%). While surprising, this is explained by the same phenomenon we saw in qualitative results\nwith Mask-RCNN in Figure 3. The GT and Mask-RCNN masks for segmentation are too close to the\nobject boundaries and thus leave object silhouettes behind when used for removal. When we dilate the\nmasks produced by Mask-RCNN before using for removal, the performance improves overall and is\non par with our model (slightly better in all classes and a bit worse in the person class). The drawback\nof weak supervision is that masks are a bit larger which leads to bit higher false removal rate (16%\nours compared to 10% Mask-RCNN dilated) and lower image quality metrics. However this is still\na signi\ufb01cant result, given that our model is trained without expensive ground truth segmentation\nannotation for each image, but instead uses only unpaired masks from a smaller dataset.\nComparison to weakly supervised segmentation. We compare to the weakly supervised SDI [31]\nmodel in Table 2. We use the the output masks generated by SDI to mask the image and use the\nin-painter trained with our model to \ufb01ll in the masked region. Simply using the masks from SDI\nwithout dilation results in poor removal performance with only 54% success overall and 45% success\non the \u2018person\u2019 class. Upon dilation, the performance improves, but is still signi\ufb01cantly worse than\nour model and Mask-RCNN.\nAdditionally, SDI method starts from boxes generated\nby a fully supervised RCNN network and generates\nsegmentation with weak supervision, whereas our\nmodel uses only image-level labels and hence is more\ngenerally applicable.\nHuman evaluation. We verify our automatic evalu-\nation results using a user study to evaluate removal\nsuccess as described in Section 4. The human judge-\n\nTable 2: Comparison to weakly supervised\nsemantic segmentation model, SDI [31]\n\nRem. Succ.\nperson\nall\n45\n54\n65\n64\n81\n73\n\nSDI: supervised\nwith GT boxes\nOurs\n\n-\n7x7\n-\n\nModel\n\ndil\n\n8\n\n\fTable 4: Evaluating in-painting components\npercep.\nGAN TV+\nloss \u2193 pSNR \u2191 ssim \u2191\nStyle\n0.730\n0.13\n-\n0.772\n0.12\n-\n0.10\n0.758\n-\n(cid:88) 0.10\n0.763\n\n20.0\n21.9\n21.5\n21.6\n\nG\nG\nL\nL\n\nMask\nbuffer\n\n-\n(cid:88)\n(cid:88)\n(cid:88)\n\nTable 5: Joint training helps improve both mask\ngeneration and in-painting\n\nInput image\n\nGlobal loss\n\nLocal loss\n\nFigure 6: Comparing global and local GAN\nloss. Global loss smooth blurry results, while\nlocal one produce sharp, texture-rich images.\n\nJoint\ntraining\n-\n(cid:88)\n\nRemoval\nsuccess \u2191\n0.68\n0.73\n\nmIou \u2191 percep.\nloss \u2193\n0.19\n0.10\n0.08\n0.22\n\nments of removal performance follow the same trend seen in automatic evaluation, except that\nhuman judges penalize the silhouettes more severely.Our model clearly outperforms the baseline\nMask-RCNN model without dilation by achieving 68% removal rate compared to only 30% achieved\nby Mask-RCNN. With dilated masks, Mask-RCNN performs similar to our model in terms of removal\nachieving 73% success rate.\n\n5.3 Ablation studies\n\nJoint optimization. We conduct an experiment to test if jointly training the mask generator and the\nin-painter helps. We pre-train the in-painter using only random boxes and hold it \ufb01xed while training\nthe mask generator. The results are shown in Table 5. Not surprisingly, the in-painting quality suffers\nwith higher perceptual loss (0.10 vs 0.08) since it has not adapted to the masks being generated. More\ninterestingly, the mask generator also degrades with a \ufb01xed in-painter, as seen by lower mIou (0.19\nvs 0.22) and lower removal success rate (0.68 vs 0.73). This result shows that it is important to train\nboth the models jointly to allow them to adapt to each other for best performance.\nIn-painting components. Table 4 shows the ablation of the in-painter network components. We\nnote that the proposed mask-buffer, which uses masks from previous batch to train the in-painter with\nreconstruction loss, signi\ufb01cantly improves the results signi\ufb01cantly in all three metrics. Using local\nloss improves the results in-terms of perceptual loss (0.10 vs 0.12) while being slightly worse in the\nother two metrics. However on examining the results visually in Figure 6, we see that the version with\nthe global GAN loss produces smooth and blurry in-painting, whereas the version with local GAN\nloss produces sharper results with richer texture. While these blurry results do better in pixel-wise\nmetrics like pSNR and ssim, they are easily seen by the human eye and are not suitable for removal.\nFinally addition of total variation and style loss helps slighlty improve the pSNR and ssim metrics.\n\n6 Conclusions\n\nWe presented an automatic object removal model which learns to \ufb01nd and remove objects from\ngeneral scene images. Our model learns to perform this task with only image level labels and unpaired\ndata. Our two-stage editor model with a mask-generator and an in-painter network avoids degenerate\nsolutions by complementing each other. We also developed a GAN based framework to impose\ndifferent priors to the mask generator, which encourages it to generate clean compact masks to\nremove objects. Results show that our model achieves similar performance as a fully-supervised\nsegmenter based removal, demonstrating the feasibility of weakly supervised solutions for the general\nscene-level editing task.\n\nAcknowledgments\n\nThis research was supported in part by the German Research Foundation (DFG CRC 1223).\n\n9\n\n\fReferences\n[1] A. Shrivastava, T. P\ufb01ster, O. Tuzel, J. Susskind, W. Wang, and R. Webb, \u201cLearning from simulated and\nunsupervised images through adversarial training,\u201d in Proceedings of the IEEE Conference on Computer\nVision and Pattern Recognition (CVPR), no. 4, 2017.\n\n[2] T. Orekondy, M. Fritz, and B. Schiele, \u201cConnecting pixels to privacy and utility: Automatic redaction of\n\nprivate information in images,\u201d arXiv preprint arXiv:1712.01066, 2017.\n\n[3] R. Huang, S. Zhang, T. Li, and R. He, \u201cBeyond face rotation: Global and local perception gan for\nphotorealistic and identity preserving frontal view synthesis,\u201d in Proceedings of the IEEE Conference on\nComputer Vision and Pattern Recognition (CVPR), 2017.\n\n[4] G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. Denoyer et al., \u201cFader networks: Manipulating images\n\nby sliding attributes,\u201d in Advances in Neural Information Processing Systems (NIPS), 2017.\n\n[5] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, \u201cStargan: Uni\ufb01ed generative adversarial\n\nnetworks for multi-domain image-to-image translation,\u201d arXiv preprint arXiv:1711.09020, 2017.\n\n[6] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, \u201cUnpaired image-to-image translation using cycle-consistent\nadversarial networks,\u201d Proceedings of the IEEE International Conference on Computer Vision (ICCV),\n2017.\n\n[7] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, \u201cHigh-resolution image synthesis\n\nand semantic manipulation with conditional gans,\u201d arXiv preprint arXiv:1711.11585, 2017.\n\n[8] K. He, G. Gkioxari, P. Doll\u00e1r, and R. Girshick, \u201cMask R-CNN,\u201d in Proceedings of the IEEE International\n\nConference on Computer Vision (ICCV), 2017.\n\n[9] X. Chen, T.-Y. L. Hao Fang, R. Vedantam, S. Gupta, P. Doll\u00e1r, and C. L. Zitnick, \u201cMicrosoft COCO\n\ncaptions: Data collection and evaluation server,\u201d arXiv preprint arxiv:1504.00325, 2015.\n\n[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio,\n\n\u201cGenerative adversarial nets,\u201d in Advances in Neural Information Processing Systems (NIPS), 2014.\n\n[11] M. Mirza and S. Osindero, \u201cConditional generative adversarial nets,\u201d arXiv preprint arXiv:1411.1784,\n\n2014.\n\n[12] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, \u201cImage-to-image translation with conditional adversarial\nnetworks,\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2016.\n\n[13] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, \u201cGenerative adversarial text to image\n\nsynthesis,\u201d in 33rd International Conference on Machine Learning, 2016.\n\n[14] X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie, \u201cStacked generative adversarial networks,\u201d in\n\nIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[15] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, \u201cAttngan: Fine-grained text to image\n\ngeneration with attentional generative adversarial networks,\u201d arXiv preprint arXiv:1711.10485, 2017.\n\n[16] J. Johnson, A. Gupta, and L. Fei-Fei, \u201cImage generation from scene graphs,\u201d arXiv preprint\n\narXiv:1804.01622, 2018.\n\n[17] T. Karras, T. Aila, S. Laine, and J. Lehtinen, \u201cProgressive growing of gans for improved quality, stability,\nand variation,\u201d Proceedings of the International Conference on Learning Representations (ICLR), 2018.\n\n[18] A. Criminisi, P. P\u00e9rez, and K. Toyama, \u201cRegion \ufb01lling and object removal by exemplar-based image\n\ninpainting,\u201d IEEE Transactions on image processing, 2004.\n\n[19] J. Hays and A. A. Efros, \u201cScene completion using millions of photographs,\u201d in ACM Transactions on\n\nGraphics (TOG), 2007.\n\n[20] S. S. Mirkamali and P. Nagabhushan, \u201cObject removal by depth-wise image inpainting,\u201d Signal, Image and\n\nVideo Processing, 2015.\n\n[21] M. Arjovsky, S. Chintala, and L. Bottou, \u201cWasserstein generative adversarial networks,\u201d in Proceedings of\n\nthe International Conference on Machine Learning (ICML), 2017.\n\n10\n\n\f[22] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, \u201cGenerative image inpainting with contextual\n\nattention,\u201d arXiv preprint arXiv:1801.07892, 2018.\n\n[23] S. Iizuka, E. Simo-Serra, and H. Ishikawa, \u201cGlobally and locally consistent image completion,\u201d ACM\n\nTransactions on Graphics (TOG), 2017.\n\n[24] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, \u201cImage inpainting for irregular holes\n\nusing partial convolutions,\u201d arXiv preprint arXiv:1804.07723, 2018.\n\n[25] L. Gatys, A. Ecker, and M. Bethge, \u201cA neural algorithm of artistic style,\u201d Nature Communications, 2015.\n\n[26] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley, \u201cLeast squares generative adversarial\n\nnetworks,\u201d in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.\n\n[27] M. Everingham, L. Van Gool, C. K.\n\nI. Williams,\n\nPASCAL Visual Object Classes Challenge 2012 (VOC2012) Results,\u201d\nnetwork.org/challenges/VOC/voc2012/workshop/index.html.\n\nJ. Winn,\n\nand A. Zisserman,\n\n\u201cThe\nhttp://www.pascal-\n\n[28] Y. Kalantidis, L. Pueyo, M. Trevisiol, R. van Zwol, and Y. Avrithis, \u201cScalable triangulation-based logo\n\nrecognition,\u201d in Proceedings of ACM International Conference on Multimedia Retrieval (ICMR), 2011.\n\n[29] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, \u201cImage quality assessment: from error visibility\n\nto structural similarity,\u201d IEEE transactions on image processing, 2004.\n\n[30] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, \u201cThe unreasonable effectiveness of deep\n\nfeatures as a perceptual metric,\u201d arXiv preprint arXiv:1801.03924, 2018.\n\n[31] A. Khoreva, R. Benenson, J. H. Hosang, M. Hein, and B. Schiele, \u201cSimple does it: Weakly supervised\n\ninstance and semantic segmentation.\u201d in CVPR, 2017.\n\n[32] J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu, \u201cCnn-rnn: A uni\ufb01ed framework for multi-label\nimage classi\ufb01cation,\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), 2016.\n\n11\n\n\f", "award": [], "sourceid": 3816, "authors": [{"given_name": "Rakshith", "family_name": "Shetty", "institution": "Max Planck Institute for Informatics"}, {"given_name": "Mario", "family_name": "Fritz", "institution": "CISPA Helmholtz Center i.G."}, {"given_name": "Bernt", "family_name": "Schiele", "institution": "Max Planck Institute for Informatics"}]}