{"title": "Video-to-Video Synthesis", "book": "Advances in Neural Information Processing Systems", "page_first": 1144, "page_last": 1156, "abstract": "We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image translation problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without modeling temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generators and discriminators, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines.  In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our method to future video prediction, outperforming several competing systems. Code, models, and more results are available at our website: https://github.com/NVIDIA/vid2vid. (Please use Adobe Reader to see the embedded videos in the paper.)", "full_text": "Video-to-Video Synthesis\n\nTing-Chun Wang1, Ming-Yu Liu1, Jun-Yan Zhu2, Guilin Liu1,\n\nAndrew Tao1, Jan Kautz1, Bryan Catanzaro1\n\n{tingchunw,mingyul,guilinl,atao,jkautz,bcatanzaro}@nvidia.com,\n\n1NVIDIA, 2MIT CSAIL\n\njunyanz@mit.edu\n\nAbstract\n\nWe study the problem of video-to-video synthesis, whose goal is to learn a mapping\nfunction from an input source video (e.g., a sequence of semantic segmentation\nmasks) to an output photorealistic video that precisely depicts the content of the\nsource video. While its image counterpart, the image-to-image translation problem,\nis a popular topic, the video-to-video synthesis problem is less explored in the\nliterature. Without modeling temporal dynamics, directly applying existing image\nsynthesis approaches to an input video often results in temporally incoherent videos\nof low visual quality. In this paper, we propose a video-to-video synthesis approach\nunder the generative adversarial learning framework. Through carefully-designed\ngenerators and discriminators, coupled with a spatio-temporal adversarial objective,\nwe achieve high-resolution, photorealistic, temporally coherent video results on\na diverse set of input formats including segmentation masks, sketches, and poses.\nExperiments on multiple benchmarks show the advantage of our method compared\nto strong baselines. In particular, our model is capable of synthesizing 2K resolution\nvideos of street scenes up to 30 seconds long, which signi\ufb01cantly advances the\nstate-of-the-art of video synthesis. Finally, we apply our method to future video\nprediction, outperforming several competing systems. Code, models, and more\nresults are available at our website.\n\n1\n\nIntroduction\n\nThe capability to model and recreate the dynamics of our visual world is essential to building\nintelligent agents. Apart from purely scienti\ufb01c interests, learning to synthesize continuous visual\nexperiences has a wide range of applications in computer vision, robotics, and computer graphics.\nFor example, in model-based reinforcement learning [2, 24], a video synthesis model \ufb01nds use in\napproximating visual dynamics of the world for training the agent with less amount of real experience\ndata. Using a learned video synthesis model, one can generate realistic videos without explicitly\nspecifying scene geometry, materials, lighting, and dynamics, which would be cumbersome but\nnecessary when using a standard graphics rendering engine [35].\nThe video synthesis problem exists in various forms, including future video prediction [15, 18, 42, 45,\n50, 64, 67, 70, 76] and unconditional video synthesis [59, 66, 68]. In this paper, we study a new form:\nvideo-to-video synthesis. At the core, we aim to learn a mapping function that can convert an input\nvideo to an output video. To the best of our knowledge, a general-purpose solution to video-to-video\nsynthesis has not yet been explored by prior work, although its image counterpart, the image-to-image\ntranslation problem, is a popular research topic [6, 31, 33, 43, 44, 63, 65, 72, 81, 82]. Our method is\ninspired by previous application-speci\ufb01c video synthesis methods [58, 60, 61, 74].\nWe cast the video-to-video synthesis problem as a distribution matching problem, where the goal is\nto train a model such that the conditional distribution of the synthesized videos given input videos\nresembles that of real videos. To this end, we learn a conditional generative adversarial model [20]\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fFigure 1: Generating a photorealistic video from an input segmentation map video on Cityscapes.\nTop left: input. Top right: pix2pixHD. Bottom left: COVST. Bottom right: vid2vid (ours). The\n\ufb01gure is best viewed with Acrobat Reader. Click the image to play the video clip.\n\ngiven paired input and output videos, With carefully-designed generators and discriminators, and a\nnew spatio-temporal learning objective, our method can learn to synthesize high-resolution, photore-\nalistic, temporally coherent videos. Moreover, we extend our method to multimodal video synthesis.\nConditioning on the same input, our model can produce videos with diverse appearances.\nWe conduct extensive experiments on several datasets on the task of converting a sequence of\nsegmentation masks to photorealistic videos. Both quantitative and qualitative results indicate that\nour synthesized footage looks more photorealistic than those from strong baselines. See Figure 1\nfor example. We further demonstrate that the proposed approach can generate photorealistic 2K\nresolution videos, up to 30 seconds long. Our method also grants users \ufb02exible high-level control\nover the video generation results. For example, a user can easily replace all the buildings with trees in\na street view video. In addition, our method works for other input video formats such as face sketches\nand body poses, enabling many applications from face swapping to human motion transfer. Finally,\nwe extend our approach to future prediction and show that our method can outperform existing\nsystems. Please visit our website for code, models, and more results.\n\n2 Related Work\n\nGenerative Adversarial Networks (GANs). We build our model on GANs [20]. During GAN\ntraining, a generator and a discriminator play a zero-sum game. The generator aims to produce\nrealistic synthetic data so that the discriminator cannot differentiate between real and the synthesized\ndata. In addition to noise distributions [14, 20, 55], various forms of data can be used as input to the\ngenerator, including images [33, 43, 81], categorical labels [52, 53], and textual descriptions [56, 79].\nSuch conditional models are called conditional GANs, and allow \ufb02exible control over the output of\nthe model. Our method belongs to the category of conditional video generation with GANs. However,\ninstead of predicting future videos conditioning on the current observed frames [41, 50, 68], our\nmethod synthesizes photorealistic videos conditioning on manipulable semantic representations, such\nas segmentation masks, sketches, and poses.\nImage-to-image translation algorithms transfer an input image from one domain to a corresponding\nimage in another domain. There exists a large body of work for this problem [6, 31, 33, 43, 44, 63, 65,\n72, 81, 82]. Our approach is their video counterpart. In addition to ensuring that each video frame\nlooks photorealistic, a video synthesis model also has to produce temporally coherent frames, which\nis a challenging task, especially for a long duration video.\nUnconditional video synthesis. Recent work [59, 66, 68] extends the GAN framework for un-\nconditional video synthesis, which learns a generator for converting a random vector to a video.\n\n2\n\n\fVGAN [68] uses a spatio-temporal convolutional network. TGAN [59] projects a latent code to a\nset of latent image codes and uses an image generator to convert those latent image codes to frames.\nMoCoGAN [66] disentangles the latent space to motion and content subspaces and uses a recurrent\nneural network to generate a sequence of motion codes. Due to the unconditional setting, these\nmethods often produce low-resolution and short-length videos.\nFuture video prediction. Conditioning on the observed frames, video prediction models are trained\nto predict future frames [15, 18, 36, 41, 42, 45, 50, 64, 67, 70, 71, 76]. Many of these models are trained\nwith image reconstruction losses, often producing blurry videos due to the classic regress-to-the-mean\nproblem. Also, they fail to generate long duration videos even with adversarial training [42, 50]. The\nvideo-to-video synthesis problem is substantially different because it does not attempt to predict\nobject motions or camera motions. Instead, our approach is conditional on an existing video and can\nproduce high-resolution and long-length videos in a different domain.\nVideo-to-video synthesis. While video super-resolution [61, 62], video matting and blending [3, 12],\nand video inpainting [73] can be considered as special cases of the video-to-video synthesis problem,\nexisting approaches rely on problem-speci\ufb01c constraints and designs. Hence, these methods cannot\nbe easily applied to other applications. Video style transfer [10, 22, 28, 58], transferring the style of a\nreference painting to a natural scene video, is also related. In Section 4 , we show that our method\noutperforms a strong baseline that combines a recent video style transfer with a state-of-the-art\nimage-to-image translation approach.\n\n3 Video-to-Video Synthesis\n\n1 to a sequence of output video frames, \u02dcxT\n\n1 \u2261 {s1, s2, ..., sT} be a sequence of source video frames. For example, it can be a sequence\nLet sT\n1 \u2261 {x1, x2, ..., xT} be the sequence of\nof semantic segmentation masks or edge maps. Let xT\ncorresponding real video frames. The goal of video-to-video synthesis is to learn a mapping function\n1 \u2261 {\u02dcx1, \u02dcx2, ..., \u02dcxT}, so that the\nthat can convert sT\n1 given sT\n1 .\nconditional distribution of \u02dcxT\n(1)\nThrough matching the conditional video distributions, the model learns to generate photorealistic,\ntemporally coherent output sequences as if they were captured by a video camera.\nWe propose a conditional GAN framework for this conditional video distribution matching task. Let\nG be a generator that maps an input source sequence to a corresponding output frame sequence:\nxT\n1 = G(sT\n\n1 is identical to the conditional distribution of xT\np(\u02dcxT\n\n1 ). We train the generator by solving the minimax optimization problem given by\n\n1 ) = p(xT\n\n1 given sT\n\n1 |sT\n1 ).\n\n1 |sT\n\nmax\n\nD\n\nmin\n\nG\n\nE(xT\n\n1 )[log D(xT\n\n1 , sT\n\n1 ,sT\n\n1 )] + EsT\n\n1\n\n[log(1 \u2212 D(G(sT\n\n1 ), sT\n\n1 ))],\n\n(2)\n\n1 |sT\n\n1 |sT\n\n1 ) and p(xT\n\n1 ) as shown by Goodfellow et al. [20].\n\nwhere D is the discriminator. We note that as solving (2), we minimize the Jensen-Shannon divergence\nbetween p(\u02dcxT\nSolving the minimax optimization problem in (2) is a well-known, challenging task. Careful designs\nof network architectures and objective functions are essential to achieve good performance as shown\nin the literature [14, 21, 30, 37, 49, 51, 55, 72, 79]. We follow the same spirit and propose new network\ndesigns and a spatio-temporal objective for video-to-video synthesis as detailed below.\nSequential generator. To simplify the video-to-video synthesis problem, we make a Markov\nassumption where we factorize the conditional distribution p(\u02dcxT\n\n1 ) to a product form given by\n\n1 |sT\n\nT(cid:89)\n\np(\u02dcxT\n\n1 |sT\n\n1 ) =\n\np(\u02dcxt|\u02dcxt\u22121\n\nt\u2212L, st\n\nt\u2212L).\n\n(3)\n\nt=1\n\nIn other words, we assume the video frames can be generated sequentially, and the generation of the\nt-th frame \u02dcxt only depends on three factors: 1) current source frame st, 2) past L source frames st\u22121\nt\u2212L,\nand 3) past L generated frames \u02dcxt\u22121\nt\u2212L. We train a feed-forward network F to model the conditional\ndistribution p(\u02dcxt|\u02dcxt\u22121\n1 by applying\nthe function F in a recursive manner. We found that a small L (e.g., L = 1) causes training instability,\nwhile a large L increases training time and GPU memory but with minimal quality improvement. In\nour experiments, we set L = 2.\n\nt\u2212L). We obtain the \ufb01nal output \u02dcxT\n\nt\u2212L) using \u02dcxt = F (\u02dcxt\u22121\n\nt\u2212L, st\n\nt\u2212L, st\n\n3\n\n\fF (\u02dcxt\u22121\n\nVideo signals contain a large amount of redundant information in consecutive frames. If the optical\n\ufb02ow [46] between consecutive frames is known, we can estimate the next frame by warping the\ncurrent frame [54, 69]. This estimation would be largely correct except for the occluded areas. Based\non this observation, we model F as\n\nt\u2212L) = (1 \u2212 \u02dcmt) (cid:12) \u02dcwt\u22121(\u02dcxt\u22121) + \u02dcmt (cid:12) \u02dcht,\n\n(4)\nwhere (cid:12) is the element-wise product operator and 1 is an image of all ones. The \ufb01rst part corresponds\nto pixels warped from the previous frame, while the second part hallucinates new pixels. The\nde\ufb01nitions of the other terms in Equation 4 are given below.\n\u2022 \u02dcwt\u22121 = W (\u02dcxt\u22121\n\nt\u2212L) is the estimated optical \ufb02ow from \u02dcxt\u22121 to \u02dcxt, and W is the optical\nt\u2212L and\n\n\ufb02ow prediction network. We estimate the optical \ufb02ow using both input source images st\npreviously synthesized images \u02dcxt\u22121\n\nt\u2212L. By \u02dcwt\u22121(\u02dcxt\u22121), we warp \u02dcxt\u22121 based on \u02dcwt\u22121.\n\nt\u2212L, st\n\nt\u2212L, st\n\n\u2022 \u02dcht = H(\u02dcxt\u22121\n\u2022 \u02dcmt = M (\u02dcxt\u22121\n\nt\u2212L, st\nt\u2212L, st\n\nt\u2212L) is the hallucinated image, synthesized directly by the generator H.\nt\u2212L) is the occlusion mask with continuous values between 0 and 1. M denotes\nthe mask prediction network. Our occlusion mask is soft instead of binary to better handle the\n\u201czoom in\u201d scenario. For example, when an object is moving closer to our camera, the object will\nbecome blurrier over time if we only warp previous frames. To increase the resolution of the\nobject, we need to synthesize new texture details. By using a soft mask, we can add details by\ngradually blending the warped pixels and the newly synthesized pixels.\n\nWe use residual networks [26] for M, W , and H. To generate high-resolution videos, we adopt a\ncoarse-to-\ufb01ne generator design similar to the method of Wang et. al [72].\nAs using multiple discriminators can mitigate the mode collapse problem during GANs training [19,\n66, 72], we also design two types of discriminators as detailed below.\nConditional image discriminator DI. The purpose of DI is to ensure that each output frame\nresembles a real image given the same source image. This conditional discriminator should output 1\nfor a true pair (xt, st) and 0 for a fake one (\u02dcxt, st).\nConditional video discriminator DV . The purpose of DV is to ensure that consecutive output\nframes resemble the temporal dynamics of a real video given the same optical \ufb02ow. While DI\nt\u2212K be K \u2212 1 optical \ufb02ow for the\nconditions on the source image, DV conditions on the \ufb02ow. Let wt\u22122\nK consecutive real images xt\u22121\nt\u2212K. This conditional discriminator DV should output 1 for a true pair\n(xt\u22121\nt\u2212K, wt\u22122\nWe introduce two sampling operators to facilitate the discussion. First, let \u03c6I be a random image\nsampling operator such that \u03c6I (xT\n1 ) = (xi, si) where i is an integer uniformly sampled from 1 to\nT . In other words, \u03c6I randomly samples a pair of images from (xT\n1 ). Second, we de\ufb01ne \u03c6V as a\nsampling operator that randomly retrieve K consecutive frames. Speci\ufb01cally, \u03c6V (wT\u22121\n1 ) =\n(wi\u22122\ni\u2212K) where i is an integer uniformly sampled from K + 1 to T + 1. This operator\nretrieves K consecutive frames and the corresponding K \u2212 1 optical \ufb02ow images. With \u03c6I and \u03c6V ,\nwe are ready to present our learning objective function.\nLearning objective function. We train the sequential video synthesis function F by solving\n\nt\u2212K) and 0 for a fake one (\u02dcxt\u22121\n\nt\u2212K, wt\u22122\n\ni\u2212K, xi\u22121\n\ni\u2212K, si\u22121\n\nt\u2212K).\n\n1 , sT\n\n1 , sT\n\n1 , sT\n\n, xT\n\n1\n\nLV (F, DV )(cid:1) + \u03bbWLW (F ),\n\nLI (F, DI ) + max\n\nF\n\nDI\n\nDV\n\nmin\n\n(5)\nwhere LI is the GAN loss on images de\ufb01ned by the conditional image discriminator DI, LV is the\nGAN loss on K consecutive frames de\ufb01ned by DV , and LW is the \ufb02ow estimation loss. The weight\n\u03bbW is set to 10 throughout the experiments based on a grid search. In addition to the loss terms\nin Equation 5, we use the discriminator feature matching loss [40, 72] and VGG feature matching\nloss [16, 34, 72] as they improve the convergence speed and training stability [72]. Please see the\nsupplementary material for more details.\nWe further de\ufb01ne the image-conditional GAN loss LI [33] using the operator \u03c6I\n1 )[log(1 \u2212 DI (\u02dcxi, si))].\n\n1 )[log DI (xi, si)] + E\u03c6I (\u02dcxT\n\nE\u03c6I (xT\n\n1 ,sT\n\n1 ,sT\n\n(6)\n\n(cid:0) max\n\nSimilarly, the video GAN loss LV is given by\nE\u03c6V (wT \u22121\n\n1 )[log DV (xi\u22121\n\ni\u2212K, wi\u22122\n\n1 ,sT\n\n,xT\n\n1\n\ni\u2212K)] + E\u03c6V (wT\u22121\n\n1\n\n1 )[log(1 \u2212 DV (\u02dcxi\u22121\n\ni\u2212K, wi\u22122\n\ni\u2212K))].\n\n,\u02dcxT\n\n1 ,sT\n\n(7)\n\n4\n\n\fRecall that we synthesize a video \u02dcxT\nThe \ufb02ow loss LW includes two terms. The \ufb01rst is the endpoint error between the ground truth and\nthe estimated \ufb02ow, and the second is the warping loss when the \ufb02ow warps the previous frame to the\nnext frame. Let wt be the ground truth \ufb02ow from xt to xt+1. The \ufb02ow loss LW is given by\n\n1 by recursively applying F .\n\n(cid:0)|| \u02dcwt \u2212 wt(cid:107)1 + (cid:107) \u02dcwt(xt) \u2212 xt+1(cid:107)1\n\n(cid:1).\n\nT\u22121(cid:88)\n\nt=1\n\nLW =\n\n1\n\nT \u2212 1\n\n(8)\n\nt\u2212L, st\n\nt\u2212L) and a background model \u02dchB,t = HB(\u02dcxt\u22121\n\nt\u2212L, st\n\nF (\u02dcxt\u22121\n\nForeground-background prior. When using semantic segmentation masks as the source video, we\ncan divide an image into foreground and background areas based on the semantics. For example,\nbuildings and roads belong to the background, while cars and pedestrians are considered as the\nforeground. We leverage this strong foreground-background prior in the generator design to further\nimprove the synthesis performance of the proposed model.\nIn particular, we decompose the image hallucination network H into a foreground model \u02dchF,t =\nt\u2212L). We note that background motion\nHF (st\ncan be modeled as a global transformation in general, where optical \ufb02ow can be estimated quite\naccurately. As a result, the background region can be generated accurately via warping, and the\nbackground hallucination network HB only needs to synthesize the occluded areas. On the other hand,\na foreground object often has a large motion and only occupies a small portion of the image, which\nmakes optical \ufb02ow estimation dif\ufb01cult. The network HF has to synthesize most of the foreground\ncontent from scratch. With this foreground\u2013background prior, F is then given by\n\nt\u2212L) = (1 \u2212 \u02dcmt) (cid:12) \u02dcwt\u22121(\u02dcxt\u22121) + \u02dcmt (cid:12)(cid:0)(1 \u2212 mB,t) (cid:12) \u02dchF,t + mB,t (cid:12) \u02dchB,t\n\n(9)\nwhere mB,t is the background mask derived from the ground truth segmentation mask st. This prior\nimproves the visual quality by a large margin with the cost of minor \ufb02ickering artifacts. In Table 2,\nour user study shows that most people prefer the results with foreground\u2013background modeling. A\nqualitative comparison is also included in the supplementary material.\nMultimodal synthesis. The synthesis network F is a unimodal mapping function. Given an input\nsource video, it can only generate one output video. To achieve multimodal synthesis [19, 72, 82], we\nadopt a feature embedding scheme [72] for the source video that consists of instance-level semantic\nsegmentation masks. Speci\ufb01cally, at training time, we train an image encoder E to encode the ground\ntruth real image xt into a d-dimensional feature map (d = 3 in our experiments). We then apply\nan instance-wise average pooling to the map so that all the pixels within the same object share the\nsame feature vectors. We then feed both the instance-wise averaged feature map zt and the input\nsemantic segmentation mask st to the generator F . Once training is done, we \ufb01t a mixture of Gaussian\ndistribution to the feature vectors that belong to the same object class. At test time, we sample a\nfeature vector for each object instance using the estimated distribution of that object class. Given\ndifferent feature vectors, the generator F can synthesize videos with different visual appearances.\n4 Experiments\n\n(cid:1),\n\nImplementation details. We train our network in a spatio-temporally progressive manner.\nIn\nparticular, we start with generating low-resolution videos with few frames, and all the way up to\ngenerating full resolution videos with 30 (or more) frames. Our coarse-to-\ufb01ne generator consists of\nthree scales: 512\u00d7 256, 1024\u00d7 512, and 2048\u00d7 1024 resolutions, respectively. The mask prediction\nnetwork M and \ufb02ow prediction network W share all the weights except for the output layer. We\nuse the multi-scale PatchGAN discriminator architecture [33, 72] for the image discriminator DI. In\naddition to multi-scale in the spatial resolution, our multi-scale video discriminator DV also looks\nat different frame rates of the video to ensure both short-term and long-term consistency. See the\nsupplementary material for more details.\nWe train our model for 40 epochs using the ADAM optimizer [39] with lr = 0.0002 and (\u03b21, \u03b22) =\n(0.5, 0.999) on an NVIDIA DGX1 machine. We use the LSGAN loss [49]. Due to the high image\nresolution, even with one short video per batch, we have to use all the GPUs in DGX1 (8 V100 GPUs,\neach with 16GB memory) for training. We distribute the generator computation task to 4 GPUs and\nthe discriminator task to the other 4 GPUs. Training takes \u223c 10 days for 2K resolution.\nDatasets. We evaluate the proposed approach on several datasets.\n\n5\n\n\fTable 1: Comparison between competing video-to-video synthesis approaches on Cityscapes.\nFr\u00e9chet Inception Dist.\n\nHuman Preference Score\n\npix2pixHD\n\nCOVST\n\nvid2vid (ours)\n\nI3D ResNeXt\n5.57\n5.55\n4.66\n\n0.18\n0.18\n0.15\n\nvid2vid (ours)\n\n/ pix2pixHD\n\nvid2vid (ours)\n\n/ COVST\n\nshort seq.\n0.87 / 0.13\n0.84 / 0.16\n\nlong seq.\n0.83 / 0.17\n0.80 / 0.20\n\nTable 2: Ablation study. We compare the proposed approach to its three variants.\n\nHuman Preference Score\n\nvid2vid (ours)\n\nvid2vid (ours)\n\n/ no background\u2013foreground prior\n\n/ no conditional video discriminator\n\nvid2vid (ours)\n\n/ no flow warping\n\n0.80 / 0.20\n0.84 / 0.16\n0.67 / 0.33\n\nTable 3: Comparison between future video prediction methods on Cityscapes.\n\nFr\u00e9chet Inception Dist.\n\nPredNet\nMCNet\n\nvid2vid (ours)\n\nI3D\n11.18\n10.00\n3.44\n\nResNeXt\n\nHuman Preference Score\n\n0.59\n0.43\n0.18\n\nvid2vid (ours)\nvid2vid (ours)\n\n/ PredNet\n/ MCNet\n\n0.92 / 0.08\n0.98 / 0.02\n\n\u2022 Cityscapes [13]. The dataset consists of 2048 \u00d7 1024 street scene videos captured in several\nGerman cities. Only a subset of images in the videos contains ground truth semantic segmentation\nmasks. To obtain the input source videos, we use those images to train a DeepLabV3 semantic\nsegmentation network [11] and apply the trained network to segment all the videos. We use\nthe optical \ufb02ow extracted by FlowNet2 [32] as the ground truth \ufb02ow w. We treat the instance\nsegmentation masks computed by the Mask R-CNN [25] as our instance-level ground truth. In\nsummary, the training set contains 2975 videos, each with 30 frames. The validation set consists\nof 500 videos, each with 30 frames. Finally, we test our method on three long sequences from\nthe Cityscapes demo videos, with 600, 1100, and 1200 frames, respectively. We will show that\nalthough trained on short videos, our model can synthesize long videos.\n\n\u2022 Apolloscape [29] consists of 73 street scene videos captured in Beijing, whose video lengths vary\nfrom 100 to 1000 frames. Similar to Cityscapes, Apolloscape is constructed for the image/video\nsemantic segmentation task. But we use it for synthesizing videos using the semantic segmentation\nmask. We split the dataset into half for training and validation.\n\n\u2022 Face video dataset [57]. We use the real videos in the FaceForensics dataset, which contains\n854 videos of news brie\ufb01ng from different reporters. We use this dataset for the sketch video to\nface video synthesis task. To extract a sequence of sketches from a video, we \ufb01rst apply a face\nalignment algorithm [38] to localize facial landmarks in each frame. The facial landmarks are\nthen connected to create the face sketch. For background, we extract Canny edges outside the\nface regions. We split the dataset into 704 videos for training and 150 videos for validation.\n\n\u2022 Dance video dataset. We download YouTube dance videos for the pose to human motion\nsynthesis task. Each video is about 3 \u223c 4 minutes long at 1280 \u00d7 720 resolution, and we crop the\ncentral 512 \u00d7 720 regions. We extract human poses with DensePose [23] and OpenPose [7], and\ndirectly concatenate the results together. Each training set includes a dance video from a single\ndancer, while the test set contains videos of other dance motions or from other dancers.\n\napproach to the video-to-video synthesis task, we process input videos frame-by-frame.\n\nBaselines. We compare our approach to two baselines trained on the same data.\n\u2022 pix2pixHD [72] is the state-of-the-art image-to-image translation approach. When applying the\n\u2022 COVST is built on the coherent video style transfer [10] by replacing the stylization network with\npix2pixHD. The key idea in COVST is to warp high-level deep features using optical \ufb02ow for\nachieving temporally coherent outputs. No additional adversarial training is applied. We feed in\nground truth optical \ufb02ow to COVST, which is impractical for real applications. In contrast, our\nmodel estimates optical \ufb02ow from source videos.\n\nEvaluation metrics. We use both subjective and objective metrics for evaluation.\n\u2022 Human preference score. We perform a human subjective test for evaluating the visual quality\nof synthesized videos. We use the Amazon Mechanical Turk (AMT) platform. During each\n\n6\n\n\fFigure 2: Apolloscape results. Left: pix2pixHD. Center: COVST. Right: proposed. The input\nsemantic segmentation mask video is shown in the left video. The \ufb01gure is best viewed with Acrobat\nReader. Click the image to play the video clip.\n\nFigure 3: Example multi-modal video synthesis results. These synthesized videos contain different\nroad surfaces. The \ufb01gure is best viewed with Acrobat Reader. Click the image to play the video clip.\n\nFigure 4: Example results of changing input semantic segmentation masks to generate diverse videos.\nLeft: tree\u2192building. Right: building\u2192tree. The original video is shown in Figure 3. The \ufb01gure is\nbest viewed with Acrobat Reader. Click the image to play the video clip.\n\ntest, an AMT participant is \ufb01rst shown two videos at a time (results synthesized by two different\nalgorithms) and then asked which one looks more like a video captured by a real camera. We\nspeci\ufb01cally ask the worker to check for both temporal coherence and image quality. A worker\nmust have a life-time task approval rate greater than 98% to participate in the evaluation. For each\nquestion, we gather answers from 10 different workers. We evaluate the algorithm by the ratio\nthat the algorithm outputs are preferred.\n\u2022 Fr\u00e9chet Inception Distance (FID) [27] is a widely used metric for implicit generative models, as\nit correlates well with the visual quality of generated samples. The FID was originally developed\nfor evaluating image generation. We propose a variant for video evaluation, which measures both\nvisual quality and temporal consistency. Speci\ufb01cally, we use a pre-trained video recognition CNN\nas a feature extractor after removing the last few layers from the network. This feature extractor\nwill be our \u201cinception\u201d network. For each video, we extract a spatio-temporal feature map with\nthis CNN. We then compute the mean \u02dc\u00b5 and covariance matrix \u02dc\u03a3 for the feature vectors from\nall the synthesized videos. We also calculate the same quantities \u00b5 and \u03a3 for the ground truth\nvideos. The FID is then calculated as (cid:107)\u00b5 \u2212 \u02dc\u00b5(cid:107)2 + Tr\n. We use two different\npre-trained video recognition CNNs in our evaluation: I3D [8] and ResNeXt [75].\n\n\u03a3 + \u02dc\u03a3 \u2212 2\n\n(cid:112)\n\n(cid:16)\n\n(cid:17)\n\n\u03a3 \u02dc\u03a3\n\nMain results. We compare the proposed approach to the baselines on the Cityscapes benchmark,\nwhere we apply the learned models to synthesize 500 short video clips in the validation set. As shown\nin Table 1, our results have a smaller FID and are often favored by the human subjects. We also\nreport the human preference scores on the three long test videos. Again, the videos rendered by our\napproach are considered more realistic by the human subjects. The human preference scores for the\nApolloscape dataset are given in the supplementary material.\n\n7\n\n\fFigure 5: Example face\u2192sketch\u2192face results. Each set shows the original video, the extracted edges,\nand our synthesized video. The \ufb01gure is best viewed with Acrobat Reader. Click the image to play the\nvideo clip.\n\nFigure 6: Example dance\u2192pose\u2192dance results. Each set shows the original dancer, the extracted\nposes, and the synthesized video. The \ufb01gure is best viewed with Acrobat Reader. Click the image to\nplay the video clip.\n\nFigures 1 and 2 show the video synthesis results. Although each frame rendered by pix2pixHD is\nphotorealistic, the resulting video lacks temporal coherence. The road lane markings and building\nappearances are inconsistent across frames. While improving upon pix2pixHD, COVST still suffers\nfrom temporal inconsistency. On the contrary, our approach produces a high-resolution, photorealistic,\ntemporally consistent video output. We can generate 30-second long videos, showing that our\napproach synthesizes convincing videos with longer lengths.\nWe conduct an ablation study to analyze several design choices of our method. Speci\ufb01cally, we\ncreate three variants. In one variant, we do not use the foreground-background prior, which is termed\nno background\u2013foreground prior. That is, instead of using Equation 9, we use Equation 4.\nThe second variant is no conditional video discriminator where we do not use the video\ndiscriminator DV for training. In the last variant, we remove the optical \ufb02ow prediction network\nW and the mask prediction network M from the generator F in Equation 4 and only use H for\nsynthesis. This variant is referred to as no flow warping. We use the human preference score on\nCityscapes for this ablation study. Table 2 shows that the visual quality of output videos degrades\nsigni\ufb01cantly without the ablated components. To evaluate the effectiveness of different components\nin our network, we also experimented with directly using ground truth \ufb02ows instead of estimated\n\ufb02ows by our network. An example can be found in the supplementary material. We found the results\nvisually similar, which suggests that our network is robust to the errors in the estimated \ufb02ows.\nMultimodal results. Figure 3 shows example multimodal synthesis results. In this example, we\nkeep the sampled feature vectors of all the object instances in the video the same except for the road\ninstance. The \ufb01gure shows temporally smooth videos with different road appearances.\nSemantic manipulation. Our approach also allows the user to manipulate the semantics of source\nvideos. In Figure 4, we show an example of changing the semantic labels. In the left video, we\nreplace all trees with buildings in the original segmentation masks and synthesize a new video. On\nthe right, we show the result of replacing buildings with trees.\nSketch-to-video synthesis for face swapping. We train a sketch-to-face synthesis video model\nusing the real face videos in the FaceForensics dataset [57]. As shown in Figure 5, our model can\nconvert sequences of sketches to photorealistic output videos. This model can be used to change the\nfacial appearance of the original face videos [5].\nPose-to-video synthesis for human motion transfer. We also apply our method to the task of\nconverting sequences of human poses to photorealistic output videos. We note that the image\ncounterpart was studied in recent works [4, 17, 47, 48]. As shown in Figure 6, our model learns to\nsynthesize high-resolution photorealistic output dance videos that contain unseen body shapes and\n\n8\n\n\fFigure 7: Future video prediction results. Top left: ground truth. Top right: PredNet [45]. Bottom\nleft: MCNet [67]. Bottom right: ours. The \ufb01gure is best viewed with Acrobat Reader. Click the image\nto play the video clip.\n\nmotions. Our method can change the clothing [78, 80] for the same dancer (Figure 6 left) as well as\ntransfer the visual appearance to new dancers (Figure 6 right) as explored in concurrent work [1,9,77].\nFuture video prediction. We show an extension of our approach to the future video prediction\ntask: learning to predict the future video given a few observed frames. We decompose the task\ninto two sub-tasks: 1) synthesizing future semantic segmentation masks using the observed frames,\nand 2) converting the synthesized segmentation masks into videos. In practice, after extracting the\nsegmentation masks from the observed frames, we train a generator to predict future semantic masks.\nWe then use the proposed video-to-video synthesis approach to convert the predicted segmentation\nmasks to a future video.\nWe conduct both quantitative and qualitative evaluations with comparisons to two start-of-the-art\napproaches: PredNet [45] and MCNet [67]. We follow the prior work [41, 69] and report the human\npreference score. We also include the FID scores. As shown in Table 3, our model produces smaller\nFIDs, and the human subjects favor our resulting videos. In Figure 7, we visualize the future video\nsynthesis results. While the image quality of the results from the competing algorithms degrades\nsigni\ufb01cantly over time, ours remains consistent.\n\n5 Discussion\nWe present a general video-to-video synthesis framework based on conditional GANs. Through\ncarefully-designed generators and discriminators as well as a spatio-temporal adversarial objective,\nwe can synthesize high-resolution, photorealistic, and temporally consistent videos. Extensive\nexperiments demonstrate that our results are signi\ufb01cantly better than the results by state-of-the-art\nmethods. Our method also compares favorably against the competing video prediction methods.\nAlthough our approach outperforms previous methods, our model still fails in a couple of situations.\nFor example, our model struggles in synthesizing turning cars due to insuf\ufb01cient information in\nlabel maps. This could be potentially addressed by adding additional 3D cues, such as depth maps.\nFurthermore, our model still can not guarantee that an object has a consistent appearance across\nthe whole video. Occasionally, a car may change its color gradually. This issue might be alleviated\nif object tracking information is used to enforce that the same object shares the same appearance\nthroughout the entire video. Finally, when we perform semantic manipulations such as turning trees\ninto buildings, visible artifacts occasionally appear as building and trees have different label shapes.\nThis might be resolved if we train our model with coarser semantic labels, as the trained model would\nbe less sensitive to label shapes.\nAcknowledgements We thank Karan Sapra, Fitsum Reda, and Matthieu Le for generating the\nsegmentation maps for us. We also thank Lisa Rhee and Miss Ketsuki for allowing us to use their\ndance videos for training. We thank William S. Peebles for proofreading the paper.\n\n9\n\n\fReferences\n\n[1] K. Aberman, M. Shi, J. Liao, D. Lischinski, B. Chen, and D. Cohen-Or. Deep video-based\n\nperformance cloning. arXiv preprint arXiv:1808.06847, 2018.\n\n[2] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath. Deep reinforcement\n\nlearning: A brief survey. IEEE Signal Processing Magazine, 34(6):26\u201338, 2017.\n\n[3] X. Bai, J. Wang, D. Simons, and G. Sapiro. Video snapcut: robust video object cutout using\n\nlocalized classi\ufb01ers. ACM Transactions on Graphics (TOG), 28(3):70, 2009.\n\n[4] G. Balakrishnan, A. Zhao, A. V. Dalca, F. Durand, and J. Guttag. Synthesizing images of\nhumans in unseen poses. In IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), 2018.\n\n[5] D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, and S. K. Nayar. Face swapping: automatically\n\nreplacing faces in photographs. In ACM SIGGRAPH, 2008.\n\n[6] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan. Unsupervised pixel-level\ndomain adaptation with generative adversarial networks. In IEEE Conference on Computer\nVision and Pattern Recognition (CVPR), 2017.\n\n[7] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime multi-person 2D pose estimation using\npart af\ufb01nity \ufb01elds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2017.\n\n[8] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics\n\ndataset. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[9] C. Chan, S. Ginosar, T. Zhou, and A. A. Efros. Everybody dance now. In European Conference\n\non Computer Vision (ECCV) Workshop, 2018.\n\n[10] D. Chen, J. Liao, L. Yuan, N. Yu, and G. Hua. Coherent online video style transfer. In IEEE\n\nInternational Conference on Computer Vision (ICCV), 2017.\n\n[11] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for\n\nsemantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.\n\n[12] T. Chen, J.-Y. Zhu, A. Shamir, and S.-M. Hu. Motion-aware gradient domain video composition.\n\nIEEE Trans. Image Processing, 22(7):2532\u20132544, 2013.\n\n[13] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth,\nand B. Schiele. The Cityscapes dataset for semantic urban scene understanding. In IEEE\nConference on Computer Vision and Pattern Recognition (CVPR), 2016.\n\n[14] E. Denton, S. Chintala, A. Szlam, and R. Fergus. Deep generative image models using a\nLaplacian pyramid of adversarial networks. In Advances in Neural Information Processing\nSystems (NIPS), 2015.\n\n[15] E. L. Denton and V. Birodkar. Unsupervised learning of disentangled representations from\n\nvideo. In Advances in Neural Information Processing Systems (NIPS), 2017.\n\n[16] A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on\n\ndeep networks. In Advances in Neural Information Processing Systems (NIPS), 2016.\n\n[17] P. Esser, E. Sutter, and B. Ommer. A variational u-net for conditional appearance and shape\ngeneration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n[18] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through\n\nvideo prediction. In Advances in Neural Information Processing Systems (NIPS), 2016.\n\n[19] A. Ghosh, V. Kulharia, V. Namboodiri, P. H. Torr, and P. K. Dokania. Multi-agent diverse gen-\nerative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), 2018.\n\n[20] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and\nY. Bengio. Generative adversarial networks. In Advances in Neural Information Processing\nSystems (NIPS), 2014.\n\n[21] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of\n\nwasserstein GANs. In Advances in Neural Information Processing Systems (NIPS), 2017.\n\n[22] A. Gupta, J. Johnson, A. Alahi, and L. Fei-Fei. Characterizing and improving stability in neural\n\nstyle transfer. In IEEE International Conference on Computer Vision (ICCV), 2017.\n\n[23] R. A. G\u00fcler, N. Neverova, and I. Kokkinos. Densepose: Dense human pose estimation in the\n\nwild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n\n[24] D. Ha and J. Schmidhuber. World models. In Advances in Neural Information Processing\n\nSystems (NIPS), 2018.\n\n10\n\n\f[25] K. He, G. Gkioxari, P. Doll\u00e1r, and R. Girshick. Mask R-CNN. In IEEE International Conference\n\non Computer Vision (ICCV), 2017.\n\n[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE\n\nConference on Computer Vision and Pattern Recognition (CVPR), 2016.\n\n[27] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two\ntime-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information\nProcessing Systems (NIPS), 2017.\n\n[28] H. Huang, H. Wang, W. Luo, L. Ma, W. Jiang, X. Zhu, Z. Li, and W. Liu. Real-time neural style\ntransfer for videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2017.\n\n[29] X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, and R. Yang. The Apol-\nloScape dataset for autonomous driving. In IEEE Conference on Computer Vision and Pattern\nRecognition (CVPR), 2018.\n\n[30] X. Huang, Y. Li, O. Poursaeed, J. E. Hopcroft, and S. J. Belongie. Stacked generative adversarial\n\nnetworks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[31] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal unsupervised image-to-image\n\ntranslation. In ECCV, 2018.\n\n[32] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution\nof optical \ufb02ow estimation with deep networks. In IEEE Conference on Computer Vision and\nPattern Recognition (CVPR), 2017.\n\n[33] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional\nadversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2017.\n\n[34] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-\n\nresolution. In European Conference on Computer Vision (ECCV), 2016.\n\n[35] J. T. Kajiya. The rendering equation. In ACM SIGGRAPH, 1986.\n[36] N. Kalchbrenner, A. v. d. Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and\n\nK. Kavukcuoglu. Video pixel networks. arXiv preprint arXiv:1610.00527, 2016.\n\n[37] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality,\nstability, and variation. In International Conference on Learning Representations (ICLR), 2018.\n[38] D. E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 2009.\n[39] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference\n\non Learning Representations (ICLR), 2015.\n\n[40] A. B. L. Larsen, S. K. S\u00f8nderby, H. Larochelle, and O. Winther. Autoencoding beyond pixels\nusing a learned similarity metric. In International Conference on Machine Learning (ICML),\n2016.\n\n[41] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video\n\nprediction. arXiv preprint arXiv:1804.01523, 2018.\n\n[42] X. Liang, L. Lee, W. Dai, and E. P. Xing. Dual motion GAN for future-\ufb02ow embedded video\n\nprediction. In Advances in Neural Information Processing Systems (NIPS), 2017.\n\n[43] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. In\n\nAdvances in Neural Information Processing Systems (NIPS), 2017.\n\n[44] M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. In Advances in Neural\n\nInformation Processing Systems (NIPS), 2016.\n\n[45] W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and\nunsupervised learning. In International Conference on Learning Representations (ICLR), 2017.\n[46] B. D. Lucas, T. Kanade, et al. An iterative image registration technique with an application to\n\nstereo vision. International Joint Conference on Arti\ufb01cial Intelligence (IJCAI), 1981.\n\n[47] L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L. Van Gool. Pose guided person image\n\ngeneration. In Advances in Neural Information Processing Systems (NIPS), 2017.\n\n[48] L. Ma, Q. Sun, S. Georgoulis, L. Van Gool, B. Schiele, and M. Fritz. Disentangled person\nimage generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2018.\n\n[49] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley. Least squares generative\n\nadversarial networks. In IEEE International Conference on Computer Vision (ICCV), 2017.\n\n[50] M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square\n\nerror. In International Conference on Learning Representations (ICLR), 2016.\n\n11\n\n\f[51] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative\nadversarial networks. In International Conference on Learning Representations (ICLR), 2018.\n[52] T. Miyato and M. Koyama. cGANs with projection discriminator. In International Conference\n\non Learning Representations (ICLR), 2018.\n\n[53] A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classi\ufb01er GANs.\n\nIn International Conference on Machine Learning (ICML), 2017.\n\n[54] K. Ohnishi, S. Yamamoto, Y. Ushiku, and T. Harada. Hierarchical video generation from\n\northogonal information: Optical \ufb02ow and texture. In AAAI, 2018.\n\n[55] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolu-\ntional generative adversarial networks. In International Conference on Learning Representations\n(ICLR), 2015.\n\n[56] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text\n\nto image synthesis. In International Conference on Machine Learning (ICML), 2016.\n\n[57] A. R\u00f6ssler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nie\u00dfner. Faceforensics: A\nlarge-scale video dataset for forgery detection in human faces. arXiv preprint arXiv:1803.09179,\n2018.\n\n[58] M. Ruder, A. Dosovitskiy, and T. Brox. Artistic style transfer for videos. In German Conference\n\non Pattern Recognition, 2016.\n\n[59] M. Saito, E. Matsumoto, and S. Saito. Temporal generative adversarial nets with singular value\n\nclipping. In IEEE International Conference on Computer Vision (ICCV), 2017.\n\n[60] A. Sch\u00f6dl, R. Szeliski, D. H. Salesin, and I. Essa. Video textures. ACM Transactions on\n\nGraphics (TOG), 2000.\n\n[61] E. Shechtman, Y. Caspi, and M. Irani. Space-time super-resolution. IEEE Transactions on\n\nPattern Analysis and Machine Intelligence (TPAMI), 27(4):531\u2013545, 2005.\n\n[62] W. Shi, J. Caballero, F. Husz\u00e1r, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang.\nReal-time single image and video super-resolution using an ef\ufb01cient sub-pixel convolutional\nneural network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2016.\n\n[63] A. Shrivastava, T. P\ufb01ster, O. Tuzel, J. Susskind, W. Wang, and R. Webb. Learning from\nIn IEEE Conference on\n\nsimulated and unsupervised images through adversarial training.\nComputer Vision and Pattern Recognition (CVPR), 2017.\n\n[64] N. Srivastava, E. Mansimov, and R. Salakhudinov. Unsupervised learning of video representa-\n\ntions using lstms. In International Conference on Machine Learning (ICML), 2015.\n\n[65] Y. Taigman, A. Polyak, and L. Wolf. Unsupervised cross-domain image generation.\n\nInternational Conference on Learning Representations (ICLR), 2017.\n\nIn\n\n[66] S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz. MoCoGAN: Decomposing motion and content\nfor video generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2018.\n\n[67] R. Villegas, J. Yang, S. Hong, X. Lin, and H. Lee. Decomposing motion and content for natural\nvideo sequence prediction. In International Conference on Learning Representations (ICLR),\n2017.\n\n[68] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In\n\nAdvances in Neural Information Processing Systems (NIPS), 2016.\n\n[69] C. Vondrick and A. Torralba. Generating the future with adversarial transformers. In IEEE\n\nConference on Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[70] J. Walker, C. Doersch, A. Gupta, and M. Hebert. An uncertain future: Forecasting from static\nimages using variational autoencoders. In European Conference on Computer Vision (ECCV),\n2016.\n\n[71] J. Walker, K. Marino, A. Gupta, and M. Hebert. The pose knows: Video forecasting by\ngenerating pose futures. In IEEE International Conference on Computer Vision (ICCV), 2017.\n[72] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image\nsynthesis and semantic manipulation with conditional GANs. In IEEE Conference on Computer\nVision and Pattern Recognition (CVPR), 2018.\n\n[73] Y. Wexler, E. Shechtman, and M. Irani. Space-time video completion. In IEEE Conference on\n\nComputer Vision and Pattern Recognition (CVPR), 2004.\n\n[74] Y. Wexler, E. Shechtman, and M. Irani. Space-time completion of video. IEEE Transactions on\n\nPattern Analysis and Machine Intelligence (TPAMI), 29(3), 2007.\n\n12\n\n\f[75] S. Xie, R. Girshick, P. Doll\u00e1r, Z. Tu, and K. He. Aggregated residual transformations for deep\nneural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),\n2017.\n\n[76] T. Xue, J. Wu, K. Bouman, and B. Freeman. Visual dynamics: Probabilistic future frame\nsynthesis via cross convolutional networks. In Advances in Neural Information Processing\nSystems (NIPS), 2016.\n\n[77] C. Yang, Z. Wang, X. Zhu, C. Huang, J. Shi, and D. Lin. Pose guided human video generation.\n\nIn European Conference on Computer Vision (ECCV), 2018.\n\n[78] S. Yang, T. Ambert, Z. Pan, K. Wang, L. Yu, T. Berg, and M. C. Lin. Detailed garment recovery\n\nfrom a single-view image. arXiv preprint arXiv:1608.01250, 2016.\n\n[79] H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas. StackGAN: Text\nIn IEEE\n\nto photo-realistic image synthesis with stacked generative adversarial networks.\nInternational Conference on Computer Vision (ICCV), 2017.\n\n[80] Z.-H. Zheng, H.-T. Zhang, F.-L. Zhang, and T.-J. Mu. Image-based clothes changing system.\n\nComputational Visual Media, 3(4):337\u2013347, 2017.\n\n[81] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-\nconsistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV),\n2017.\n\n[82] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward\nmultimodal image-to-image translation. In Advances in Neural Information Processing Systems\n(NIPS), 2017.\n\n13\n\n\f", "award": [], "sourceid": 603, "authors": [{"given_name": "Ting-Chun", "family_name": "Wang", "institution": "NVIDIA"}, {"given_name": "Ming-Yu", "family_name": "Liu", "institution": "NVIDIA"}, {"given_name": "Jun-Yan", "family_name": "Zhu", "institution": "MIT"}, {"given_name": "Guilin", "family_name": "Liu", "institution": "NVIDIA"}, {"given_name": "Andrew", "family_name": "Tao", "institution": "Nvidia Corporation"}, {"given_name": "Jan", "family_name": "Kautz", "institution": "NVIDIA"}, {"given_name": "Bryan", "family_name": "Catanzaro", "institution": "NVIDIA"}]}