{"title": "Neural Voice Cloning with a Few Samples", "book": "Advances in Neural Information Processing Systems", "page_first": 10019, "page_last": 10029, "abstract": "Voice cloning is a highly desired feature for personalized speech interfaces. We introduce a neural voice cloning system that learns to synthesize a person's voice from only a few audio samples. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model. Speaker encoding is based on training a separate model to directly infer a new speaker embedding, which will be applied to a multi-speaker generative model. In terms of naturalness of the speech and similarity to the original speaker, both approaches can achieve good performance, even with a few cloning audios.  While speaker adaptation can achieve slightly better naturalness and similarity, cloning time and required memory for the speaker encoding approach are significantly less, making it more favorable for low-resource deployment.", "full_text": "Neural Voice Cloning with a Few Samples\n\nSercan \u00d6. Ar\u0131k\u2217\n\nJitong Chen\u2217\n\nKainan Peng\u2217\n\nsercanarik@baidu.com\n\nchenjitong01@baidu.com\n\npengkainan@baidu.com\n\nWei Ping\u2217\n\npingwei01@baidu.com\n\nYanqi Zhou\n\nyanqiz@baidu.com\n\nBaidu Research\n\n1195 Bordeaux Dr. Sunnyvale, CA 94089\n\nAbstract\n\nVoice cloning is a highly desired feature for personalized speech interfaces. We\nintroduce a neural voice cloning system that learns to synthesize a person\u2019s voice\nfrom only a few audio samples. We study two approaches: speaker adaptation\nand speaker encoding. Speaker adaptation is based on \ufb01ne-tuning a multi-speaker\ngenerative model. Speaker encoding is based on training a separate model to\ndirectly infer a new speaker embedding, which will be applied to a multi-speaker\ngenerative model. In terms of naturalness of the speech and similarity to the original\nspeaker, both approaches can achieve good performance, even with a few cloning\naudios. 2 While speaker adaptation can achieve slightly better naturalness and\nsimilarity, cloning time and required memory for the speaker encoding approach\nare signi\ufb01cantly less, making it more favorable for low-resource deployment.\n\n1\n\nIntroduction\n\nGenerative models based on deep neural networks have been successfully applied to many domains\nsuch as image generation [e.g., Oord et al., 2016b, Karras et al., 2017], speech synthesis [e.g., Oord\net al., 2016a, Arik et al., 2017a, Wang et al., 2017], and language modeling [e.g., Jozefowicz et al.,\n2016]. Deep neural networks are capable of modeling complex data distributions and can be further\nconditioned on external inputs to control the content and style of generated samples.\n\nIn speech synthesis, generative models can be conditioned on text and speaker identity [e.g., Arik\net al., 2017b]. While text carries linguistic information and controls the content of the generated\nspeech, speaker identity captures characteristics such as pitch, speech rate and accent. One approach\nfor multi-speaker speech synthesis is to jointly train a generative model and speaker embeddings\non triplets of text, audio and speaker identity [e.g., Ping et al., 2018]. The idea is to encode the\nspeaker-dependent information with low-dimensional embeddings, while sharing the majority of the\nmodel parameters across all speakers. One limitation of such methods is that they can only generate\nspeech for observed speakers during training. An intriguing task is to learn the voice of an unseen\nspeaker from a few speech samples, a.k.a. voice cloning, which corresponds to few-shot generative\nmodeling of speech conditioned on the speaker identity. While a generative model can be trained\nfrom scratch with a large amount of audio samples 3, we focus on voice cloning of a new speaker\nwith a few minutes or even few seconds data. It is challenging as the model has to learn the speaker\ncharacteristics from very limited amount of data, and still generalize to unseen texts.\n\n\u2217Equal contribution\n2Cloned audio samples can be found in https://audiodemos.github.io\n3A single speaker model can require \u223c20 hours of training data [e.g., Arik et al., 2017a, Wang et al., 2017],\nwhile a multi-speaker model for 108 speakers [Arik et al., 2017b] requires about \u223c20 minutes data per speaker.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fIn this paper, we investigate voice cloning in sequence-to-sequence neural speech synthesis sys-\ntems [Ping et al., 2018]. Our contributions are the following:\n\n1. We demonstrate and analyze the strength of speaker adaption approaches for voice cloning,\nbased on \ufb01ne-tuning a pre-trained multi-speaker model for an unseen speaker using a few\nsamples.\n\n2. We propose a novel speaker encoding approach, which provides comparable naturalness\nand similarity in subjective evaluations while yielding signi\ufb01cantly less cloning time and\ncomputational resource requirements.\n\n3. We propose automated evaluation methods for voice cloning based on neural speaker classi\ufb01-\n\ncation and speaker veri\ufb01cation.\n\n4. We demonstrate voice morphing for gender and accent transformation via embedding manip-\n\nulations.\n\n2 Related Work\n\nOur work builds upon the state-of-the-art in neural speech synthesis and few-shot generative modeling.\n\nNeural speech synthesis: Recently, there is a surge of interest in speech synthesis with neu-\nral networks, including Deep Voice 1 [Arik et al., 2017a], Deep Voice 2 [Arik et al., 2017b],\nDeep Voice 3 [Ping et al., 2018], WaveNet [Oord et al., 2016a], SampleRNN [Mehri et al., 2016],\nChar2Wav [Sotelo et al., 2017], Tacotron [Wang et al., 2017] and VoiceLoop [Taigman et al., 2018].\nAmong these methods, sequence-to-sequence models [Ping et al., 2018, Wang et al., 2017, Sotelo\net al., 2017] with attention mechanism have much simpler pipeline and can produce more natural\nspeech [e.g., Shen et al., 2017]. In this work, we use Deep Voice 3 as the baseline multi-speaker\nmodel, because of its simple convolutional architecture and high ef\ufb01ciency for training and fast model\nadaptation. It should be noted that our techniques can be seamlessly applied to other neural speech\nsynthesis models.\n\nFew-shot generative modeling: Humans can learn new generative tasks from only a few examples,\nwhich motivates research on few-shot generative models. Early studies mostly focus on Bayesian\nmethods. For example, hierarchical Bayesian models are used to exploit compositionality and\ncausality for few-shot generation of characters [Lake et al., 2013, 2015] and words in speech [Lake\net al., 2014]. Recently, deep neural networks achieve great successes in few-shot density estimation\nand conditional image generation [e.g., Rezende et al., 2016, Reed et al., 2017, Azadi et al., 2017],\nbecause of the great potential for composition in their learned representation. In this work, we\ninvestigate few-shot generative modeling of speech conditioned on a particular speaker. We train a\nseparate speaker encoding network to directly predict the parameters of multi-speaker generative\nmodel by only taking unsubscribed audio samples as inputs.\n\nSpeaker-dependent speech processing: Speaker-dependent modeling has been widely studied for\nautomatic speech recognition (ASR), with the goal of improving the performance by exploiting\nspeaker characteristics. In particular, there are two groups of methods in neural ASR, in alignment\nwith our two voice cloning approaches. The \ufb01rst group is speaker adaptation for the whole-model [Yu\net al., 2013], a portion of the model [Miao and Metze, 2015, Cui et al., 2017], or merely to a speaker\nembedding [Abdel-Hamid and Jiang, 2013, Xue et al., 2014]. Speaker adaptation for voice cloning is\nin the same vein as these approaches, but differences arise when text-to-speech vs. speech-to-text\nare considered [Yamagishi et al., 2009]. The second group is based on training ASR models jointly\nwith embeddings. Extraction of the embeddings can be based on i-vectors [Miao et al., 2015], or\nbottleneck layers of neural networks trained with a classi\ufb01cation loss [Li and Wu, 2015]. Although\nthe general idea of speaker encoding is also based on extracting the embeddings directly, as a\nmajor distinction, our speaker encoder models are trained with an objective function that is directly\nrelated to speech synthesis. Lastly, speaker-dependent modeling is essential for multi-speaker speech\nsynthesis. Using i-vectors to represent speaker-dependent characteristics is one approach [Wu et al.,\n2015], however, they have the limitation of being separately trained, with an objective that is not\ndirectly related to speech synthesis. Also they may not be accurately extracted with small amount of\naudio [Miao et al., 2015]. Another approach for multi-speaker speech synthesis is using trainable\nspeaker embeddings [Arik et al., 2017b], which are randomly initialized and jointly optimized from a\ngenerative loss function.\n\n2\n\n\fVoice conversion: A closely related task of voice cloning is voice conversion. The goal of voice\nconversion is to modify an utterance from source speaker to make it sound like the target speaker,\nwhile keeping the linguistic contents unchanged. Unlike voice cloning, voice conversion systems\ndo not need to generalize to unseen texts. One common approach is dynamic frequency warping,\nto align spectra of different speakers. Agiomyrgiannakis and Roupakia [2016] proposes a dynamic\nprogramming algorithm that simultaneously estimates the optimal frequency warping and weighting\ntransform while matching source and target speakers using a matching-minimization algorithm. Wu\net al. [2016] uses a spectral conversion approach integrated with the locally linear embeddings for\nmanifold learning. There are also approaches to model spectral conversion using neural networks\n[Desai et al., 2010, Chen et al., 2014, Hwang et al., 2015]. Those models are typically trained with a\nlarge amount of audio pairs of target and source speakers.\n\n3 From Multi-Speaker Generative Modeling to Voice Cloning\n\nWe consider a multi-speaker generative model, f (ti,j, si; W, esi ), which takes a text ti,j and a\nspeaker identity si. The trainable parameters in the model is parameterized by W , and esi. The\nlatter denotes the trainable speaker embedding corresponding to si. Both W and esi are optimized\nby minimizing a loss function L that penalizes the difference between generated and ground-truth\naudios (e.g. a regression loss for spectrograms):\n\nE\n\nmin\nW,e\n\nsi\u223cS,\n\n(ti,j ,ai,j )\u223cTsi\n\n{L (f (ti,j, si; W, esi ), ai,j)}\n\n(1)\n\nwhere S is a set of speakers, Tsi is a training set of text-audio pairs for speaker si, and ai,j is the\nground-truth audio for ti,j of speaker si. The expectation is estimated over text-audio pairs of all\n\ntraining speakers. We use cW and be to denote the trained parameters and embeddings. Speaker\n\nembeddings have been shown to effectively capture speaker characteristics with low-dimensional vec-\ntors [Arik et al., 2017b, Ping et al., 2018]. Despite training with only a generative loss, discriminative\nproperties (e.g. gender and accent) are observed in the speaker embedding space [Arik et al., 2017b].\n\nFor voice cloning, we extract the speaker characteristics for an unseen speaker sk from a set of cloning\naudios Ask , and generate an audio given any text for that speaker. The two performance metrics for\nthe generated audio are speech naturalness and speaker similarity (i.e., whether the generated audio\nsounds like it is pronounced by the target speaker). The two approaches for neural voice cloning are\nsummarized in Fig. 1 and explained in the following sections.\n\n3.1 Speaker adaptation\n\nThe idea of speaker adaptation is to \ufb01ne-tune a trained multi-speaker model for an unseen speaker\nusing a few audio-text pairs. Fine-tuning can be applied to either the speaker embedding [Taigman\net al., 2018] or the whole model. For embedding-only adaptation, we have the following objective:\n\nmin\nesk\n\nE(tk,j ,ak,j )\u223cTsk nL(cid:16)f (tk,j , sk;cW , esk ), ak,j(cid:17)o ,\n\n(2)\n\nwhere Tsk is a set of text-audio pairs for the target speaker sk. For whole model adaptation, we have\nthe following objective:\n\nmin\nW,esk\n\nE(tk,j ,ak,j )\u223cTsk\n\n{L (f (tk,j, sk; W, esk ), ak,j)} .\n\n(3)\n\nAlthough the entire model provides more degrees of freedom for speaker adaptation, its optimization\nis challenging for small amount of cloning data. Early stopping is required to avoid over\ufb01tting.\n\n3.2 Speaker encoding\n\nWe propose a speaker encoding method to directly estimate the speaker embedding from audio\nsamples of an unseen speaker. Such a model does not require any \ufb01ne-tuning during voice cloning.\nThus, the same model can be used for all unseen speakers. The speaker encoder, g(Ask ; \u0398), takes a\nset of cloning audio samples Ask and estimates esk for speaker sk. The model is parametrized by\n\u0398. Ideally, the speaker encoder can be jointly trained with the multi-speaker generative model from\nscratch, with a loss function de\ufb01ned for the generated audio:\n\nE\n\nmin\nW,\u0398\n\nsi\u223cS,\n\n(ti,j ,ai,j )\u223cTsi\n\n{L (f (ti,j, si; W, g(Asi ; \u0398)), ai,j)} .\n\n(4)\n\n3\n\n\fTrainable\n\nFixed\n\nSpeaker adaptation\n\nSpeaker encoding\n\nText\n\nMulti-speaker \n\ngenerative model\n\nAudio\n\nText\n\nMulti-speaker \n\ngenerative model\n\nAudio\n\nSpeaker embedding\n\nSpeaker embedding\n\nCloning\n\ntext\n\nMulti-speaker \n\ngenerative model\n\nCloning \naudio\n\nSpeaker embedding\n\nor\n\nCloning\n\ntext\n\nMulti-speaker \n\ngenerative model\n\nCloning \naudio\n\nSpeaker embedding\n\nSpeaker embedding\n\nText\n\nMulti-speaker \n\ngenerative model\n\nAudio\n\nSpeaker encoder \n\nmodel\n\nCloning audio\n\nSpeaker embedding\n\nSpeaker encoder \n\nmodel\n\nCloning audio\n\nSpeaker embedding\n\nSpeaker encoder \n\nmodel\n\nCloning audio\n\nText\n\nMulti-speaker \n\ngenerative model\n\nAudio\n\nText\n\nMulti-speaker \n\ngenerative model\n\nAudio\n\nSpeaker embedding\n\nSpeaker embedding\n\ni\n\ng\nn\nn\na\nr\nT\n\ni\n\ni\n\ng\nn\nn\no\nC\n\nl\n\nn\no\ni\nt\na\nr\ne\nn\ne\ng\no\nd\nu\nA\n\n \n\ni\n\nFigure 1: Illustration of speaker adaptation and speaker encoding approaches for voice cloning.\n\nNote that the speaker encoder is trained with the speakers for the multi-speaker generative model.\nDuring training, a set of cloning audio samples Asi are randomly sampled for training speaker si.\nDuring inference, audio samples from the target speaker sk, Ask , are used to compute g(Ask ; \u0398).\nWe observed optimization challenges with joint training from scratch: the speaker encoder tends\nto estimate an average voice to minimize the overall generative loss. One possible solution is to\nintroduce discriminative loss functions for intermediate embeddings4 or generated audios5. In our\ncase, however, such approaches only slightly improve speaker differences. Instead, we propose a\n\nseparate training procedure for speaker encoder. Speaker embeddingsbesi are extracted from a trained\n\nmulti-speaker generative model f (ti,j, si; W, esi ). Then, the speaker encoder g(Ask ; \u0398) is trained\nwith an L1 loss to predict the embeddings from sampled cloning audios:\n\nmin\n\n\u0398\n\nEsi\u223cS {|g(Asi ; \u0398) \u2212besi )|} .\n\n(5)\n\nEventually, entire model can be jointly \ufb01ne-tuned following Eq. 4, with pre-trained cW and b\u0398 as the\n\ninitialization. Fine-tuning encourages the generative model to compensate for embedding estimation\nerrors, and may reduce attention problems. However, generative loss still dominates learning and\nspeaker differences in generated audios may be slightly reduced as well (see Section 4.3 for details).\n\nMel \n\nspectrograms\n\nPrenet\n\nConv. block\n\nMean pooling\n\nSelf attention\n\nSpeaker \nembedding\n\n\u00d7 Nconv\n\nFigure 2: Speaker encoder architecture. See Appendix A for details.\n\nFor speaker encoder g(Ask ; \u0398), we propose a neural network architecture comprising three parts (as\nshown in Fig. 2):\n\n(i) Spectral processing: We input mel-spectrograms of cloning audio samples to prenet, which\n\ncontains fully-connected layers with exponential linear unit for feature transformation.\n\n(ii) Temporal processing: To utilize long-term context, we use convolutional layers with gated linear\n\nunit and residual connections, average pooling is applied to summarize the whole utterance.\n\n4We have experimented classi\ufb01cation loss by mapping the embeddings to labels via a softmax layer.\n5We have experimented integrating a pre-trained classi\ufb01er to encourage discrimination in generated audios.\n\n4\n\n\f(iii) Cloning sample attention: Considering that different cloning audios contain different amount of\nspeaker information, we use a multi-head self-attention mechanism [Vaswani et al., 2017] to\ncompute the weights for different audios and get aggregated embeddings.\n\n3.3 Discriminative models for evaluation\n\nBesides human evaluations, we propose two evaluation methods using discriminative models for\nvoice cloning performance.\n\n3.3.1 Speaker classi\ufb01cation\n\nSpeaker classi\ufb01er determines which speaker an audio sample belongs to. For voice cloning evaluation,\na speaker classi\ufb01er is trained with the set of speakers used for cloning. High-quality voice cloning\nwould result in high classi\ufb01cation accuracy. The architecture is composed of similar spectral and\ntemporal processing layers in Fig. 6 and an additional embedding layer before the softmax function.\n\n3.3.2 Speaker veri\ufb01cation\n\nSpeaker veri\ufb01cation is the task of authenticating the claimed identity of a speaker, based on a test\naudio and enrolled audios from the speaker. In particular, it performs binary classi\ufb01cation to identify\nwhether the test audio and enrolled audios are from the same speaker [e.g., Snyder et al., 2016].\nWe consider an end-to-end text-independent speaker veri\ufb01cation model [Snyder et al., 2016] (see\nAppendix C for more details of model architecture). The speaker veri\ufb01cation model can be trained on\na multi-speaker dataset, and then used to verify if the cloned audio and the ground-truth audio are\nfrom the same speaker. Unlike the speaker classi\ufb01cation approach, speaker veri\ufb01cation model does\nnot require training with the audios from the target speaker for cloning, hence it can be used for unseen\nspeakers with a few samples. As the quantitative performance metric, the equal error-rate (EER) 6\ncan be used to measure how close the cloned audios are to the ground truth audios.\n\n4 Experiments\n\n4.1 Datasets\n\nIn our \ufb01rst set of experiments (Sections 4.3 and 4.4), the multi-speaker generative model and speaker\nencoder are trained using LibriSpeech dataset [Panayotov et al., 2015], which contains audios\n(16 KHz) for 2484 speakers, totalling 820 hours. LibriSpeech is a dataset for automatic speech\nrecognition, and its audio quality is lower compared to speech synthesis datasets.7 Voice cloning\nis performed on VCTK dataset [Veaux et al., 2017]. VCTK consists of audios sampled at 48 KHz\nfor 108 native speakers of English with various accents. To be consistent with LibriSpeech dataset,\nVCTK audios are downsampled to 16 KHz. For a chosen speaker, a few cloning audios are randomly\nsampled for each experiment. The sentences presented in Appendix B are used to generate audios for\nevaluation. In our second set of experiments (Section 4.5), we aim to investigate the impact of the\ntraining dataset. We split the VCTK dataset for training and testing: 84 speakers are used for training\nthe multi-speaker model, 8 speakers for validation, and 16 speakers for cloning.\n\n4.2 Model speci\ufb01cations\n\nOur multi-speaker generative model is based on the convolutional sequence-to-sequence architecture\nproposed in Ping et al. [2018], with similar hyperparameters and Grif\ufb01n-Lim vocoder. To get better\nperformance, we increase the time-resolution by reducing the hop length and window size parameters\nto 300 and 1200, and add a quadratic loss term to penalize large amplitude components superlinearly.\nFor speaker adaptation experiments, we reduce the embedding dimensionality to 128, as it yields\nless over\ufb01tting problems. Overall, the baseline multi-speaker generative model has around 25M\ntrainable parameters when trained for the LibriSpeech dataset. For the second set of experiments,\nhyperparameters of the VCTK model is used from Ping et al. [2018] to train a multi-speaker model\nfor the 84 speakers of VCTK, with Grif\ufb01n-Lim vocoder.\n\n6EER is the point when the false acceptance rate and false rejection rate are equal.\n7We designed a segmentation and denoising pipeline to process LibriSpeech, as in Ping et al. [2018].\n\n5\n\n\fWe train speaker encoders for different number of cloning audios separately. Initially, cloning audios\nare converted to log-mel spectrograms with 80 frequency bands, with a hop length of 400, a window\nsize of 1600. Log-mel spectrograms are fed to spectral processing layers, which are composed of\n2-layer prenet of size 128. Then, temporal processing is applied with two 1-D convolutional layers\nwith a \ufb01lter width of 12. Finally, multi-head attention is applied with 2 heads and a unit size of 128 for\nkeys, queries and values. The \ufb01nal embedding size is 512. Validation set consists 25 held-out speakers.\nA batch size of 64 is used, with an initial learning rate of 0.0006 with annealing rate of 0.6 applied\nevery 8000 iterations. Mean absolute error for the validation set is shown in Fig. 11 in Appendix\nD. More cloning audios leads to more accurate speaker embedding estimation, especially with the\nattention mechanism (see Appendix D for more details about the learned attention coef\ufb01cients).\n\nWe train a speaker classi\ufb01er using VCTK dataset to classify which of the 108 speakers an audio\nsample belongs to. Speaker classi\ufb01er has a fully-connected layer of size 256, 6 convolutional layers\nwith 256 \ufb01lters of width 4, and a \ufb01nal embedding layer of size 32. The model achieves 100% accuracy\nfor validation set of size 512.\n\nWe train a speaker veri\ufb01cation model using LibriSpeech dataset. Validation sets consists 50 held-out\nspeakers from Librispeech. EERs are estimated by randomly pairing up utterances from the same\nor different speakers (50% for each case) in test set. We perform 40960 trials for each test set. We\ndescribe the details of speaker veri\ufb01cation model in Appendix C.\n\n4.3 Voice cloning performance\n\nApproaches\n\nData\n\nCloning time\nInference time\n\nSpeaker adaptation\n\nSpeaker encoding\n\nEmbedding-only Whole-model Without \ufb01ne-tuning With \ufb01ne-tuning\n\nText and audio\n\nAudio\n\n\u223c 8 hours\n\n\u223c 0.5 \u2212 5 mins\n\n\u223c 1.5 \u2212 3.5 secs\n\n\u223c 1.5 \u2212 3.5 secs\n\n\u223c 0.4 \u2212 0.6 secs\n\nParameters per speaker\n\n128\n\n\u223c 25 million\n\n512\n\n512\n\nTable 1: Comparison of speaker adaptation and speaker encoding approaches.\n\nFor speaker adaptation approach, we pick the optimal number of iterations using speaker classi\ufb01cation\naccuracy. For speaker encoding, we consider voice cloning with and without joint \ufb01ne-tuning of the\nspeaker encoder and multi-speaker generative model.8 Table 1 summarizes the approaches and lists\nthe requirements for training, data, cloning time and memory footprint.\n\nFigure 3: Performance of whole-model adaptation and speaker embedding adaptation for voice\ncloning in terms of speaker classi\ufb01cation accuracy for 108 VCTK speakers.\n\nFor speaker adaptation, Fig. 3 shows the speaker classi\ufb01cation accuracy vs. the number of iterations.\nFor both, the classi\ufb01cation accuracy signi\ufb01cantly increases with more samples, up to ten samples. In\nthe low sample count regime, adapting the speaker embedding is less likely to over\ufb01t the samples\n\n8The learning rate and annealing parameters are optimized for joint \ufb01ne-tuning.\n\n6\n\n\f(a)\n\n(b)\n\nFigure 4: (a) Speaker classi\ufb01cation accuracy with different numbers of cloning samples. (b) EER\n(using 5 enrollment audios) for different numbers of cloning samples. LibriSpeech (unseen speakers)\nand VCTK represent EERs estimated from random pairing of utterances from ground-truth datasets.\n\nthan adapting the whole model. The two methods also require different numbers of iterations to\nconverge. Compared to whole-model adaptation (which converges around 1000 iterations for even\n100 cloning audio samples), embedding adaptation takes signi\ufb01cantly more iterations to converge,\nthus it results in much longer cloning time.\n\nFigs. 4a and 4b show the classi\ufb01cation accuracy and EER, obtained by speaker classi\ufb01cation and\nspeaker veri\ufb01cation models. Both speaker adaptation and speaker encoding bene\ufb01t from more cloning\naudios. When the number of cloning audio samples exceed \ufb01ve, whole-model adaptation outperforms\nother techniques. Speaker encoding yields a lower classi\ufb01cation accuracy compared to embedding\nadaptation, but they achieve a similar speaker veri\ufb01cation performance.\n\nBesides evaluations by discriminative models, we conduct subject tests on Amazon Mechanical\nTurk framework. For assessment of the naturalness, we use the 5-scale mean opinion score (MOS).\nFor assessment of how similar the generated audios are to the ground-truth audios from target\nspeakers, we use the 4-scale similarity score with the question and categories in [Wester et al., 2016].9\nTables 2 and 3 show the results of human evaluations. Higher number of cloning audios improve\nboth metrics. The improvement is more signi\ufb01cant for whole model adaptation, due to the more\ndegrees of freedom provided for an unseen speaker. Indeed, for high sample counts, the naturalness\nsigni\ufb01cantly exceeds the baseline model, due to the dominance of better quality adaptation samples\nover training data. Speaker encoding achieves naturalness similar or better than the baseline model.\nThe naturalness is even further improved with \ufb01ne-tuning since it allows the generative model to\nlearn how to compensate for the errors of the speaker encoder. Similarity scores slightly improve with\nhigher sample counts for speaker encoding, and match the scores for speaker embedding adaptation.\n\nApproach\n\nGround-truth (16 KHz sampling rate)\n\nMulti-speaker generative model\n\nSpeaker adaptation (embedding-only)\n\nSpeaker adaptation (whole-model)\n\nSpeaker encoding (without \ufb01ne-tuning)\n\nSpeaker encoding (with \ufb01ne-tuning)\n\n1\n\n2\n\n2.27\u00b10.10\n2.32\u00b10.10\n2.76\u00b10.10\n2.93\u00b10.10\n\n2.38\u00b10.10\n2.87\u00b10.09\n2.76\u00b10.09\n3.02\u00b10.11\n\nSample count\n\n3\n\n4.66\u00b10.06\n2.61\u00b10.10\n2.43\u00b10.10\n2.98\u00b10.11\n2.78\u00b10.10\n2.97\u00b10.1\n\n5\n\n10\n\n2.46\u00b10.09\n2.67\u00b10.11\n2.75\u00b10.10\n2.93\u00b10.10\n\n2.67\u00b10.10\n3.16\u00b10.09\n2.79\u00b10.10\n2.99\u00b10.12\n\nTable 2: Mean Opinion Score (MOS) evaluations for naturalness with 95% con\ufb01dence intervals\n(training with LibriSpeech speakers and cloning with 108 VCTK speakers).\n\n9We conduct each evaluation independently, so the cloned audios of two different models are not directly\n\ncompared during rating. Multiple votes on the same sample are aggregated by a majority voting rule.\n\n7\n\n1235102050100Number of samples02468101214Equal error rate (in %)LibriSpeech (unseen speakers)VCTKspeaker adaption: embedding-onlyspeaker adaption: whole-modelspeaker encoding: without fine-tuningspeaker encoding: with fine-tuning\fApproach\n\nGround-truth (same speaker)\n\nGround-truth (different speakers)\n\n1\n\n2\n\nSpeaker adaptation (embedding-only)\n\nSpeaker adaptation (whole-model)\n\nSpeaker encoding (without \ufb01ne-tuning)\n\nSpeaker encoding (with \ufb01ne-tuning)\n\n2.66\u00b10.09\n2.59\u00b10.09\n2.48\u00b10.10\n2.59\u00b10.12\n\n2.64\u00b10.09\n2.95\u00b10.09\n2.73\u00b10.10\n2.67\u00b10.12\n\nSample count\n\n3\n\n3.91\u00b10.03\n1.52\u00b10.09\n2.71\u00b10.09\n3.01\u00b10.10\n2.70\u00b10.11\n2.73\u00b10.13\n\n5\n\n10\n\n2.78\u00b10.10\n3.07\u00b10.08\n2.81\u00b10.10\n2.77\u00b10.12\n\n2.95\u00b10.09\n3.16\u00b10.08\n2.85\u00b10.10\n2.77\u00b10.11\n\nTable 3: Similarity score evaluations with 95% con\ufb01dence intervals (training with LibriSpeech\nspeakers and cloning with 108 VCTK speakers).\n\n4.4 Voice morphing via embedding manipulation\n\nAs shown in Fig. 5, speaker encoder maps speakers into a meaningful latent space. Inspired by word\nembedding manipulation (e.g. to demonstrate the existence of simple algebraic operations as king\n- queen = male - female), we apply algebraic operations to inferred embeddings to transform their\nspeech characteristics. To transform gender, we estimate the averaged speaker embeddings for each\ngender, and add their difference to a particular speaker. For example, BritishMale + AveragedFemale -\nAveragedMale yields a British female speaker. Similarly, we consider region of accent transformation\nvia BritishMale + AveragedAmerican - AveragedBritish to obtain an American male speaker. Our\nresults demonstrate high quality audios with speci\ufb01c gender and accent characteristics.10\n\nFigure 5: Visualization of estimated speaker embeddings by speaker encoder. The \ufb01rst two principal\ncomponents of speaker embeddings (averaged across 5 samples for each speaker). Only British and\nNorth American regional accents are shown as they constitute the majority of the labeled speakers in\nthe VCTK dataset. Please see Appendix E for more detailed analysis.\n\n4.5\n\nImpact of training dataset\n\nApproach\n\nSpeaker adaptation (embedding-only)\n\nSpeaker adaptation (whole-model)\n\n3.01\u00b10.11\n2.34\u00b10.13\n\n1\n\n5\n-\n\n2.99\u00b10.10\n\nSample count\n\n10\n\n3.13\u00b10.11\n3.07\u00b10.09\n\n20\n-\n\n3.40\u00b10.10\n\n100\n\n3.13\u00b10.11\n3.38\u00b10.09\n\nTable 4: Mean Opinion Score (MOS) evaluations for naturalness with 95% con\ufb01dence intervals\n(training with 84 VCTK speakers and cloning with 16 VCTK speakers).\n\nApproach\n\nSpeaker adaptation (embedding-only)\n\nSpeaker adaptation (whole-model)\n\n2.42\u00b10.13\n2.55\u00b10.11\n\n1\n\n5\n-\n\n2.93\u00b10.11\n\nSample count\n\n10\n\n2.37\u00b10.13\n2.95\u00b10.10\n\n20\n-\n\n3.01\u00b10.10\n\n100\n\n2.37\u00b10.12\n3.14\u00b10.10\n\nTable 5: Similarity score evaluations with 95% con\ufb01dence intervals (training with 84 VCTK speakers\nand cloning with 16 VCTK speakers).\n\nTo evaluate the impact of the dataset, we consider training with a subset of the VCTK containing\n84 speakers, and cloning on another 16 speakers. Tables 4 and 5 present the human evaluations\n\n10https://audiodemos.github.io/\n\n8\n\n-0.6-0.4-0.20.00.20.40.6-0.2-0.10.00.10.20.3Average North American maleAverage North American femaleAverage British maleAverage British femaleAverage maleAverage female-0.6-0.4-0.20.00.20.40.6-0.2-0.10.00.10.20.3Average North American maleAverage North American femaleAverage British maleAverage British femaleAverage BritishAverage North American\ffor speaker adaptation.11 Speaker veri\ufb01cation results are given in Appendix C. One the one hand,\ncompared to LibriSpeech, cleaner VCTK data improves the multi-speaker generative model, leading\nto better whole-model adaptation results. On the other hand, embedding-only adaptation signi\ufb01cantly\nunderperforms whole-model adaptation due to the limited speaker diversity in VCTK dataset.\n\n5 Conclusions\n\nWe study two approaches for neural voice cloning: speaker adaptation and speaker encoding. We\ndemonstrate that both approaches can achieve good cloning quality even with only a few cloning\naudios. For naturalness, we show that both speaker adaptation and speaker encoding can achieve\nan MOS similar to the baseline multi-speaker generative model. Thus, the proposed techniques can\npotentially be improved with better multi-speaker models in the future (such as replacing Grif\ufb01n-\nLim with WaveNet vocoder). For similarity, we demonstrate that both approaches bene\ufb01t from a\nlarger number of cloning audios. The performance gap between whole-model and embedding-only\nadaptation indicates that some discriminative speaker information still exists in the generative model\nbesides speaker embeddings. The bene\ufb01t of compact representation via embeddings is fast cloning\nand small footprint per speaker. We observe drawbacks of training the multi-speaker generative\nmodel using a speech recognition dataset with low-quality audios and limited speaker diversity.\nImprovements in the quality of dataset would result in higher naturalness. We expect our techniques\nto bene\ufb01t signi\ufb01cantly from a large-scale and high-quality multi-speaker dataset.\n\nReferences\n\nO. Abdel-Hamid and H. Jiang. Fast speaker adaptation of hybrid nn/hmm model for speech recogni-\n\ntion based on discriminative learning of speaker code. In IEEE ICASSP, 2013.\n\nY. Agiomyrgiannakis and Z. Roupakia. Voice morphing that improves tts quality using an optimal\n\ndynamic frequency warping-and-weighting transform. IEEE ICASSP, 2016.\n\nD. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro,\nQ. Cheng, G. Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin.\nIn International Conference on Machine Learning, pages 173\u2013182, 2016.\n\nS. \u00d6. Arik, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li, J. Miller, J. Raiman,\n\nS. Sengupta, and M. Shoeybi. Deep Voice: Real-time neural text-to-speech. In ICML, 2017a.\n\nS. \u00d6. Arik, G. F. Diamos, A. Gibiansky, J. Miller, K. Peng, W. Ping, J. Raiman, and Y. Zhou. Deep\n\nVoice 2: Multi-speaker neural text-to-speech. In NIPS, pages 2966\u20132974, 2017b.\n\nS. Azadi, M. Fisher, V. Kim, Z. Wang, E. Shechtman, and T. Darrell. Multi-content gan for few-shot\n\nfont style transfer. CoRR, abs/1708.02182, 2017.\n\nL. H. Chen, Z. H. Ling, L. J. Liu, and L. R. Dai. Voice conversion using deep neural networks\nwith layer-wise generative training. IEEE/ACM Transactions on Audio, Speech, and Language\nProcessing, 2014.\n\nX. Cui, V. Goel, and G. Saon. Embedding-based speaker adaptive training of deep neural networks.\n\narXiv preprint arXiv:1710.06937, 2017.\n\nS. Desai, A. W. Black, B. Yegnanarayana, and K. Prahallad. Spectral mapping using arti\ufb01cial neural\nnetworks for voice conversion. IEEE Transactions on Audio, Speech, and Language Processing,\n2010.\n\nH. T. Hwang, Y. Tsao, H. M. Wang, Y. R. Wang, and S. H. Chen. A probabilistic interpretation for\narti\ufb01cial neural network-based voice conversion. In 2015 Asia-Paci\ufb01c Signal and Information\nProcessing Association Annual Summit and Conference (APSIPA), 2015.\n\nR. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the limits of language\n\nmodeling. arXiv preprint arXiv:1602.02410, 2016.\n\n11The speaker encoder models generalize poorly for unseen speakers due to limited training speakers.\n\n9\n\n\fT. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of gans for improved quality,\n\nstability, and variation. CoRR, abs/1710.10196, 2017.\n\nB. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. One-shot learning by inverting a compositional\n\ncausal process. In NIPS, 2013.\n\nB. M. Lake, C. ying Lee, J. R. Glass, and J. B. Tenenbaum. One-shot learning of generative speech\n\nconcepts. In CogSci, 2014.\n\nB. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba-\n\nbilistic program induction. Science, 2015.\n\nX. Li and X. Wu. Modeling speaker variability using long short-term memory networks for speech\n\nrecognition. In INTERSPEECH, 2015.\n\nS. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio. Samplernn:\nAn unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837,\n2016.\n\nY. Miao and F. Metze. On speaker adaptation of long short-term memory recurrent neural networks.\n\nIn Sixteenth Annual Conference of the International Speech Communication Association, 2015.\n\nY. Miao, H. Zhang, and F. Metze. Speaker adaptive training of deep neural network acoustic models\n\nusing i-vectors. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015.\n\nA. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner,\nA. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint\narXiv:1609.03499, 2016a.\n\nA. v. d. Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image\ngeneration with pixelcnn decoders. In Advances in Neural Information Processing Systems, 2016b.\n\nV. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an ASR corpus based on public\n\ndomain audio books. In IEEE ICASSP, 2015.\n\nW. Ping, K. Peng, A. Gibiansky, S. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller. Deep Voice\n\n3: Scaling text-to-speech with convolutional sequence learning. In ICLR, 2018.\n\nS. Prince and J. Elder. Probabilistic linear discriminant analysis for inferences about identity. In\n\nICCV, 2007.\n\nS. E. Reed, Y. Chen, T. Paine, A. van den Oord, S. M. A. Eslami, D. J. Rezende, O. Vinyals, and\nN. de Freitas. Few-shot autoregressive density estimation: Towards learning to learn distributions.\nCoRR, 2017.\n\nD. Rezende, Shakir, I. Danihelka, K. Gregor, and D. Wierstra. One-shot generalization in deep\n\ngenerative models. In ICML, 2016.\n\nJ. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-\nRyan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. arXiv\npreprint arXiv:1712.05884, 2017.\n\nD. Snyder, P. Ghahremani, D. Povey, D. Garcia-Romero, Y. Carmiel, and S. Khudanpur. Deep neural\nnetwork-based speaker embeddings for end-to-end speaker veri\ufb01cation. In IEEE Spoken Language\nTechnology Workshop (SLT), pages 165\u2013170, 2016.\n\nJ. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. Courville, and Y. Bengio. Char2wav:\n\nEnd-to-end speech synthesis. 2017.\n\nY. Taigman, L. Wolf, A. Polyak, and E. Nachmani. Voiceloop: Voice \ufb01tting and synthesis via a\n\nphonological loop. In ICLR, 2018.\n\nA. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.\n\nAttention is all you need. In NIPS. 2017.\n\n10\n\n\fC. Veaux, J. Yamagishi, and K. e. a. MacDonald. Cstr vctk corpus: English multi-speaker corpus for\n\ncstr voice cloning toolkit, 2017.\n\nY. Wang, R. J. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen,\nS. Bengio, Q. V. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous. Tacotron: A fully\nend-to-end text-to-speech synthesis model. CoRR, abs/1703.10135, 2017.\n\nM. Wester, Z. Wu, and J. Yamagishi. Analysis of the voice conversion challenge 2016 evaluation\n\nresults. In INTERSPEECH, pages 1637\u20131641, 09 2016.\n\nY.-C. Wu, H.-T. Hwang, C.-C. Hsu, Y. Tsao, and H.-m. Wang. Locally linear embedding for\n\nexemplar-based spectral conversion. In INTERSPEECH, pages 1652\u20131656, 09 2016.\n\nZ. Wu, P. Swietojanski, C. Veaux, S. Renals, and S. King. A study of speaker adaptation for dnn-based\n\nspeech synthesis. In INTERSPEECH, 2015.\n\nS. Xue, O. Abdel-Hamid, H. Jiang, L. Dai, and Q. Liu. Fast adaptation of deep neural network based\non discriminant codes for speech recognition. IEEE/ACM Transactions on Audio, Speech, and\nLanguage Processing, 2014.\n\nJ. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Isogai. Analysis of speaker adaptation\nalgorithms for hmm-based speech synthesis and a constrained smaplr adaptation algorithm. IEEE\nTransactions on Audio, Speech, and Language Processing, 2009.\n\nD. Yu, K. Yao, H. Su, G. Li, and F. Seide. Kl-divergence regularized deep neural network adaptation\n\nfor improved large vocabulary speech recognition. In IEEE ICASSP, 2013.\n\n11\n\n\f", "award": [], "sourceid": 6483, "authors": [{"given_name": "Sercan", "family_name": "Arik", "institution": "Google"}, {"given_name": "Jitong", "family_name": "Chen", "institution": "ByteDance"}, {"given_name": "Kainan", "family_name": "Peng", "institution": "Baidu Research"}, {"given_name": "Wei", "family_name": "Ping", "institution": "Baidu Silicon Valley AI Lab"}, {"given_name": "Yanqi", "family_name": "Zhou", "institution": "Baidu Research"}]}