{"title": "Re-examination of the Role of Latent Variables in Sequence Modeling", "book": "Advances in Neural Information Processing Systems", "page_first": 7814, "page_last": 7824, "abstract": "With latent variables, stochastic recurrent models have achieved state-of-the-art performance in modeling sound-wave sequence.\nHowever, opposite results are also observed in other domains, where standard recurrent networks often outperform stochastic models.\nTo better understand this discrepancy, we re-examine the roles of latent variables in stochastic recurrent models for speech density estimation.\nOur analysis reveals that under the restriction of fully factorized output distribution in previous evaluations, the stochastic variants were implicitly leveraging intra-step correlation but the deterministic recurrent baselines were prohibited to do so, resulting in an unfair comparison.\nTo correct the unfairness, we remove such restriction in our re-examination, where all the models can explicitly leverage intra-step correlation with an auto-regressive structure.\nOver a diverse set of univariate and multivariate sequential data, including human speech, MIDI music, handwriting trajectory, and frame-permuted speech, our results show that stochastic recurrent models fail to deliver the performance advantage claimed in previous work. \n%exhibit any practical advantage despite the claimed theoretical superiority. \nIn contrast, standard recurrent models equipped with an auto-regressive output distribution consistently perform better, dramatically advancing the state-of-the-art results on three speech datasets.", "full_text": "Re-examination of the Role of Latent Variables in\n\nSequence Modeling\n\nGuokun Lai\u21e41, Zihang Dai\u21e41, Yiming Yang1, Shinjae Yoo2\n1Carnegie Mellon University, 2Brookhaven National Laboratory\n1{guokun,dzihang,yiming}@cs.cmu.edu, 2sjyoo@bnl.gov\n\nAbstract\n\nWith latent variables, stochastic recurrent models have achieved state-of-the-art\nperformance in modeling sound-wave sequence. However, opposite results are also\nobserved in other domains, where standard recurrent networks often outperform\nstochastic models. To better understand this discrepancy, we re-examine the roles\nof latent variables in stochastic recurrent models for speech density estimation.\nOur analysis reveals that under the restriction of fully factorized output distribution\nin previous evaluations, the stochastic variants were implicitly leveraging intra-\nstep correlation but the deterministic recurrent baselines were prohibited to do\nso, resulting in an unfair comparison. To correct the unfairness, we remove such\nrestriction in our re-examination, where all the models can explicitly leverage\nintra-step correlation with an auto-regressive structure. Over a diverse set of\nunivariate and multivariate sequential data, including human speech, MIDI music,\nhandwriting trajectory and frame-permuted speech, our results show that stochastic\nrecurrent models fail to deliver the performance advantage claimed in previous\nwork. In contrast, standard recurrent models equipped with an auto-regressive\noutput distribution consistently perform better, dramatically advancing the state-of-\nthe-art results on three speech datasets.\n\n1 Introduction\n\nAs a fundamental problem in machine learning, probabilistic sequence modeling aims at capturing\nthe sequential correlations in both short and long ranges. Among many possible model choices, deep\nauto-regressive models [1, 2] have become one of the most widely adopted solutions. Typically,\na deep auto-regressive model factorizes the likelihood function of sequences in an auto-regressive\n\nmanner, i.e., p(x) =Q|x|t=1 p(xt | x<t). Then, a neural network (e.g. RNN) is employed to encode\nthe conditional context x<t into a compact hidden representation ht = f (x<t), which is then used to\nde\ufb01ne the output distribution p(xt | x<t) , p(xt | ht).\nDespite the state-of-the-art (SOTA) performance in many domains [3, 4, 5, 6], the hidden representa-\ntions of standard auto-regressive models are produced in a completely deterministic way. Hence, the\nstochastic aspects of the observed sequences can only be modeled by the output distribution, which\nhowever, usually has a simple parametric form such as a unimodal distribution or a \ufb01nite mixture of\nunimodal distributions. A potential weakness of such simple forms is that they may not be suf\ufb01ciently\nexpressive for modeling real-world sequential data with complex stochastic dynamics.\nRecently, many efforts have been made to enrich the expressive power of auto-regressive models\nby injecting stochastic latent variables into the computation of hidden states. Notably, relying\non the variational auto-encoding (VAE) framework [7, 8], stochastic recurrent models (SRNN)\nhave outperformed standard RNN-based auto-regressive models by a large margin in modeling raw\nsound-wave sequences [9, 10, 11, 12, 13, 14].\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fHowever, the success of stochastic latent variables does not necessarily generalize to other domains\nsuch as text and images. For instance, the authors [12] report that an SRNN trained by Z-Forcing\nlags behind a baseline RNN in language modeling. Similarly, for the density estimation of natural\nimages, PixelCNN [15, 16, 17] consistently outperforms generative models with latent variables [18,\n19, 20, 21, 22].\nTo better understand the discrepancy, we perform a re-examination on the role of stochastic variables\nin SRNN models. By carefully inspecting of the previous experiment settings for sound-wave density\nestimation, and systematically analyzing the properties of SRNN, we identify two potential causes\nof the performance gap between SRNN and RNN. Controlled experiments are designed to test\neach hypothesis, where we \ufb01nd that previous evaluations impose an unnecessary restriction of fully\nfactorized output distributions, which has led to an unfair comparison between SRNN and RNN.\nSpeci\ufb01cally, under the factorized parameterization, SRNN can still implicitly leverage the intra-step\ncorrelation, i.e., the simultaneity [23], while the RNN baselines are prohibited to do so. Meanwhile,\nwe also observe that the posterior learned by SRNN can get outperformed by a simple hand-crafted\nposterior, raising serious doubt about the general effectiveness of injecting latent variables.\nTo provide a fair comparison, we propose an evaluation setting where both the SRNN and RNN can\nutilize an auto-regressive output distribution to model the intra-step correlation explicitly. Under the\nnew setting, we re-evaluate SRNN and RNN on a diverse collection of sequential data, including\nhuman speech, MIDI music, handwriting trajectory and frame-permuted speech. Empirically, we \ufb01nd\nthat sequential models with continuous latent variables fail to offer any practical bene\ufb01ts, despite their\nwidely believed theoretical superiority. On the contrary, explicitly capturing the intra-step correlation\nwith an auto-regressive output distribution consistently performs better, substantially improving the\nSOTA performances in modeling speech signals. Overall, these observations show that the previously\nreported performance \u201cadvantage\u201d of SRNN is merely the result of a long-existing experiment bias\nof using factorized output distributions.\n\n2 Background\n\nIn this section, we brie\ufb02y review SRNN and RNN for probabilistic sequence modeling. The other\napproaches are summarized in appendix E. Throughout the paper, we will use bold font x to denote a\nsequence, x<t and x\uf8fft to indicate the sub-sequence of \ufb01rst t  1 and t elements respectively, and xt\nto represent the t-th element. Note that xt can either be a scalar or a multivariate vector. In the latter\ncase, xt,i denotes the i-th element of the vector xt.\nGiven a set of sequences D =x1, x2,\u00b7\u00b7\u00b7 , x|D| , we are interested in building a density estimation\n\nmodel for sequences. A widely adapted solution is to employ an auto-regressive model powered by a\nneural network, and utilize MLE to perform the training:\n\n\u2713 LD = E\nmax\n\nx\u21e0D\" TxXt=1\n\nlog p\u2713(xt | x<t)#,\n\n(1)\n\nht;\n\nCategorical or Gaussian Mixture;\n\nwhere Tx is the length of the sequence x. More concretely, the conditional distribution p\u2713(xt | x<t)\nis usually jointly modeled by two sub-modules:\n\u2022 The pre-de\ufb01ned distribution family of the output distribution p\u2713(xt | x<t), such as a Gaussian,\n\u2022 The sequence model f\u2713, which encodes the contextual sequence x<t into a compact hidden vector\nUnder this general framework, RNN and SRNN can be seen as two different instantiations of\nthe sequence model. As we have discussed in Section 1, the computation inside RNN is fully\ndeterministic.\nTo improve the model expressiveness, SRNN takes an alternative route and incorporates continuous\nlatent variables into the sequence model. Typically, SRNN associates the observed data sequence\nx with a sequence of latent variables z = [z1, . . . , zTx], one for each step. With latent variables,\nthe internal dynamics of the sequence model is not deterministic any more, offering a theoretical\npossibility to capture more complex stochastic patterns. However, the improved capacity comes with\n\n2\n\n\fa computational burden \u2014 the log-likelihood is generally intractable due to the integral:\n\nLSRNN\nD\n\n= E\n\nx\u21e0D\uf8fflogZ p\u2713(x, z)dz.\n\nHence, standard MLE training cannot be performed.\nTo handle the intractability, SRNN utilizes the VAE framework and maximizes the evidence lower\nbound (ELBO) of the log-likelihood (1) for training:\n\n\u2713, FD = E\nmax\n\nx\u21e0D\" E\n\nq(z|x) TxXt=1\n\nlog\n\np\u2713(xt | z\uf8fft, x<t)p\u2713(zt|z<t, x<t)\n\nq(zt | z<t, x)\n\n!# \uf8ffL SRNN\n\nD\n\n,\n\n(2)\n\nwhere q(z | x) is the approximate posterior distribution modeled by an encoder network. Compu-\ntationally, several SRNN variants have been proposed [9, 10, 11, 12], mostly differing in how the\ngenerative distribution p\u2713(x, z) and the variational posterior q(z | x) are parameterized. In this\nwork, we follow the parameterization and optimization in Z-forcing SRNN method [12], which is\nthe one with the best performance. We include a detailed introduction for related SRNN models in\nappendix B.\n\n3 Revisiting SRNN for Speech Modeling\n\n3.1 Previous Setting for Speech Density Estimation\n\nTo compare SRNN and RNN, previous studies largely rely on the density estimation of sound-wave\nsequences. Usually, a sound-wave dataset consists of a collection of audio sequences with a sample\nrate of 16Hz, where each frame (element) of the sequence is a scalar in [1, 1], representing the\nnormalized amplitude of the sound. Instead of treating each frame as a single step, the authors\n[10] propose a multi-frame setting, where every 200 consecutive frames are taken as a single step.\nEffectively, the data can be viewed as a sequence of 200-dimensional real-valued vectors, i.e.,\nxt 2 RL with L = 200. During training, every T = 40 steps (8,000 frames) are taken as an i.i.d.\nsequence to form the training set.\nUnder this data format, notice that the output distributions p\u2713(xt | x<t) and p\u2713(xt | z\uf8fft, x<t) now\ncorrespond to an L-dimensional random vector xt. Therefore, how to parameterize this multivariate\ndistribution can largely in\ufb02uence empirical performance. That said, recent approaches [11, 12] have\nall followed [10] to employ a fully factorized parametric form which ignores the inner dependency:\n\np\u2713(xt | x) \u21e1\n\np\u2713(xt | z\uf8fft, x<t) \u21e1\n\nLYi=1\nLYi=1\n\np\u2713(xt,i | x<t),\n\np\u2713(xt,i | z\uf8fft, x<t).\n\n(3)\n\n(4)\n\nHere, we have used the \u21e1 to emphasize this choice effectively poses an independent assumption.\nDespite this convenience, note that the restriction of a fully factorized form is not necessary at all.\nNevertheless, we will refer to the models in Eqn. (3) and Eqn. (4), respectively, as factorized RNN\n(F-RNN) and factorized SRNN (F-SRNN) in the following.\nTo provide a baseline for further discussion, we replicate the experiments under the setting introduced\nabove and evaluate them on three speech datasets, namely TIMIT, VCTK, and Blizzard. Following the\nprevious work [10], we choose a Gaussian mixture to model the per-frame distribution p\u2713(xt,i | x<t)\nof F-RNN, which enables a basic multi-modality.\nWe report the averaged test log-likelihood in Table 1. For consistency with previous results in the\nliterature, the results of TIMIT and Blizzard are based on sequence-level average, while the result of\nVCTK is frame-level average. As we can see, similar to previous observations, F-SRNN outperforms\nF-RNN on all three datasets by a dramatic margin.\n\n3\n\n\fTIMIT VCTK Blizzard\nModels\n7,610\nF-RNN\n32,745\nF-SRNN 69,296\n15,258\n\n0.786\n2.383\n\nTable 1: Performance comparison on three benchmark datasets.\n\n3.2 Decomposing the Advantages of Factorized SRNN\n\nTo understand why the F-SRNN outperforms F-RNN by such a large margin, it is helpful to examine\nthe effective output distribution p\u2713(xt | x<t) of F-SRNN after marginalizing out the latent variables:\n(5)\n\np\u2713(xt,i| z\uf8fft, x<t)dz\uf8fft.\n\np\u2713(xt| x<t) =Z p\u2713(z\uf8fft| x<t)\n\nLYi=1\n\nFrom this particular form, we can see two potential causes of the performance gap between F-SRNN\nand F-RNN in the multi-frame setting:\n\u2022 Advantage under High Volatility: By incorporating the continuous latent variable, the distribu-\ntion p\u2713(xt | x<t) of F-SRNN essentially forms an in\ufb01nite mixture of simpler distributions (see\n\ufb01rst line of Eqn. (5)). As a result, the distribution is signi\ufb01cantly more expressive and \ufb02exible, and\nit is believed to be particularly suitable for modeling high-entropy sequential dynamics [10].\nThe multi-frame setting introduced above well matches this description. Concretely, since the\nmodel is required to predict the next L frames all together in this setting, the long prediction horizon\nwill naturally involve a higher uncertainty. Therefore, the high volatility of the multi-frame setting\nmay provide a perfect scenario for SRNN to exhibit its theoretical advantage in expressiveness.\n\u2022 Utilizing the Intra-Step Correlation: From Eqn. (5), notice that the distribution p\u2713(xt | x<t)\nafter marginalization is generally not factorized any more, due to the coupling with z. In con-\ntrast, recall the same distribution of the F-RNN (Eqn. (3)) is fully factorized p\u2713(xt | x<t) =\nQL\ni=1 p\u2713(xt,i | x<t). Therefore, in theory, a factorized SRNN could still model the correlation\namong the L frames within each step, if properly trained, while the factorized RNN has no means\nto do so at all. Thus, SRNN may also bene\ufb01t from this difference.\n\nWhile both advantages could have jointly led to the performance gap in Table 1, the implications\nare totally different. The \ufb01rst advantage under high volatility is a unique property of latent-variable\nmodels that other generative models without latent variables can hardly to obtain. Therefore, if this\nproperty signi\ufb01cantly contributes to the superior performance of F-SRNN over F-RNN, it suggests\nmore general effectiveness of incorporating stochastic latent variables.\nQuite the contrary, being able to utilize the intra-step correlation is more like an unfair bene\ufb01t to\nSRNN, since it is the unnecessary restriction of fully factorized output distributions in previous\nexperimental design that prevents RNNs from modeling the correlation. In practice, one can easily\nenable RNNs to do so by employing a non-factorized output distribution. In this case, it remains\nunclear whether this particular advantage will sustain. Motivated by the distinct implications, in the\nsequel, we will try to \ufb01gure out how much each of the two hypotheses above actually contributes to\nthe performance gap.\n\n3.3 Advantage under High Volatility\n\nIn order to test the advantage of F-SRNN in modeling high-volatile data in isolation, the idea is to\nconstruct a sequential dataset where each step consists of a single frame (i.e., a uni-variate variable),\nwhile there exists high volatility between every two consecutive steps.\nConcretely, for each sequence x 2D , we create a sub-sequence by selecting one frame from every\nM consecutive frames, i.e., \u02c6x = [x1, xM +1, x2M +1, . . .] with xt 2 R. Intuitively, a larger stride M\nwill lead to a longer horizon between two selected frames and hence a higher uncertainty. Moreover,\nsince each step corresponds to a single scalar, the second advantage (i.e., the potential confounding\nfactor) automatically disappears.\nFollowing this idea, from the original datasets, we derive the stride-TIMIT, stride-VCTK and stride-\nBlizzard with different stride values M, and evaluate the RNN and SRNN on each of them. Again,\nwe report the sequence- or frame-average test likelihood in Table 2.\n\n4\n\n\fStride = 50\n\nStride = 200\n\nModel TIMIT VCTK Blizzard TIMIT VCTK Blizzard\nRNN 20,655\nSRNN 14,469\n\n4,124\n0.177\n-1,137 0.0187\n\n-320\n-1,231\n\n4,607\n3,603\n\n0.668\n0.605\n\nTable 2: Performance comparison on high-volatility datasets.\n\nSurprisingly, RNN consistently achieves a better performance than SRNN in this setting. It suggests\nthe theoretically better expressiveness of SRNN does not help that much in high-volatility scenarios.\nHence, this potential advantage does not really contribute to the performance gap observed in Table 1.\n\n3.4 Utilizing the Intra-Step Correlation\nAfter ruling out the \ufb01rst hypothesis, it becomes more likely that being able to utilize the intra-step\ncorrelation actually leads to the superior performance of F-SRNN. However, despite the non-factorized\nform in Eqn. (5), it is still not clear how F-SRNN computationally captures the correlation in practice.\nHere, we provide a particular possibility.\nRecall that in ELBO function of SRNN method (Eqn. (2)), the vector xt, we hope to reconstruct at\nstep t, is included in the conditional input to the posterior q(zt | z<t, x). With this computational\nstructure, the encoder could theoretically leak a subset of the vector xt into the latent variable zt, and\nleverage the leaked subset to predict (reconstruct) the rest elements in xt. Intuitively, the procedure\nof using the leaked subset to predict the remained subset is essentially exploiting the dependency\nbetween the two subsets, or in other words, the correlation within xt.\nProposition 1. Given a vector xt, we split its elements into two arbitrary disjoint subsets, the leaked\nsubset xa\nt . Assume that the latent variables and leaked subset have\nthe same dimensionality, |zt| = |xa\n\nt |. De\ufb01ne the posterior distribution as a delta function:\n\nt and its complement xb\n\nt = xt\\xa\n\nq(zt | z<t, x) = zt=xa\n\nt =\u21e21,\n\n0,\n\nif zt = xa\nt\notherwise ,\n\nWe further assume p\u2713(xt | z\uf8fft, x<t) \u21e1 p\u2713(xt | zt, x<t). The ELBO function (Eqn. (2)) would\nreduce to a special case of auto-regressive factorization:\n\n(6)\n\n(7)\n\n\u2713 LD = E\nmax\n\nx\u21e0D\" TxXt=1hlog p\u2713(xa\n\nt | x<t) + log p\u2713(xb\n\nt | xa\n\nt , x<t)i#.\n\nt to predict xb\n\nThis proposition can be proved by substituting the posterior distribution into the ELBO function and\nthe detail derivation is provided in the supplementary material. Now, the second term in Eqn. (7)\nis conditioned on the leaked subset of xa\nt, which is exactly utilizing the correlation\nbetween the two subsets. In other words, with a proper posterior, F-SRNN can recover a certain\nauto-regressive parameterization, making it possible to utilize the intra-step correlation, even with a\nfully factorized output distribution.\nAlthough the analysis and construction above provide a theoretical possibility, we still lack concrete\nevidence to support the hypothesis that F-SRNN has signi\ufb01cantly bene\ufb01ted from modeling the\nintra-step correlation. While it is dif\ufb01cult to verify this hypothesis in general, we can parameterize an\nRNN according to Eqn. (7), which is equivalent to an F-SRNN with a delta posterior. Therefore, by\nmeasuring the performance of this special RNN, we can get a conservative estimate of how much\nmodeling the intra-step correlation can contribute to the performance of F-SRNN.\nTo \ufb01nish the special RNN idea, we still need to specify how xt is split into xa\nconsider two methods with different intuitions:\n\u2022 Interleaving: The \ufb01rst method takes one out of every U elements to construct xa\n\nt =\nt. In the\nt the even ones. Hence, when\nt, the output distribution is conditioned on both the elements\n\u2022 Random: The second method simply uniformly selects V random elements from xt to form\nt. Intuitively, this can be viewed as an informal \u201clower bound\u201d of\n\n{xt,1, xt,U +1, xt,2U +1, . . .}. Essentially, this method interleaves the two subsets xa\nextreme case of U = 2, xa\npredicting an even element xt,2k 2 xb\nto the left xt,2k1 and to the right xt,2k+1, making the problem much easier.\nt , and leaves the rest for xb\nxa\nperformance gain through modeling the intra-step correlation.\n\nt includes the odd elements of xt and xb\n\nt. Here, we\n\nt and xb\n\nt and xb\n\n5\n\n\fModels\nF-RNN\nF-SRNN\n-RNN (U = 2)\n-RNN (U = 3)\n-RNN (V = 50)\n-RNN (V = 75)\n\nTIMIT VCTK Blizzard\n7,610\n32,745\n15,258\n69,296\n15,306\n70,900\n72,067\n15,284\n14,389\n66,122\n66,453\n14,585\n\n0.786\n2.383\n2.027\n2.262\n2.199\n2.120\n\nTable 3: Performance comparison between -RNN and F-SRNN. Note that a smaller U corresponds\nto leaking more elements.\n\nSince the parametric form Eqn. (7) is derived from a delta posterior, we will refer to the special\nRNN model as -RNN. Based on the two split methods, we train -RNN on TIMIT, VCTK and\nBlizzard with different values of U and V . The results are summarized in Table 3. As we can see,\nwhen the interleaving split scheme is used, -RNN signi\ufb01cantly improves upon F-RNN and becomes\nvery competitive with F-SRNN. Speci\ufb01cally, on TIMIT and Blizzard, -RNN can even outperform\nF-SRNN in certain cases. More surprisingly, the -RNN with the random-copy scheme can also\nachieve a performance that is very close to that of F-SRNN, especially compared to F-RNN.\nRecall that -RNN is equivalent to employing a manually designed delta posterior that can only copy\nbut never compresses (auto-encodes) the information in xt. As a result, compared to a posterior that\ncan learn to compress information, the delta posterior will involve a higher KL cost when leaking\ninformation through the posterior. Furthermore, the correlation between historical latent variables\nand outputs is ignored. It would decrease the model capacity of -RNN. Despite these disadvantages,\n-RNN is still able to match or even surpasses the performance of F-SRNN, suggesting the learned\nposterior in F-SRNN is far from satisfying. Quite contrary to that, the limited performance gap\nbetween F-SRNN and the random copy baseline raises a serious concern about the effectiveness of\ncurrent variational inference techniques.\nNevertheless, putting the analysis and empirical evidence together, we can conclude that the perfor-\nmance advantage of F-SRNN in the multi-frame setting can be entirely attributed to the second cause.\nThat is, under the factorized constraint in previous experiments, F-SRNN can still implicitly leverage\nthe intra-step correlation, while F-RNN is prohibited to do so. However, as we have discussed earlier\nin Section 3.2, this is essentially an unfair comparison. More importantly, the claimed superiority\nof SRNN over RNN may be misleading, as it is unclear whether performance advantage of SRNN\nwill sustain or not when a non-factorized output distribution is employed to capture the intra-step\ncorrelation explicitly.\nAs far as we know, no previous work has carefully compared the performance of SRNN and RNN\nwhen non-factorized output distribution is allowed. On the other hand, as shown in Table 3, by\nmodeling the multivariate simultaneity in the simplest way, -RNN can achieve dramatic performance\nimprovement. Motivated by the huge potential as well as the lack of a systematic study, we will next\ninclude non-factorized output distributions in our consideration, and properly re-evaluate SRNN and\nRNN for multivariate sequence modeling.\n\n4 Proper Multivariate Sequence Modeling with or without Latent Variables\n\n4.1 Avoiding the Implicit Data Bias\nIn this section, we aim to eliminate any experimental bias and provide a proper evaluation of SRNN\nand RNN for multivariate sequence modeling. Apart from the \u201cmodel bias\u201d of employing fully\nfactorized output distributions we have discussed, another possible source of bias is actually the\nexperimental data. For example, as we discussed in Section 3.1, the multi-frame speech sequences are\nconstructed by reshaping L consecutive real-valued frames into L-dimensional vectors. Consequently,\nelements within each step xt are simply temporally correlated with a natural order, which would\nfavor a model that recurrently process each element from xt,1 to xt,L with parameter sharing.\nThus, to avoid such \u201cdata bias\u201d, besides speech sequences, we additionally consider three more\ntypes of multivariate sequences with different patterns of intra-step correlation, they are MIDI sound\nsequence data (including Muse and Nottingham datasets), handwriting trajectory data (IAM-OnDB)\n\n6\n\n\fModels\nVRNN\u2020\nSRNN\u2020\nZ-Forcing\u2020\nSWaveNet\u2020\u2021\nSTCN\u2020\u2021\nF-RNN\nF-SRNN\n-RNN-random\n\nRNN-\ufb02at\nSRNN-\ufb02at\nRNN-hier\nSRNN-hier\n\nTIMIT\n28,982\n60,550\n70,469\n72,463\n77,438\n32,745\n69,296\n66,453\n117,721?\n109,284\n109,641\n107,912\n\nVCTK Blizzard Muse\n\nNottingham IAM-OnDB Perm-TIMIT\n\n-\n-\n-\n-\n-\n\n0.786\n2.383\n2.199\n3.2173?\n3.2062\n3.1822\n3.1423\n\n9,392\n11,991\n15,430\n15,708\n17,670\n7,610\n15,258\n14,585\n22,714?\n22,290\n21,950\n21,845\n\n-6.28\n\n-\n\n-\n-\n-\n\n-2.94\n\n-\n\n-\n-\n-\n\n-6.991\n-6.438\n-6.252\n\n-5.251\n-5.616\n-5.161\n-5.483\n\n-3.400\n-2.811\n-2.834\n\n-2.180\n-2.324\n-2.028\n-2.065\n\n1384\n\n-\n-\n\n1301\n1796\n1397\n1402\nN/A\n\nN/A\nN/A\n1440\n1395\n\n-\n-\n-\n-\n-\n\n25,679\n67,613\n61,103\n\n15,763\n14,278\n95,161\n94,402\n\nTable 4: Performance comparison on a diverse set of datasets. The models with \u2020 indicate that the\nperformances are directly copied from previous publications. Numbers with ? indicate the state-of-\nthe-art performances. N/A suggests the model is not application on the dataset. The models with \u2021\nhave other architectures than recurrent neural network as the backbone.\nand the Perm-TIMIT dataset. The Perm-TIMIT is a variant of multivariate TIMIT dataset. It permutes\nthe elements within each time step, which is designed to remove the temporal bias. We include the\ndetail information of these datasets in Appendix C.\n\n4.2 Modeling Simultaneity with Auto-Regressive Decomposition\n\nWith proper datasets, we now consider how to construct a family of non-factorized distributions that\n(1) can be easily integrated into RNN and SRNN as the output distribution, and (2) are reasonably\nexpressive for modeling multivariate correlations. Among many possible choices, the most straight-\nforward choice would be the auto-regressive parameterization. Compared to other options such as the\nnormalizing \ufb02ow or Markov Random Field (e.g. RBM), the auto-regressive structure is conceptually\nsimpler and can be applied to both discrete and continuous data with full tractability. In light of these\nbene\ufb01ts, we choose to follow this simple idea, and decompose the output distribution of the RNN and\nSRNN, respectively, as\n\np\u2713(xt | x<t) =\n\np\u2713(xt | z\uf8fft, x<t) =\n\np\u2713(xt,i | x<t, xt,<i),\n\np\u2713(xt,i | z\uf8fft, x<t, xt,<i).\n\n(8)\n\n(9)\n\nLYi=1\nLYi=1\n\nNotice that although we use the natural decomposition order from smallest index to largest one, this\nparticular order is generally not optimal for modeling multivariate distributions. A better choice\ncould be adapting the orderless training previously explored in literature [2]. But for simplicity, we\nwill stick to this simple approach.\nGiven the auto-regressive decomposition, a natural neural instantiation would be a recurrent hierar-\nchical model that utilizes a two-level architecture to process the sequence:\n\u2022 Firstly, a high-level RNN or SRNN is employed to encode the multivariate steps x = [x1, . . . , xT ]\ninto a sequence of high-level hidden vectors h = [h1, . . . , hT ], which follows exactly the same as\nthe computational procedure used in F-RNN and F-SRNN . Recall that, in the case of SRNN, the\ncomputation of high-level vectors involves sampling the latent variables.\n\n\u2022 Based on the high-level representations, for each multivariate step xt, another neural model flow will\ntake both the elements [xt,1,\u00b7\u00b7\u00b7 , xt,L] and the high-level vector ht as input, and auto-regressively\nproduce a sequence of low-level hidden vectors [gt,1,\u00b7\u00b7\u00b7 , gt,L] where gt,i = flow(xt,<i, ht). They\ncan be then used to form the per-element output distributions in Eqn. (8) and (9).\nIn practice, the low-level model could simply be an RNN or a causally masked MLP [24], depending\non our prior about the data. For convenience, we will refer to the hierarchical models as RNN-hier\nand SRNN-hier.\n\n7\n\n\fIn some cases where all the elements within a step share the same statistical type, such as on the\nspeech or MIDI dataset, one may alternatively consider a \ufb02at model. As the name suggests, the \ufb02at\nmodel will break the boundary between steps and \ufb02atten the data into a new uni-variate sequence,\nwhere each step is simply a single element. Then, the new uni-variate sequence can be directly fed\ninto a standard RNN or SRNN model, producing each conditional factor in Eqn. (8) and (9) in an\nauto-regressive manner. Similarly, this class of RNN and SRNN will be referred to as RNN-\ufb02at and\nSRNN-\ufb02at, respectively. Compared to the hierarchical model, the \ufb02at variant implicitly assumes a\nsequential continuity between xt,L and xt+1,1. Since this inductive bias matches the characteristics\nof multi-frame speech sequences, we expect the \ufb02at model to perform better in this case.\n\n4.3 Experiment Results\n\nBased on the seven datasets, we compare the performance of the models introduced above. To provide\na random baseline, we include the -RNN with the random split scheme in the comparison. Moreover,\nprevious results, if exist, are also presented to provide additional information. For a fair comparison,\nwe make sure all models share the same parameter size. For more implementation details, please refer\nto the Supplementary D as well as the source code1. We also include the running time comparison\nin Appendix 4.4. Finally, the results are summarized in Table 4, where we make several important\nobservations.\nFirstly, on the speech and MIDI datasets, models with auto-regressive (lower-half) output distribu-\ntions obtain a dramatic advantage over models with fully factorized output distributions (upper-half),\nachieving new SOTA results on three speech datasets. This observation reminds us that, besides cap-\nturing the long-term temporal structure across steps, how to properly model the intra-step dependency\nis equally, if not more, crucial to the practical performance.\nSecondly, when the auto-regressive output distribution is employed (lower-half), the non-stochastic\nrecurrent models consistently outperform their stochastic counterparts across all datasets. In other\nwords, the advantage of SRNN completely disappears once a powerful output distribution is used.\nCombined with the previous observation, it veri\ufb01es our earlier concern that the so-called superiority\nof F-SRNN over F-RNN is merely a result of the biased experiment design in previous work.\nIn addition, as we expected, when the inductive bias of the \ufb02at model matches the characteristics of\nspeech data, it will achieve a better performance than the hierarchical model. Inversely, when the\nprior does not match data property on the other datasets, the hierarchical model is always better. In\nthe extreme case of permuted TIMIT, the \ufb02at model even falls behind factorized models, while the\nhierarchical model achieves a very decent performance that is even much better than what F-SRNN\ncan achieve on the original TIMIT. This shows that the hierarchical model is usually more robust,\nespecially when we don\u2019t have a good prior.\nOverall, we don\u2019t \ufb01nd any advantage of employing stochastic latent variables for multivariate\nsequence modeling. Instead, relying on a full auto-regressive solution yields better or even state-\nof-the-art performances. Combined with the observation that -RNN-random can often achieve a\ncompetitive performance to F-SRNN, we believe that the theoretical advantage of latent-variable\nmodels in sequence modeling is still far from ful\ufb01lled, if ultimately possible. In addition, we suggest\nfuture development along this line compare with the simple but extremely robust baselines with an\nauto-regressive output distribution.\n\n4.4 Training Time Comparison\n\nHere, we report the training time of different methods in TIMIT dataset. The running times of training\nmodels for 40k updating steps on TIMIT are summarized in Table 5. The input length indicates the\nsample length of input during the training phrase. Admittedly, modeling the intra-step correlation\n(*-hier and *-\ufb02at model) would require extra computation time. Hence, this leads to a trade-off\nbetween quality and speed. Ideally, latent-variable models would provide a solution close to the sweet\npoint of this trade-off. However, in our experiment, we \ufb01nd a simple hierarchical auto-regressive\nmodel trained with a shorter input length could already achieve signi\ufb01cantly better performance with\na comparable computation time (RNN-hier vs. F-SRNN in Table 5).\n\n1https://github.com/zihangdai/reexamine-srnn\n\n8\n\n\fInput Length\nModel Name\nTraining Time\nLog-Likelihood\n\n8000\n\n1000\n\nF-RNN F-SRNN -RNN RNN-hier\n0.54h\n32,745\n\n9.92h\n109,641\n\n0.94h\n69,296\n\n0.90h\n66,453\n\nSRNN-hier RNN-\ufb02at\n37.48h\n117,721\n\n12.52h\n107,912\n\nSRNN-\ufb02at RNN-hier\n\n42.26h\n109,284\n\n1.7h\n\n101,713\n\nTable 5: Training time comparison between various models.\n\n5 Conclusion and Discussion\n\nIn summary, our re-examination reveals a misleading impression on the bene\ufb01ts of latent variables\nin sequence modeling. From our empirical observation, the main effect of latent variables is only\nto provide a mechanism to leverage the intra-step correlation, which is however, not as powerful as\nemploying the straightforward auto-regressive decomposition. It remains unclear what leads to the\nsigni\ufb01cant gap between the theoretical potential of latent variables and their practical effectiveness,\nwhich we believe deserves more research attention. Meanwhile, given the large gain of modeling\nsimultaneity, using sequential structures to better capture local patterns is another good future\ndirection in sequence modeling.\n\nAcknowledgment\n\nThis work is supported in part by the National Science Foundation (NSF) under grant IIS-1546329\nand by DOE-Of\ufb01ce of Science under grant ASCR #KJ040201.\n\nReferences\n[1] Alex Graves. Generating sequences with recurrent neural networks.\n\narXiv:1308.0850, 2013.\n\narXiv preprint\n\n[2] Benigno Uria, Marc-Alexandre C\u00f4t\u00e9, Karol Gregor, Iain Murray, and Hugo Larochelle. Neural\nautoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184\u2013\n7220, 2016.\n\n[3] Valentin Flunkert, David Salinas, and Jan Gasthaus. Deepar: Probabilistic forecasting with\n\nautoregressive recurrent networks. arXiv preprint arXiv:1704.04110, 2017.\n\n[4] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved\n\nautoregressive generative model. arXiv preprint arXiv:1712.09763, 2017.\n\n[5] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, \u0141ukasz Kaiser, Noam Shazeer, and Alexander\n\nKu. Image transformer. arXiv preprint arXiv:1802.05751, 2018.\n\n[6] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le,\nand Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a \ufb01xed-length\ncontext. arXiv preprint arXiv:1901.02860, 2019.\n\n[7] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint\n\narXiv:1312.6114, 2013.\n\n[8] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation\nand approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.\n[9] Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint\n\narXiv:1411.7610, 2014.\n\n[10] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua\nBengio. A recurrent latent variable model for sequential data. In Advances in neural information\nprocessing systems, pages 2980\u20132988, 2015.\n\n[11] Marco Fraccaro, S\u00f8ren Kaae S\u00f8nderby, Ulrich Paquet, and Ole Winther. Sequential neural\nmodels with stochastic layers. In Advances in neural information processing systems, pages\n2199\u20132207, 2016.\n\n9\n\n\f[12] Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre C\u00f4t\u00e9, Nan Rosemary\nKe, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in\nneural information processing systems, pages 6713\u20136723, 2017.\n\n[13] Guokun Lai, Bohan Li, Guoqing Zheng, and Yiming Yang. Stochastic wavenet: A generative\n\nlatent variable model for sequential data. arXiv preprint arXiv:1806.06116, 2018.\n\n[14] Emre Aksan and Otmar Hilliges. Stcn: Stochastic temporal convolutional networks. arXiv\n\npreprint arXiv:1902.06568, 2019.\n\n[15] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural\n\nnetworks. arXiv preprint arXiv:1601.06759, 2016.\n\n[16] Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.\nConditional image generation with pixelcnn decoders. In Advances in Neural Information\nProcessing Systems, pages 4790\u20134798, 2016.\n\n[17] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the\npixelcnn with discretized logistic mixture likelihood and other modi\ufb01cations. arXiv preprint\narXiv:1701.05517, 2017.\n\n[18] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw:\n\nA recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015.\n\n[19] Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra.\nTowards conceptual compression. In Advances In Neural Information Processing Systems,\npages 3549\u20133557, 2016.\n\n[20] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing \ufb02ows.\n\narXiv preprint arXiv:1505.05770, 2015.\n\n[21] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp.\n\narXiv preprint arXiv:1605.08803, 2016.\n\n[22] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.\nImproved variational inference with inverse autoregressive \ufb02ow. In Advances in neural informa-\ntion processing systems, pages 4743\u20134751, 2016.\n\n[23] Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling temporal\ndependencies in high-dimensional sequences: Application to polyphonic music generation and\ntranscription. arXiv preprint arXiv:1206.6392, 2012.\n\n[24] Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder\nfor distribution estimation. In International Conference on Machine Learning, pages 881\u2013889,\n2015.\n\n[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\n[26] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv\n\npreprint arXiv:1608.03983, 2016.\n\n[27] Sam Roweis and Zoubin Ghahramani. A unifying review of linear gaussian models. Neural\n\ncomputation, 11(2):305\u2013345, 1999.\n\n[28] Lawrence R Rabiner and Biing-Hwang Juang. An introduction to hidden markov models. ieee\n\nassp magazine, 3(1):4\u201316, 1986.\n\n[29] Rudolph Emil Kalman. A new approach to linear \ufb01ltering and prediction problems. Journal of\n\nbasic Engineering, 82(1):35\u201345, 1960.\n\n[30] Ilya Sutskever, Geoffrey E Hinton, and Graham W Taylor. The recurrent temporal restricted\nboltzmann machine. In Advances in neural information processing systems, pages 1601\u20131608,\n2009.\n\n10\n\n\f[31] Zhe Gan, Chunyuan Li, Ricardo Henao, David E Carlson, and Lawrence Carin. Deep temporal\nsigmoid belief networks for sequence modeling. In Advances in Neural Information Processing\nSystems, pages 2467\u20132475, 2015.\n\n[32] Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman \ufb01lters. arXiv preprint\n\narXiv:1511.05121, 2015.\n\n[33] Rahul G Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear\n\nstate space models. In AAAI, pages 2101\u20132109, 2017.\n\n[34] Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition\nand nonlinear dynamics model for unsupervised learning. In Advances in Neural Information\nProcessing Systems, pages 3601\u20133610, 2017.\n\n[35] Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta.\nComposing graphical models with neural networks for structured representations and fast\ninference. In Advances in neural information processing systems, pages 2946\u20132954, 2016.\n\n11\n\n\f", "award": [], "sourceid": 4225, "authors": [{"given_name": "Guokun", "family_name": "Lai", "institution": "Carnegie Mellon University"}, {"given_name": "Zihang", "family_name": "Dai", "institution": "Carnegie Mellon University"}, {"given_name": "Yiming", "family_name": "Yang", "institution": "CMU"}, {"given_name": "Shinjae", "family_name": "Yoo", "institution": "Brookhaven National Lab"}]}