{"title": "Variational Graph Recurrent Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 10701, "page_last": 10711, "abstract": "Representation learning over graph structured data has been mostly studied in static graph settings while efforts for modeling dynamic graphs are still scant. In this paper, we develop a novel hierarchical variational model that introduces additional latent random variables to jointly model the hidden states of a graph recurrent neural network (GRNN) to capture both topology and node attribute changes in dynamic graphs. We argue that the use of high-level latent random variables in this variational GRNN (VGRNN) can better capture potential variability observed in dynamic graphs as well as the uncertainty of node latent representation. With semi-implicit variational inference developed for this new VGRNN architecture (SI-VGRNN), we show that flexible non-Gaussian latent representations can further help dynamic graph analytic tasks. Our experiments with multiple real-world dynamic graph datasets demonstrate that SI-VGRNN and VGRNN consistently outperform the existing baseline and state-of-the-art  methods by a significant margin in dynamic link prediction.", "full_text": "Variational Graph Recurrent Neural Networks\n\nEhsan Hajiramezanali\u2020\u2217, Arman Hasanzadeh\u2020\u2217, Nick Duf\ufb01eld\u2020, Krishna Narayanan\u2020,\n\nMingyuan Zhou\u2021, Xiaoning Qian\u2020\n\n\u2020 Department of Electrical and Computer Engineering, Texas A&M University\n\n{ehsanr, armanihm, duffieldng, krn, xqian}@tamu.edu\n\u2021 McCombs School of Business, The University of Texas at Austin\n\nmingyuan.zhou@mccombs.utexas.edu\n\nAbstract\n\nRepresentation learning over graph structured data has been mostly studied in static\ngraph settings while efforts for modeling dynamic graphs are still scant. In this\npaper, we develop a novel hierarchical variational model that introduces additional\nlatent random variables to jointly model the hidden states of a graph recurrent\nneural network (GRNN) to capture both topology and node attribute changes in\ndynamic graphs. We argue that the use of high-level latent random variables in\nthis variational GRNN (VGRNN) can better capture potential variability observed\nin dynamic graphs as well as the uncertainty of node latent representation. With\nsemi-implicit variational inference developed for this new VGRNN architecture (SI-\nVGRNN), we show that \ufb02exible non-Gaussian latent representations can further\nhelp dynamic graph analytic tasks. Our experiments with multiple real-world\ndynamic graph datasets demonstrate that SI-VGRNN and VGRNN consistently\noutperform the existing baseline and state-of-the-art methods by a signi\ufb01cant\nmargin in dynamic link prediction.\n\n1\n\nIntroduction\n\nNode embedding maps each node in a graph to a vector in a low-dimensional latent space, in which\nclassical feature vector-based machine learning formulations can be adopted [5]. Most of the existing\nnode embedding techniques assume that the graph is static and that learning tasks are performed\non \ufb01xed sets of nodes and edges [19, 23, 12, 20, 14, 1]. However, many real-world problems are\nmodeled by dynamic graphs, where graphs are constantly evolving over time. Such graphs have been\ntypically observed in social networks, citation networks, and \ufb01nancial transaction networks. A naive\nsolution to node embedding for dynamic graphs is simply applying static methods to each snapshot of\ndynamic graphs. Among many potential problems of such a naive solution, it is clear that it ignores\nthe temporal dependencies between snapshots.\nSeveral node embedding methods have been proposed to capture the temporal graph evolution for\nboth networks without attributes [10, 26] and attributed networks [24, 16]. However, all of the\nexisting dynamic graph embedding approaches represent each node by a deterministic vector in\na low-dimensional space [2]. Such deterministic representations lack the capability of modeling\nuncertainty of node embedding, which is a natural consideration when having multiple information\nsources, i.e. node attributes and graph structure.\nIn this paper, we propose a novel node embedding method for dynamic graphs that maps each node to\na random vector in the latent space. More speci\ufb01cally, we \ufb01rst introduce a dynamic graph autoencoder\nmodel, namely graph recurrent neural network (GRNN), by extending the use of graph convolutional\n\n\u2217Both authors contributed equally.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fneural networks (GCRN) [21] to dynamic graphs. Then, we argue that GRNN lacks the expressive\npower for fully capturing the complex dependencies between topological evolution and time-varying\nnode attributes because the output probability in standard RNNs is limited to either a simple unimodal\ndistribution or a mixture of unimodal distributions [3, 22, 6, 8]. Next, to increase the expressive\npower of GRNN in addition to modeling the uncertainty of node latent representations, we propose\nvariational graph recurrent neural network (VGRNN) by adopting high-level latent random variables\nin GRNN. Our proposed VGRNN is capable of learning interpretable latent representation as well as\nbetter modeling of very sparse dynamic graphs.\nTo further boost the expressive power and interpretability of our new VGRNN method, we integrate\nsemi-implicit variational inference [25] with VGRNN. We show that semi-implicit variational graph\nrecurrent neural network (SI-VGRNN) is capable of inferring more \ufb02exible and complex posteriors.\nOur experiments demonstrate the superior performance of VGRNN and SI-VGRNN in dynamic link\nprediction tasks in several real-world dynamic graph datasets compared to baseline and state-of-the-art\nmethods.\n\n2 Background\n\nGraph convolutional recurrent networks (GCRN). GCRN was introduced by Seo et al. [21]\nto model time series data de\ufb01ned over nodes of a static graph. Series of frames in videos and\nspatio-temporal measurements on a network of sensors are two examples of such datasets. GCRN\ncombines graph convolutional networks (GCN) [4] with recurrent neural networks (RNN) to capture\nspatial and temporal patterns in data. More precisely, given a graph G with N nodes, whose\ntopology is determined by the adjacency matrix A \u2208 RN\u00d7N , and a sequence of node attributes\nX = {X(1), X(2), . . . , X(T )}, GCRN reads M-dimensional node attributes X(t) \u2208 RN\u00d7M and\nupdates its hidden state ht \u2208 Rp at each time step t:\n\n(cid:16)\n\n(cid:17)\n\nht = f\n\nA, X(t), ht\u22121\n\n.\n\n(1)\n\nHere f is a non-probabilistic deep neural network. It can be any recursive network including gated\nactivation functions such as long short-term memory (LSTM) or gated recurrent units (GRU), where\nthe deep layers inside them are replaced by graph convolutional layers. GCRN models node attribute\nsequences by parameterizing a factorization of the joint probability distribution as a product of\nconditional probabilities such that\n\n(cid:16)\n\n(cid:17)\n\nT(cid:89)\n\n(cid:16)\n\n(cid:17)\n\n(cid:16)\n\n(cid:17)\n\nX(1), X(2), . . . , X(T ) | A\n\np\n\n=\n\nX(t) | X(<t), A\n\np\n\n;\n\np\n\nX(t) | X(<t), A\n\n= g(A, ht\u22121).\n\nt=1\n\nDue to the deterministic nature of the transition function f, the choice of the mapping func-\ntion g here effectively de\ufb01nes the only source of variability in the joint probability distributions\np(X(1), X(2), . . . , X(T ) | A) that can be expressed by the standard GCRN. This can be problematic\nfor sequences that are highly variable. More speci\ufb01cally, when the variability of X is high, the\nmodel tries to map this variability in hidden states h, leading to potentially high variations in h and\nthereafter over\ufb01tting of training data. Therefore, GCRN is not fully capable of modeling sequences\nwith high variations. This fundamental problem of autoregressive models has been addressed for\nnon-graph-structured datasets by introducing stochastic hidden states to the model [7, 3, 9].\nIn this paper, we integrate GCN and RNN into a graph RNN (GRNN) framework, which is a dynamic\ngraph autoencoder model. While GCRN aims to model dynamic node attributes de\ufb01ned over a static\ngraph, GRNN can get different adjacency matrices at different time snapshots and reconstruct the\ngraph at time t by adopting an inner-product decoder on the hidden state ht. More speci\ufb01cally, ht\ncan be viewed as node embedding of the dynamic graph at time t. To further improve the expressive\npower of GRNN, we introduce stochastic latent variables by combining GRNN with variational graph\nautoencoder (VGAE) [14]. This way, not only we can capture time dependencies between graphs\nwithout making smoothness assumption, but also each node is represented with a distribution in the\nlatent space. Moreover, the prior construction devised in VGRNN allows it to predict links in the\nfuture time steps.\nSemi-implicit variational inference (SIVI). SIVI has been shown effective to learn posterior distri-\nbutions with skewness, kurtosis, multimodality, and other characteristics, which were not captured\n\n2\n\n\fby the existing variational inference methods [25]. To characterize the latent posterior q(z|x), SIVI\nintroduces a mixing distribution on the parameters of the original posterior distribution to expand\nthe variational family with a hierarchical construction: z \u223c q(z|\u03c8) with \u03c8 \u223c q\u03c6(\u03c8). \u03c6 denotes\nthe distribution parameter to be inferred. While the original posterior q(z|\u03c8) is required to have\nan analytic form, its mixing distribution is not subject to such a constraint, and so the marginal\nposterior distribution is often implicit and more expressive that has no analytic density function.\nIt is also common that the marginal of the hierarchy is implicit, even if both the posterior and its\nmixing distribution are explicit. We will integrate SIVI in our new model to infer more \ufb02exible and\ninterpretable node embedding for dynamic graphs.\n\n3 Variational graph recurrent neural network (VGRNN)\n\n3.1 Overview\nWe consider a dynamic graph G = {G(1), G(2), . . . , G(T )} where G(t) = (V (t),E (t)) is the graph at\ntime step t with V (t) and E (t) being the corresponding node and edge sets, respectively. In this paper,\nwe aim to develop a model that is universally compatible with potential changes in both node and\nedge sets. In particular, the cardinality of both V (t) and E (t) can change across time. There are no\nconstraints on the relationships between (V (t),E (t)) and (V (t+1),E (t+1)), namely new nodes can join\nthe dynamic graph and create edges to the existing nodes or previous nodes can disappear from the\ngraph. On the other hand, new edges can form between snapshots while existing edges can disappear.\nLet Nt denotes the number of nodes , i.e., the cardinality of V (t), at time step t. Therefore, VGRNN\ncan take as input a variable-length adjacency matrix sequence A = {A(1), A(2), . . . , A(T )}. In\naddition, when considering node attributes, different attributes can be observed at different snapshots\nwith a variable-length node attribute sequence X = {X(1), X(2), . . . , X(T )}. Note that A(t) and X(t)\nare Nt \u00d7 Nt and Nt \u00d7 M matrices, respectively, where M is the dimension of the node attributes that\nis constant across time. Inspired by variational recurrent neural networks (VRNN) [3], we construct\nVGRNN by integrating GRNN and VGAE so that complex dependencies between topological and\nnode attribute dynamics are modeled suf\ufb01ciently and simultaneously. Moreover, each node at each\ntime is represented with a distribution, hence uncertainty of latent representations of nodes are also\nmodelled in VGRNN.\n\n3.2 VGRNN model\n\nGeneration. The VGRNN model adopts a VGAE to model each graph snapshot. The VGAEs\nacross time are conditioned on the state variable ht\u22121, modeled by a GRNN. Such an architecture\ndesign will help each VGAE to take into account the temporal structure of the dynamic graph. More\ncritically, unlike a standard VGAE, our VGAE in VGRNN takes a new prior on the latent random\nvariables by allowing distribution parameters to be modelled by either explicit or implicit complex\nfunctions of information of the previous time step. More speci\ufb01cally, instead of imposing a standard\nmultivariate Gaussian distribution with deterministic parameters, VGAE in our VGRNN learns the\nprior distribution parameters based on the hidden states in previous time steps. Hence, our VGRNN\nallows more \ufb02exible latent representations with greater expressive power that captures dependencies\nbetween and within topological and node attribute evolution processes. In particular, we can write the\nconstruction of the prior distribution adopted in our experiments as follows,\n\n(cid:17)\n\n(cid:110)\n\n(cid:111)\n\n(cid:16)\n\nZ(t)(cid:17)\n\np\n\n(cid:16)\n\nNt(cid:89)\n\ni=1\n\n(cid:17)\n\ni \u223c N(cid:16)\n\n=\n\np\n\nZ(t)\ni\n\n; Z(t)\n\n\u00b5(t)\ni,prior, diag((\u03c3(t)\n\ni,prior)2)\n\n,\n\n\u00b5(t)\nprior, \u03c3(t)\nprior\n\n= \u03d5prior(ht\u22121),\n\nprior \u2208 RNt\u00d7l and \u03c3(t)\ni,prior and \u03c3(t)\n\nwhere \u00b5(t)\nand \u00b5(t)\ndistribution will be conditioned on Z(t) as:\n\ni,prior are the i-th row of \u00b5(t)\n\n(2)\nprior \u2208 RNt\u00d7l denote the parameters of the conditional prior distribution,\nprior, respectively. Moreover, the generating\n\nprior and \u03c3(t)\n\n(cid:16)\n\u03c0(t)(cid:17)\n\n,\n\n\u03c0(t) = \u03d5dec(cid:16)\n\nZ(t)(cid:17)\n\nA(t) | Z(t) \u223c Bernoulli\n\n,\n\n(3)\n\nwhere \u03c0(t) denotes the parameter of the generating distribution; \u03d5prior and \u03d5dec can be any highly\n\ufb02exible functions such as neural networks.\n\n3\n\n\fZ(t)\n\nZ(t)\n\nZ(t)\n\nZ(t)\n\nht\u22121\n\nht\n\nht\u22121\n\nht\n\nht\u22121\n\nht\n\nht\u22121\n\nht\n\nX(t)\n\nA(t)\n\nX(t)\n\nA(t)\n\nX(t)\n\nA(t)\n\nX(t)\n\nA(t)\n\n(a) Prior\n\n(b) Generation\n\n(c) Recurrence\n\n(d) Inference\n\nFigure 1: Graphical illustrations of each operation of VGRNN; (a) computing the conditional prior\nby (2); (b) decoder function (3); (c) updating the GRNN hidden states using (4); and (d) inference of\nthe posterior distribution for latent variables by (3.2).\n\nOn the other hand, the backbone GRNN enables \ufb02exible modeling of complex dependency involving\nboth graph topological dynamics and node attribute dynamics. The GRNN updates its hidden states\nusing the recurrence equation:\n\n(cid:16)\n\nA(t), \u03d5x(cid:16)\n\nX(t)(cid:17)\n\n, \u03d5z(cid:16)\n\nZ(t)(cid:17)\n\nht =f\n\n(cid:17)\n\n, ht\u22121\n\n,\n\n(4)\n\nwhere f is originally the transition function from equation (1). Unlike the GRNN de\ufb01ned in [21],\ngraph topology can change in different time steps as it does in real-world dynamic graphs, and the\nadjacency matrix A(t) is time dependent in VGRNN. To further enhance the expressive power, \u03d5x\nand \u03d5z are deep neural networks which operate on each node independently and extract features\nfrom X(t) and Z(t), respectively. These feature extractors are crucial for learning complex graph\ndynamics. Based on (4), ht is a function of A\u2264(t), X\u2264(t), and Z\u2264(t). Therefore, the prior and\ngenerating distributions in equations (2) and (3) de\ufb01ne the distributions p(Z(t) | A(<t), X(<t), Z(<t))\nand p(A(t) | Z(t)), respectively. The generative model can be factorized as\n\n(cid:16)\n\nZ(t) | A(<t), X(<t), Z(<t)(cid:17)\n\np\n\n(cid:16)\n\nA(t) | Z(t)(cid:17)\n\n,\n\n(5)\n\n(cid:16)\n\nA(\u2264T ), Z(\u2264T ) | X(<T )(cid:17)\n\np\n\nT(cid:89)\n\nt=1\n\n=\n\np\n\ni\n\nwhere the prior of the \ufb01rst snapshot is considered to be a standard multivariate Gaussian distribution,\n|\u2212) \u223c N (0, I) for i \u2208 {1, . . . , N0} and h0 = 0. Also, if a previously unobserved node is\ni.e. p(Z(0)\nadded to the graph at snapshot t, we consider the hidden state of that node at snapshot t \u2212 1 is zero\nand hence the prior for that node at time t is N (0, I). If node deletion occurs, we assume that the\nidentity of nodes can be maintained thus removing a node, which is equivalent to removing all the\nedges connected to it, will not affect the prior construction for the next step. More speci\ufb01cally, the\nsizes of A and X can change in time while their latent space maintains across time.\nInference. With the VGRNN framework, the node embedding for dynamic graphs can be derived\nby inferring the posterior distribution of Z(t) which is also a function of ht\u22121. More speci\ufb01cally,\n\nZ(t) | A(t), X(t), ht\u22121\n\nq\n\n=\n\nq\n\nZ(t)\ni\n\n| A(t), X(t), ht\u22121\n\n\u00b5(t)\ni,enc, diag((\u03c3(t)\n\ni,enc)2)\n\n(cid:16)\n\n(cid:17)\n\nNt(cid:89)\n\n(cid:16)\n(cid:16)\n(cid:16)\n\ni=1\n\n\u00b5(t)\n\nenc = GNN\u00b5\n\n\u03c3(t)\n\nenc = GNN\u03c3\n\nA(t), CONCAT\n\nA(t), CONCAT\n\n=\n\n(cid:17)\nNt(cid:89)\n(cid:16)\nX(t)(cid:17)\n\u03d5x(cid:16)\nX(t)(cid:17)\n\u03d5x(cid:16)\n(cid:16)\n\ni=1\n\nN(cid:16)\n\n, ht\u22121\n\n, ht\u22121\n\n(cid:17)(cid:17)\n(cid:17)(cid:17)\n\n,\n\n,\n\n(cid:17)\n\n,\n\n(6)\n\nwhere \u00b5(t)\nenc and \u03c3(t)\ni,enc are\nthe i-th row of \u00b5(t)\nenc, respectively. GNN\u00b5 and GNN\u03c3 are the encoder functions and can be\nany of the various types of graph neural networks, such as GCN [15], GCN with Chebyshev \ufb01lters\n[4] and GraphSAGE [13].\n\nenc denote the parameters of the approximated posterior, and \u00b5(t)\nenc and \u03c3(t)\n\ni,enc and \u03c3(t)\n\n4\n\n\fLearning. The objective function of VGRNN is derived from the variational lower bound at each\nsnapshot. More precisely, using equation (5) , the evidence lower bound of VGRNN can be written\nas follows,\n\n(cid:26)\nT(cid:88)\n\nt=1\n\nL =\n\nE\nZ(t)\u223cq(Z(t) | A(\u2264t),X(\u2264t),Z(<t))log p\n\n(cid:16)\n\nA(t) | Z(t)(cid:17)\n(cid:16)\nZ(t) | A(\u2264t), X(\u2264t), Z(<t)(cid:17) || p\n\n(cid:16)\n\n(cid:18)\n\n\u2212 KL\n\nq\n\nZ(t) | A(<t), X(<t), Z(<t)(cid:17)(cid:19)(cid:27)\n\n(7)\n\n.\n\nWe learn the parameters of the generative and inference models jointly by optimizing the variational\nlower bound with respect to the variational parameters. The graphical representation of VGRNN is\nillustrated in Fig. 1, operations (a)\u2013(d) correspond to equations (2) \u2013 (4), and (3.2), respectively. We\nnote that if we don\u2019t use hidden state variables ht\u22121 in the derivation of the prior distribution, then\nthe prior in (2) becomes independent across snapshots and reduces to the prior of vanilla VGAE.\nThe inner-product decoder is adopted in VGRNN for the experiments in this paper\u2013 \u03d5dec in (3)\u2013to\nclearly demonstrate the advantages of the stochastic recurrent models for the encoder. Potential\nextensions with other decoders can be integrated with VGRNN if necessary. More speci\ufb01cally,\n\n(cid:16)\n\nA(t) | Z(t)(cid:17)\n\np\n\n=\n\nNt(cid:89)\n\nNt(cid:89)\n\n(cid:16)\n\n(cid:17)\n\n(cid:16)\n\n(A(t)\n\ni,j | z(t)\n\ni\n\n, z(t)\nj\n\np\n\n; p\n\ni,j = 1| z(t)\nA(t)\n\ni\n\n, z(t)\nj\n\n= sigmoid\n\nz(t)\ni (z(t)\n\n(cid:17)\n\n(cid:16)\n\nj )T(cid:17)\n\n,\n\ni=1\n\nj=1\n\ncorresponds to the embedding representation of node v(t)\n\n(8)\ni \u2208 V (t) at time step t. Note the\nwhere z(t)\ni\ngenerating distribution can also be conditioned on ht\u22121 if we want to generate X(t) in addition to\nthe adjacency matrix for other applications. In such cases, \u03d5dec should be a highly \ufb02exible neural\nnetwork instead of a simple inner-product function.\n\n3.3 Semi-implicit VGRNN (SI-VGRNN)\n\nTo further increase the expressive power of the variational posterior of VGRNN, we introduce a\nSI-VGRNN dynamic node embedding model. We impose a mixing distributions on the variational\ndistribution parameters in (8) to model the posterior of VGRNN with a semi-implicit hierarchical\nconstruction:\n\nZ(t) \u223c q(Z(t) | \u03c8t),\n\n\u03c8t \u223c q\u03c6(\u03c8t | A(\u2264t), X(\u2264t), Z(<t)) = q\u03c6(\u03c8t|A(t), X(t), ht\u22121).\n\n(9)\nWhile the variational distribution q(Z(t) | \u03c8t) is required to be explicit, the mixing distribution, q\u03c6, is\n\u03c8t\u223cq\u03c6(\u03c8t|A(t),X(t),ht\u22121)(q(zt|\u03c8t)).\nnot subject to such a constraint, leading to considerably \ufb02exible E\nMore speci\ufb01cally, SI-VGRNN draws samples from q\u03c6 by transforming random noise \u0001t via a graph\nneural network, which generally leads to an implicit distribution for q\u03c6.\nInference. Under the SI-VGRNN construction, the generation, prior and recurrence models are the\nsame as VGRNN (equations (2) to (5)). We indeed have updated the encoder functions as follows:\n\nj = GNNj(A(t), CONCAT(ht\u22121, \u0001(t)\n(cid:96)(t)\n\nj , (cid:96)(t)\nenc(A(t), X(t), ht\u22121) = GNN\u00b5(A(t), (cid:96)(t)\n\u00b5(t)\nq(Z(t)\ni\n\n| A(t), X(t), ht\u22121, \u00b5(t)\n\ni,enc, \u03a3(t)\n\ni,enc) = N (\u00b5(t)\n\nj\u22121)); \u0001(t)\nL ), \u03a3(t)\n\nj \u223c qj(\u0001) for j = 1, . . . , L, (cid:96)(t)\n0 = \u03d5x\nenc(A(t), X(t), ht\u22121) = GNN\u03a3(A(t), (cid:96)(t)\nL ),\ni,enc(A(t), X(t), ht\u22121)),\n\ni,enc(A(t), X(t), ht\u22121), \u03a3(t)\n\n\u03c4\n\n(cid:16)\n\nX(t)(cid:17)\n\nwhere L is the number of stochastic layers and \u0001(t)\nis Nt-dimensional random noise drawn from a\nj\ndistribution qj with Nt denoting number of nodes at time t. Note that given {A(t), X(t), ht\u22121}, \u00b5(t)\ni,enc\nand \u03a3(t)\ni,enc are now random variables rather than analytic and thus the posterior is not Gaussian after\nmarginalizing.\n\n5\n\n\fMetrics\n\nEnron\n\nCOLAB Facebook\n\nHEP-TH\n\nTable 1: Dataset statistics.\n\nNumber of Snapshots\n\nNumber of Nodes\nNumber of Edges\nAverage Density\n\n11\n184\n\nNumber of Node Attributes\n\n-\n\n115-266\n0.01284\n\n165-308\n0.00514\n\n844-1068\n0.00591\n\n10\n315\n\n-\n\n9\n663\n\n-\n\n40\n\n1199-7623\n769-34941\n0.00117\n\n-\n\nCora\n11\n\n708-2708\n406-5278\n0.00154\n\n1433\n\nSocial Evolution\n\n27\n84\n\n303-1172\n0.21740\n\n168\n\nLearning.\nELBO goes beyond the simple VGRNN in (7) and can be written as\n\nIn this construction, because the parameters of the posterior are random variables, the\n\n(cid:26)\nT(cid:88)\n\nt=1\n\nL =\n\n(cid:18)\n\nE\n\u03c8t\u223cq\u03c6(\u03c8t|A(t),X(t),ht\u22121)\n\nEZ(t)\u223cq(Z(t) | \u03c8t)log\n\n\u2212 KL\n\nE\n\u03c8t\u223cq\u03c6(\u03c8t|A(t),X(t),ht\u22121)q\n\n(cid:16)\n\np(A(t) | Z(t), ht\u22121)\n\n(cid:17)\n(cid:19)(cid:27)\n(cid:17) || p(Z(t) | ht\u22121)\n\n(cid:16)\nZ(t) | \u03c8t\n\n(10)\n\n.\n\nDirect optimization of the ELBO in SIVI is not tractable [25], hence to infer variational parameters\nof SI-VGRNN, we derive a lower bound for the ELBO as follows (see the supplements for more\ndetails.).\n\nE\n\u03c8t\u223cq\u03c6(\u03c8t|A(t),X(t),ht\u22121)\n\nEZ(t)\u223cq(Z(t) | \u03c8t)log\n\np(A(t) | Z(t), ht\u22121) p(Z(t) | ht\u22121)\n\nq(Z(t) | \u03c8t)\n\nT(cid:88)\n\nt=1\n\nL =\n\n(cid:32)\n\n(cid:33)\n\n.\n\n(11)\n\n4 Experiments\n\nDatasets. We evaluate our proposed methods, VGRNN and SI-VGRNN, and baselines on six\nreal-world dynamic graphs as described in Table 1. More detailed descriptions of the datasets can be\nfound in the supplement.\nCompeting methods. We compare the performance of our proposed methods against four com-\npeting node embedding methods, three of which have the capability to model evolving graphs with\nchanging node and edge sets. Among these four, two (DynRNN and DynAERNN [11]) are based\non RNN models. By comparing our models to these methods, we will be able to see how much\nimprovement we may obtain by improving the backbone RNN with our new prior construction\ncompared to these RNNs with deterministic hidden states. We also compare our methods against a\ndeep autoencoder with fully connected layers (DynAE [11]) to show the advantages of RNN based\nsequential learning methods. Last but not least, our methods are compared with VGAE [14], which\nis implemented to analyze each snapshot separately, to demonstrate how temporal dependencies\ncaptured through hidden states in the backbone GRNN can improve the performance. More detailed\ndescriptions of these selected competing methods are described in the supplements.\nEvaluation tasks.\nIn the dynamic graph embedding literature, the term link prediction has been\nused with different de\ufb01nitions. While some of the previous works focused on link prediction in\na transductive setting and others proposed inductive models, our models are capable of working\nin both settings. We evaluate our proposed models on three different link prediction tasks that\nhave been widely used in the dynamic graph representation learning studies. More speci\ufb01cally,\ngiven partially observed snapshots of a dynamic graph G = {G(1), . . . , G(T )} with node attributes\nX = {X(1), . . . , X(T )}, dynamic link prediction problems are de\ufb01ned as follows: 1) dynamic link\ndetection, i.e. detect unobserved edges in G(T ); 2) dynamic link prediction, i.e. predict edges in\nG(T +1); 3) dynamic new link prediction, i.e. predict edges in G(T +1) that are not in G(T ).\nExperimental setups. For performance comparison, we evaluate different methods based on their\nability to correctly classify true and false edges. For dynamic link detection problem, we randomly\nremove 5% and 10% of all edges at each time for validation and test sets, respectively. We also\nrandomly select the equal number of non-links as validation and test sets to compute average precision\n(AP) and area under the ROC curve (AUC) scores. For dynamic (new) link prediction, all (new)\nedges are set to be true edges and the same number of non-links are randomly selected to compute AP\nand AUC scores. In all of our experiments, we test the model on the last three snapshots of dynamic\n\n6\n\n\fTable 2: AUC and AP scores of inductive dynamic link detection on dynamic graphs.\n\nMetrics Methods\n\nEnron\n\nAUC\n\nAP\n\n88.26 \u00b1 1.33\nVGAE\n84.06 \u00b1 3.30\nDynAE\n77.74 \u00b1 5.31\nDynRNN\nDynAERNN 91.71 \u00b1 0.94\n91.09 \u00b1 0.67\nGRNN\n94.41 \u00b1 0.73\nVGRNN\n95.03 \u00b1 1.07\nSI-VGRNN\n89.95 \u00b1 1.45\nVGAE\n86.30 \u00b1 2.43\nDynAE\n81.85 \u00b1 4.44\nDynRNN\nDynAERNN 93.16 \u00b1 0.88\n93.47 \u00b1 0.35\nGRNN\n95.17 \u00b1 0.41\nVGRNN\n96.31 \u00b1 0.72\nSI-VGRNN\n\nCOLAB\n\n70.49 \u00b1 6.46\n66.83 \u00b1 2.62\n68.01 \u00b1 5.50\n77.38 \u00b1 3.84\n86.40 \u00b1 1.48\n88.67 \u00b1 1.57\n89.15\u00b1 1.31\n73.08 \u00b1 5.70\n67.92 \u00b1 2.43\n73.12 \u00b1 3.15\n83.02 \u00b1 2.59\n88.21 \u00b1 1.35\n89.74 \u00b1 1.31\n89.90 \u00b1 1.06\n\nFacebook\n80.37 \u00b1 0.12\n60.71 \u00b1 1.05\n69.77 \u00b1 2.01\n81.71 \u00b1 1.51\n85.60 \u00b1 0.59\n88.00 \u00b1 0.57\n88.12 \u00b1 0.83\n79.80 \u00b1 0.22\n60.83 \u00b1 0.94\n70.63 \u00b1 1.75\n83.36 \u00b1 1.83\n84.77 \u00b1 0.62\n87.32 \u00b1 0.60\n87.69 \u00b1 0.92\n\nSocial Evo.\n79.85 \u00b1 0.85\n71.41 \u00b1 0.66\n74.13 \u00b1 1.74\n78.67 \u00b1 1.07\n78.27 \u00b1 0.47\n82.69 \u00b1 0.55\n83.36 \u00b1 0.53\n79.41 \u00b1 1.12\n70.18 \u00b1 1.98\n72.15 \u00b1 2.30\n77.41 \u00b1 1.47\n76.93\u00b1 0.35\n81.41 \u00b1 0.53\n83.20\u00b1 0.57\n\nHEP-TH\n79.31 \u00b1 1.97\n63.94 \u00b1 0.18\n72.39 \u00b1 0.63\n82.01 \u00b1 0.49\n89.00 \u00b1 0.46\n91.12 \u00b1 0.71\n91.05 \u00b1 0.92\n81.05 \u00b1 1.53\n63.87 \u00b1 0.21\n74.12 \u00b1 0.75\n85.57 \u00b1 0.93\n89.50 \u00b1 0.42\n91.35 \u00b1 0.77\n91.42 \u00b1 0.86\n\nCora\n\n87.60 \u00b1 0.54\n53.71 \u00b1 0.48\n76.09 \u00b1 0.97\n74.35 \u00b1 0.85\n91.35 \u00b1 0.21\n92.08 \u00b1 0.35\n94.07 \u00b1 0.44\n89.61 \u00b1 0.87\n53.84 \u00b1 0.51\n76.54 \u00b1 0.66\n79.34 \u00b1 0.77\n91.37 \u00b1 0.27\n92.92 \u00b1 0.28\n94.44 \u00b1 0.52\n\ngraphs while learning the parameters of the models based on the rest of the snapshots except for\nHEP-TH where we test the model on the last 10 snapshots. For the datasets without node attributes,\nwe consider the Nt-dimensional identity matrix as node attributes at time t. Numbers show mean\nresults and standard error for 10 runs on random datasets splits with random initializations.\nFor all datasets, we set up our VGRNN model to have a single recurrent hidden layer with 32\nGRU units. All \u03d5\u2019s in equations (3), (4), and (6) are modeled by a 32-dimensional fully-connected\nlayer. We use two 32-dimensional fully-connected layers for \u03d5prior in (2) and 2-layer GCN with\nsizes equal to [32, 16] to model \u00b5(t)\nenc in (6). For SI-VGRNN, a stochastic GCN layer\nwith size 32 and an additional GCN layer of size 16 are used to model the \u00b5. The dimension of\ninjected standard Gaussian noise \u0001 is 16. The covariance matrix \u03a3 is deterministic and is inferred\nthrough two layers of GCNs with sizes equal to [32, 16]. For fair comparison, the number of\nparameters are the same for the competing methods.\nIn all experiments, we train the models\nfor 1500 epochs with the learning rate 0.01. We use the validation set for the early stopping.\nThe supplement contains additional implementation details with hyperparmaeter selection. We\nimplemented (SI-)VGRNN in PyTorch [18] and the implementation of our proposed models is\naccessible at https://github.com/VGraphRNN/VGRNN.\n\nenc and \u03c3(t)\n\n4.1 Results and discussion\n\nDynamic link detection. Table 2 summarizes the results for inductive link detection in different\ndatasets. Our proposed methods, VGRNN and SI-VGRNN, outperform competing methods across all\ndatasets by large margins. Improvement made by (SI-)VGRNN compared to GRNN and DynAERNN\nsupports our claim that latent random variables carry more information than deterministic hidden\nstates specially for dynamic graphs with complex temporal changes. Comparing the (SI-)VGRNN\nwith VGAE, which is a static graph embedding method, shows that the improvement of the proposed\nmethods is not only because of introducing stochastic latent variables, but also successful modelling\nof temporal dependencies. We note that methods that take node attributes as input, i.e VGAE, GRNN\nand (SI-)VGRNN, outperform other competing methods by a larger margin in Cora dataset which\nincludes node attributes.\nComparing SI-VGRNN with VGRNN shows that the Gaussian latent distribution may not always\nbe the best choice for latent node representations. SI-VGRNN with \ufb02exible variational inference\ncan learn more complex latent structures. The results for the Cora dataset, which also includes\nattributes, clearly magnify the bene\ufb01ts of \ufb02exible posterior as SI-VGRNN improves the accuracy\nby 2% compared to VGRNN. We also note that the improvement made by SI-VGRNN compared\nto VGRNN is marginal in Facebook dataset. The reason could be that Gaussian latent variables\nalready represent the graph well. Therefore, more \ufb02exible posteriors do not enhance the performance\nsigni\ufb01cantly.\nDynamic (new) link prediction. Tables 3 and 4 show the results for link prediction and new link\nprediction, respectively. Since GRNN is trained as an autoencoder, it cannot predict edges in the\nnext snapshot. However, in (SI-)VGRNN, the prior construction based on previous time steps allows\nus to predict links in the future. Note that none of the methods can predict new nodes, therefore,\nHEP-TH, Cora and Citeseer datasets are not evaluated for these tasks. VGRNN and SI-VGRNN\noutperform the competing methods signi\ufb01cantly in both tasks for all of the datasets which proves\n\n7\n\n\fTable 3: AUC and AP scores of dynamic link prediction on real-world dynamic graphs.\n\nMetrics Methods\n\nEnron\n\n74.22 \u00b1 0.74\nDynAE\n86.41 \u00b1 1.36\nDynRNN\nDynAERNN 87.43 \u00b1 1.19\n93.10 \u00b1 0.57\nVGRNN\n93.93 \u00b1 1.03\nSI-VGRNN\n76.00 \u00b1 0.77\nDynAE\n85.61 \u00b1 1.46\nDynRNN\nDynAERNN 89.37 \u00b1 1.17\n93.29 \u00b1 0.69\nVGRNN\n94.44 \u00b1 0.85\nSI-VGRNN\n\nCOLAB\n63.14 \u00b1 1.30\n75.7 \u00b1 1.09\n76.06 \u00b1 1.08\n85.95 \u00b1 0.49\n85.45 \u00b1 0.91\n64.02 \u00b1 1.08\n78.95 \u00b1 1.55\n81.84 \u00b1 0.89\n87.77 \u00b1 0.79\n88.36 \u00b1 0.73\n\nFacebook\n56.06 \u00b1 0.29\n73.18 \u00b1 0.60\n76.02 \u00b1 0.88\n89.47 \u00b1 0.37\n90.94 \u00b1 0.37\n56.04 \u00b1 0.37\n75.88 \u00b1 0.42\n78.55 \u00b1 0.73\n89.04 \u00b1 0.33\n90.19 \u00b1 0.27\n\nSocial Evo.\n65.50 \u00b1 1.66\n71.37 \u00b1 0.72\n73.47 \u00b1 0.49\n77.54 \u00b1 1.04\n77.84 \u00b1 0.79\n63.66 \u00b1 2.27\n69.02 \u00b1 1.71\n71.79 \u00b1 0.81\n77.03 \u00b1 0.83\n77.40 \u00b1 0.43\n\nAUC\n\nAP\n\nAUC\n\nAP\n\nTable 4: AUC and AP scores of dynamic new link prediction on real-world dynamic graphs.\n\nMetrics Methods\n\nEnron\n\n66.10 \u00b1 0.71\nDynAE\n83.20 \u00b1 1.01\nDynRNN\nDynAERNN 83.77 \u00b1 1.65\n88.43 \u00b1 0.75\nVGRNN\n88.60 \u00b1 0.95\nSI-VGRNN\n66.50 \u00b1 1.12\nDynAE\n80.96 \u00b1 1.37\nDynRNN\nDynAERNN 85.16 \u00b1 1.04\n87.57 \u00b1 0.57\nVGRNN\n87.88 \u00b1 0.84\nSI-VGRNN\n\nCOLAB\n58.14 \u00b1 1.16\n71.71\u00b10.73\n71.99 \u00b1 1.04\n77.09 \u00b1 0.23\n77.95 \u00b1 0.41\n58.82 \u00b1 1.06\n75.34 \u00b1 0.67\n77.68 \u00b1 0.66\n79.63 \u00b1 0.94\n81.26 \u00b1 0.38\n\nFacebook\n54.62 \u00b1 0.22\n73.32 \u00b1 0.60\n76.35 \u00b1 0.50\n87.20 \u00b1 0.43\n87.74 \u00b1 0.53\n54.57 \u00b1 0.20\n75.52 \u00b1 0.50\n78.70 \u00b1 0.44\n86.30 \u00b1 0.29\n86.72 \u00b1 0.54\n\nSocial Evo.\n55.25 \u00b1 1.34\n65.69 \u00b1 3.11\n66.61 \u00b1 2.18\n75.00 \u00b1 0.97\n76.45 \u00b1 1.19\n54.05 \u00b1 1.63\n63.47 \u00b1 2.70\n65.03 \u00b1 1.74\n73.48 \u00b1 1.11\n73.85 \u00b1 1.33\n\nFigure 2: Evolution of graph statistics through time.\n\nthat our proposed models have better generalization, which is the result of including random latent\nvariables in our model. We note that our proposed methods improve new link prediction more\nsubstantially which shows that they can capture temporal trends better than the competing methods.\nComparing VGRNN with SI-VGRNN shows that the prediction results are almost the same for all\ndatasets. The reason is that although the posterior is more \ufb02exible in SI-VGRNN, the prior on which\nour predictions are based, is still Gaussian, hence the improvement is marginal. A possible avenue\nfor further improvements is constructing more \ufb02exible priors such as semi-implicit priors proposed\nby Molchanov et al. [17], which we leave for future studies.\nTo \ufb01nd out when VGRNN and SI-VGRNN show more improvements compared to the baselines,\nwe take a closer look at three of the datasets. Figure 2 shows the temporal evolution of density and\nclustering coef\ufb01cients of COLAB, Enron, and Facebook datasets. Enron shows the highest density\nand clustering coef\ufb01cients, indicating that it contains dense clusters who are densely connected with\neach other. COLAB have low density and high clustering coef\ufb01cients across time, which means that\nalthough it is very sparse but edges are mostly within the clusters. Facebook, which has both low\ndensity and clustering coef\ufb01cients, is very sparse with almost no clusters. Looking back at (new)\nlink prediction results, we see that the improvement margin of (SI-)VGRNN compared to competing\nmethods is more substantial for Facebook. Moreover, the improvement margin diminishes when\nthe graph has more clusters and is more dense. Predicting the evolution very sparse graphs with\nno clusters is indeed a very dif\ufb01cult task (arguably more dif\ufb01cult than dense graphs), in which our\nproposed (SI-)VGRNN is very successful. The stochastic latent variables in our models can capture\nthe temporal trend while other methods tend to over\ufb01t very few observed links.\n\n8\n\n0246810Snapshot0.0750.1000.1250.1500.1750.2000.2250.250Clustering CoefficientCOLABEnronFacebook0246810Snapshot0.0040.0060.0080.0100.0120.0140.016DensityCOLABEnronFacebook\fFigure 3: Evolution of simulated graph topology through time.\n\nFigure 4: Latent representations of the simulated graph in different time steps in 2-d space using VGRNN.\n4.2\n\nInterpretable latent representations\n\nTo show that VGRNN learns more interpretable latent representations, we simulated a dynamic graph\nwith three communities in which a node (red colored node) transfers from one community into another\nin two time steps (Figure 3). We embedded the node into 2-d latent space using VGRNN (Figure 4)\nand DynAERNN (the best performed baseline; Figure S1 in the supplementary material). While the\nadvantages of modeling uncertainty for latent representations and its relation to node labels (classes)\nfor static graphs have been discussed in Bojchevski and G\u00fcnnemann [2], we argue that the uncertainty\nis also directly related to topological evolution in dynamic graphs.\nMore speci\ufb01cally, the variance of the latent variables for the node of interest increases in time (left to\nright) marked with the red contour. In time steps 2 and 3 (where the node is moving in the graph),\nthe information from previous and current time steps contradicts each other; hence we expect the\nrepresentation uncertainty to increase. We also plotted the variance of a node whose community\ndoesn\u2019t change in time (marked with the green contour). As we expected, the variance of this\nnode does not increase over time. We argue that the uncertainty helps to better encode non-smooth\nevolution, in particular abrupt changes, in dynamic graphs. Moreover, at time step 2, the moving node\nhave multiple edges with nodes in two communities. Considering the inner-product decoder, which is\nbased on the angle between the latent representations, the moving node can be connected to both of\nthe communities which is consistent with the graph topology. We note that DynAERNN (Figure S1)\nfails to produce such an interpretable latent representation. We can see that VGRNN can separate the\ncommunities in the latent space more distinctively than what DynAERNN does.\n\n5 Conclusion\n\nWe have proposed VGRNN and SI-VGRNN, the \ufb01rst node embedding methods for dynamic graphs\nthat embed each node to a random vector in the latent space. We argue that adding high level latent\nvariables to graph recurrent neural networks not only increases its expressiveness to better model\nthe complex dynamics of graphs, but also generates interpretable random latent representation for\nnodes. SI-VGRNN is also developed by combining VGRNN and semi-implicit variational inference\nfor \ufb02exible variational inference. We have tested our proposed methods on dynamic link prediction\ntasks and they outperform competing methods substantially, specially for very sparse graphs.\n\n6 Acknowledgments\n\nThe presented materials are based upon the research supported by the National Science Foundation\nunder Grants ENG-1839816, IIS-1848596, CCF-1553281, IIS-1812641 and IIS-1812699. We also\nthank Texas A&M High Performance Research Computing and Texas Advanced Computing Center\nfor providing computational resources to perform experiments in this work.\n\n9\n\n\fReferences\n[1] Mohammadreza Armandpour, Patrick Ding, Jianhua Huang, and Xia Hu. Robust negative sam-\npling for network embedding. In Proceedings of the AAAI Conference on Arti\ufb01cial Intelligence,\nvolume 33, pages 3191\u20133198. AAAI, 2019.\n\n[2] Aleksandar Bojchevski and Stephan G\u00fcnnemann. Deep gaussian embedding of graphs: Unsuper-\nvised inductive learning via ranking. In International Conference on Learning Representations,\n2018. URL https://openreview.net/forum?id=r1ZdKJ-0W.\n\n[3] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua\nBengio. A recurrent latent variable model for sequential data. In Advances in neural information\nprocessing systems, pages 2980\u20132988, 2015.\n\n[4] Micha\u00ebl Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks\non graphs with fast localized spectral \ufb01ltering. In Advances in Neural Information Processing\nSystems, pages 3844\u20133852, 2016.\n\n[5] Claire Donnat, Marinka Zitnik, David Hallac, and Jure Leskovec. Learning structural node\nembeddings via diffusion wavelets. In International ACM Conference on Knowledge Discovery\nand Data Mining (KDD), volume 24, 2018.\n\n[6] Marco Fraccaro, S\u00f8ren Kaae S\u00f8 nderby, Ulrich Paquet, and Ole Winther. Sequential\nneural models with stochastic layers.\nIn D. D. Lee, M. Sugiyama, U. V. Luxburg,\nI. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29,\npages 2199\u20132207. Curran Associates, Inc., 2016. URL http://papers.nips.cc/paper/\n6039-sequential-neural-models-with-stochastic-layers.pdf.\n\n[7] Marco Fraccaro, S\u00f8ren Kaae S\u00f8nderby, Ulrich Paquet, and Ole Winther. Sequential neural\nmodels with stochastic layers. In Advances in neural information processing systems, pages\n2199\u20132207, 2016.\n\n[8] Alias Parth Goyal, Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre C\u00f4t\u00e9, Nan Rose-\nmary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks.\nIn\nI. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and\nR. Garnett, editors, Advances in Neural Information Processing Systems 30, pages\n6713\u20136723. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/\n7248-z-forcing-training-stochastic-recurrent-networks.pdf.\n\n[9] Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre C\u00f4t\u00e9, Nan Rosemary Ke, and Yoshua\nBengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information\nprocessing systems, pages 6713\u20136723, 2017.\n\n[10] Palash Goyal, Nitin Kamra, Xinran He, and Yan Liu. Dyngem: Deep embedding method for\n\ndynamic graphs. arXiv preprint arXiv:1805.11273, 2018.\n\n[11] Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. dyngraph2vec: Capturing network\n\ndynamics using dynamic graph representation learning. Knowledge-Based Systems, 2019.\n\n[12] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks.\n\nIn\nProceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and\ndata mining, pages 855\u2013864. ACM, 2016.\n\n[13] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large\n\ngraphs. In Advances in Neural Information Processing Systems, pages 1024\u20131034, 2017.\n\n[14] Thomas N Kipf and Max Welling. Variational graph auto-encoders.\n\narXiv:1611.07308, 2016.\n\narXiv preprint\n\n[15] Thomas N Kipf and Max Welling. Semi-supervised classi\ufb01cation with graph convolutional\n\nnetworks. In International Conference on Learning Representations, 2017.\n\n[16] Jundong Li, Harsh Dani, Xia Hu, Jiliang Tang, Yi Chang, and Huan Liu. Attributed network\nIn Proceedings of the 2017 ACM on\n\nembedding for learning in a dynamic environment.\nConference on Information and Knowledge Management, pages 387\u2013396. ACM, 2017.\n\n10\n\n\f[17] Dmitry Molchanov, Valery Kharitonov, Artem Sobolev, and Dmitry Vetrov. Doubly semi-\n\nimplicit variational inference. arXiv preprint arXiv:1810.02789, 2018.\n\n[18] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,\nZeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in\npytorch. In NIPS-W, 2017.\n\n[19] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre-\nsentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge\ndiscovery and data mining, pages 701\u2013710. ACM, 2014.\n\n[20] Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node\nrepresentations from structural identity. In Proceedings of the 23rd ACM SIGKDD International\nConference on Knowledge Discovery and Data Mining, pages 385\u2013394. ACM, 2017.\n\n[21] Youngjoo Seo, Micha\u00ebl Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured\nsequence modeling with graph convolutional recurrent networks. In International Conference\non Neural Information Processing, pages 362\u2013373. Springer, 2018.\n\n[22] Samira Shabanian, Devansh Arpit, Adam Trischler, and Yoshua Bengio. Variational bi-lstms.\n\narXiv preprint arXiv:1711.05717, 2017.\n\n[23] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-\nscale information network embedding. In Proceedings of the 24th International Conference\non World Wide Web, pages 1067\u20131077. International World Wide Web Conferences Steering\nCommittee, 2015.\n\n[24] Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Dyrep: Learning\nrepresentations over dynamic graphs. In International Conference on Learning Representations,\n2019.\n\n[25] Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International\n\nConference on Machine Learning, pages 5660\u20135669, 2018.\n\n[26] Lekui Zhou, Yang Yang, Xiang Ren, Fei Wu, and Yueting Zhuang. Dynamic network embedding\nby modeling triadic closure process. In Thirty-Second AAAI Conference on Arti\ufb01cial Intelligence,\n2018.\n\n11\n\n\f", "award": [], "sourceid": 5712, "authors": [{"given_name": "Ehsan", "family_name": "Hajiramezanali", "institution": "Texas A&M University"}, {"given_name": "Arman", "family_name": "Hasanzadeh", "institution": "Texas A&M University"}, {"given_name": "Krishna", "family_name": "Narayanan", "institution": "Texas A&M University"}, {"given_name": "Nick", "family_name": "Duffield", "institution": "Texas A&M University"}, {"given_name": "Mingyuan", "family_name": "Zhou", "institution": "University of Texas at Austin"}, {"given_name": "Xiaoning", "family_name": "Qian", "institution": "Texas A&M"}]}