{"title": "Region-specific Diffeomorphic Metric Mapping", "book": "Advances in Neural Information Processing Systems", "page_first": 1098, "page_last": 1108, "abstract": "We introduce a region-specific diffeomorphic metric mapping (RDMM) registration approach. RDMM is non-parametric, estimating spatio-temporal velocity fields which parameterize the sought-for spatial transformation. Regularization of these velocity fields is necessary. In contrast to existing non-parametric registration approaches using a fixed spatially-invariant regularization, for example, the large displacement diffeomorphic metric mapping (LDDMM) model, our approach allows for spatially-varying regularization which is advected via the estimated spatio-temporal velocity field. Hence, not only can our model capture large displacements, it does so with a spatio-temporal regularizer that keeps track of how regions deform, which is a more natural mathematical formulation. We explore a family of RDMM registration approaches: 1) a registration model where regions with separate regularizations are pre-defined (e.g., in an atlas space or for distinct foreground and background regions), 2) a registration model where a general spatially-varying regularizer is estimated, and 3) a registration model where the spatially-varying regularizer is obtained via an end-to-end trained deep learning (DL) model. We provide a variational derivation of RDMM, showing that the model can assure diffeomorphic transformations in the continuum, and that LDDMM is a particular instance of RDMM. To evaluate RDMM performance we experiment 1) on synthetic 2D data and 2) on two 3D datasets: knee magnetic resonance images (MRIs) of the Osteoarthritis Initiative (OAI) and computed tomography images (CT) of the lung. Results show that our framework achieves comparable performance to state-of-the-art image registration approaches, while providing additional information via a learned spatio-temporal regularizer. Further, our deep learning approach allows for very fast RDMM and LDDMM estimations. Code is available at https://github.com/uncbiag/registration.", "full_text": "Region-speci\ufb01c Diffeomorphic Metric Mapping\n\nZhengyang Shen\nUNC Chapel Hill\nzyshen@cs.unc.edu\n\nFran\u00e7ois-Xavier Vialard\n\nLIGM, UPEM\n\nfrancois-xavier.vialard@u-pem.fr\n\nMarc Niethammer\nUNC Chapel Hill\nmn@cs.unc.edu\n\nAbstract\n\nWe introduce a region-speci\ufb01c diffeomorphic metric mapping (RDMM) registra-\ntion approach. RDMM is non-parametric, estimating spatio-temporal velocity\n\ufb01elds which parameterize the sought-for spatial transformation. Regularization of\nthese velocity \ufb01elds is necessary. In contrast to existing non-parametric registra-\ntion approaches using a \ufb01xed spatially-invariant regularization, for example, the\nlarge displacement diffeomorphic metric mapping (LDDMM) model, our approach\nallows for spatially-varying regularization which is advected via the estimated\nspatio-temporal velocity \ufb01eld. Hence, not only can our model capture large dis-\nplacements, it does so with a spatio-temporal regularizer that keeps track of how\nregions deform, which is a more natural mathematical formulation. We explore a\nfamily of RDMM registration approaches: 1) a registration model where regions\nwith separate regularizations are pre-de\ufb01ned (e.g., in an atlas space or for dis-\ntinct foreground and background regions), 2) a registration model where a general\nspatially-varying regularizer is estimated, and 3) a registration model where the\nspatially-varying regularizer is obtained via an end-to-end trained deep learning\n(DL) model. We provide a variational derivation of RDMM, showing that the model\ncan assure diffeomorphic transformations in the continuum, and that LDDMM is a\nparticular instance of RDMM. To evaluate RDMM performance we experiment\n1) on synthetic 2D data and 2) on two 3D datasets: knee magnetic resonance\nimages (MRIs) of the Osteoarthritis Initiative (OAI) and computed tomography\nimages (CT) of the lung. Results show that our framework achieves comparable\nperformance to state-of-the-art image registration approaches, while providing\nadditional information via a learned spatio-temporal regularizer. Further, our deep\nlearning approach allows for very fast RDMM and LDDMM estimations. Code is\navailable at https://github.com/uncbiag/registration.\n\n1\n\nIntroduction\n\nQuantitative analysis of medical images frequently requires the estimation of spatial correspondences,\ni.e.image registration. For example, one may be interested in capturing knee cartilage changes\nover time, localized changes of brain structures, or how organs at risk move between planning\nand treatment for radiation treatment. Speci\ufb01cally, image registration seeks to estimate the spatial\ntransformation between a source image and a target image, subject to a chosen transformation model.\nTransformations can be parameterized via low-dimensional parametric models (e.g., an af\ufb01ne transfor-\nmation), but more \ufb02exible models are required to capture subtle local deformations. Such registration\nmodels [3, 32] may have large numbers of parameters, e.g., a large number of B-spline control\npoints [30] or may even be non-parametric where vector \ufb01elds are estimated [5, 22]. Spatial regu-\nlarity can be achieved by appropriate constraints on displacement \ufb01elds [14] or by parameterizing\nthe transformation via integration of a suf\ufb01ciently regular stationary or time-dependent velocity\n\ufb01eld [5, 15, 38, 8, 41]. Given suf\ufb01cient regularization, diffeomorphic transformations can be assured\nin the continuum. A popular approach based on time-dependent velocity \ufb01elds is LDDMM [5, 15].\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fOptimal LDDMM solutions are geodesics and minimize a geodesic distance. Consequentially, one\nmay directly optimize over a geodesic\u2019s initial conditions in a shooting approach [40].\nMost existing non-parametric image registration approaches use spatially-invariant regularizers.\nHowever, this may not be realistic. E.g., when registering inhale to exhale images of a lung one\nexpects large deformations of the lung, but not of the surrounding tissue. Hence, a spatially-varying\nregularization would be more appropriate. As the regularizer encodes the deformation model this\nthen allows anticipating different levels of deformation at different image locations.\nWhile spatially-varying regularizers may be used in LDDMM variants [31] existing approaches\ndo not allow for time-varying spatially-varying regularizers. However, such regularizers would be\nnatural for large displacement as they can move with a deforming image. Hence, we propose a family\nof registration approaches with spatio-temporal regularizers based on advecting spatially-varying\nregularizers via an estimated spatio-temporal velocity \ufb01eld. Speci\ufb01cally, we extend LDDMM theory,\nwhere the original LDDMM model becomes a special case. In doing so, our entire model, including\nthe spatio-temporal regularizer is expressed via the initial conditions of a partial differential equation.\nWe propose three different approaches based on this model: 1) A model for which the regularizer is\nspeci\ufb01ed region-by-region. This would, for example, be natural when registering a labeled atlas image\nto a target image, as illustrated in Fig. 1. 2) A model in which the regularizer is estimated jointly with\nthe spatio-temporal velocity \ufb01elds. 3) A deep learning model which predicts the regularizer and the\ninitial velocity \ufb01eld, thereby resulting in a very fast registration approach.\nRelated Work Only limited work [26, 27, 36, 39] on spatially-varying regularizers exists. The most\nrelated work [23] learns a spatially-varying regularizer for a stationary velocity \ufb01eld registration\nmodel.\nIn [23] both the velocity \ufb01eld and the regularizer are assumed to be constant in time.\nIn contrast, both are time-dependent in RDMM. Spatially-varying regularization has also been\naddressed from a Bayesian view in [34] by putting priors on B-spline transformation parameters.\nHowever, metric estimation in [34] is in a \ufb01xed atlas-space, whereas RDMM addresses general\npairwise image registration.\n\nContributions: 1) We propose RDMM, a new registration model for large diffeomorphic deforma-\ntions with a spatio-temporal regularizer capable of following deforming objects and thereby providing\na more natural representation of deformations than existing non-parametric models, such as LDDMM.\n2) Via a variational formulation we derive shooting equations that allow specifying RDMM solutions\nentirely based on their initial conditions: an initial momentum \ufb01eld and an initial spatially-varying\nregularizer. 3) We prove that diffeomorphisms can be obtained for RDMM in the continuum for\nsuf\ufb01ciently regular regularizers. 4) We explore an entire new family of registration models based on\nRDMM and provide optimization-based and very fast deep-learning-based approaches to estimate\nthe initial conditions of these registration models. 5) We demonstrate the utility of our approach via\nexperiments on synthetic data and on two 3D medical image datasets.\n\n2 Standard LDDMM Model\n\nLDDMM [5] is a non-parametric registration approach based on principles from \ufb02uid mechanics. It\nis based on the estimation of a spatio-temporal velocity \ufb01eld v(t, x) from which the sought-for spatial\ntransformation \u03d5 can be computed via integration of \u2202t\u03d5(t, x) = v(t, \u03d5(t, x)) . For appropriately\nregularized velocity \ufb01elds [13], diffeomorphic transformations can be guaranteed. The optimization\nproblem underlying LDDMM for images can be written as (\u2207 being the gradient; (cid:104)\u00b7,\u00b7(cid:105) indicating the\ninner product)\n\n(cid:90) 1\n\n0\n\nv\u2217 = argmin\n\nv\n\n1\n2\n\n(cid:107)v(t)(cid:107)2\n\nL dt + Sim(I(1), I1),\n\ns.t. \u2202tI + (cid:104)\u2207I, v(cid:105) = 0; I(0) = I0 .\n\n(2.1)\n\nHere, the goal is to register the source image I0 to the target image I1 in unit time. Sim(A, B) is a\nsimilarity measure between images, often sum of squared differences, normalized cross correlation,\nor mutual information. Furthermore, we note that I(1, y) = I0 \u25e6 \u03d5\u22121(1, y), where \u03d5\u22121 denotes the\ninverse of \u03d5 in the target image space. The evolution of this map can be expressed as\n\n\u2202t\u03d5\u22121 + D\u03d5\u22121v = 0 ,\n\n(2.2)\n\n2\n\n\fFigure 1: RDMM registration example. The goal is to register the dark blue area with high \ufb01delity\n(i.e., allowing large local deformations), while assuring small deformations in the cyan area. Initially\n(t = 0), a spatially-varying regularizer (\ufb01fth column) is speci\ufb01ed in the source image space, where\ndark blue indicates small regularization and yellow large regularization. Speci\ufb01cally, regularizer\nvalues indicate effective local standard deviations of a local multi-Gaussian regularizer. Since the\ntransformation map and the regularizer are both advected according to the estimated velocity \ufb01eld,\nthe shape of the regularizer follows the shape of the deforming dark blue region and is of the same\nshape as the region of interest in the target space at the \ufb01nal time (t = 1) (as can be seen in the\nsecond and the last columns). Furthermore, objects inside the dark blue region are indeed aligned\nwell, whereas objects in the cyan region were not strongly deformed due to the larger regularization\nthere.\n\nwhere D denotes the Jacobian. Equivalently, in Eq. (2.1), we directly advect the image [15, 40] via\n\u2202tI + (cid:104)\u2207I, v(cid:105) = 0. To assure smooth transformations, LDDMM penalizes non-smooth velocity\n\ufb01elds via the norm (cid:107)v(cid:107)2\nAt optimality of Eq. (2.1) the Euler-Lagrange equations are (div denoting the divergence)\n\u2202tI+(cid:104)\u2207I, v(cid:105) = 0, I(0) = I0; \u2202t\u03bb+div(\u03bbv) = 0, \u03bb(1) =\n\nL = (cid:104)Lv, v(cid:105), where L is a differential operator.\n\n(2.3)\nHere, \u03bb is the adjoint variable to I, also known as the scalar momentum [15, 40]. As L\u22121 is a\nsmoother it is often chosen as a convolution, i.e., v = K (cid:63) (\u03bb\u2207I). Note that m(t, x) := \u03bb\u2207I is the\nvector-valued momentum and thus v = K (cid:63) m. One can directly optimize over v solving Eq. (2.3) [5]\nor regard Eq. (2.3) as a constraint de\ufb01ning a geodesic path [40] and optimize over all such solutions\nsubject to a penalty on the initial scalar momentum as well as the similarity measure. Alternatively,\none can express 1 these equations entirely with respect to the vector-valued momentum, m, resulting\nin the Euler-Poincar\u00e9 equation for diffeomorphisms (EPDiff) [44]:\n\n\u03b4\n\nSim(I(1), I1) = 0; v = L\u22121(\u03bb\u2207I).\n\n\u03b4I(1)\n\n\u2202tm + div(v)m + DvT (m) + Dm(v) = 0, m(0) = m0, v = K (cid:63) m ,\n\n(2.4)\n\nwhich de\ufb01nes the evolution of the spatio-temporal velocity \ufb01eld based on the initial condition, m0,\nof the momentum, from which the transformation \u03d5 can be computed using Eq. (2.2). Both (2.3)\nand (2.4) can be used to implement shooting-based LDDMM [40, 35]. As LDDMM preserves the\nmomentum, (cid:107)v(cid:107)2\n\nL is constant over time and hence a shooting formulation can be written as\n\nm(0)\u2217 = argmin\n\nm(0)\n\n1\n2\n\n(cid:107)v(0)(cid:107)2\n\nL + Sim(I(1), I1),\n\n(2.5)\n\nsubject to the EPDiff equation (2.4) including the advection of \u03d5\u22121, where I(1) = I0 \u25e6 \u03d5\u22121(1).\nA shooting-formulation has multiple bene\ufb01ts: 1) it allows for a compact representation of \u03d5 via its\ninitial conditions; 2) as the optimization is w.r.t. the initial conditions, a solution is a geodesic by\nconstruction; 3) these initial conditions can be predicted via deep-learning, resulting in very fast\nregistration algorithms which inherit the theoretical properties of LDDMM [43, 42]. We therefore\nuse this formulation as the starting point for RDMM in Sec 3.\n\n1Thm. (2) in suppl. 7.2 provides a more generalized derivation.\n\n3\n\n\f3 Region-Speci\ufb01c Diffeomorphic Metric Mapping (RDMM)\n\nIn standard LDDMM approaches, the regularizer L (equivalently the kernel K) is spatially invariant.\nWhile recent work introduced spatially-varying metrics in LDDMM, for stationary velocity \ufb01elds, or\nfor displacement-regularized registration [27, 31, 23, 26], all of these approaches use a temporally\n\ufb01xed regularizer. Hereafter, we generalize LDDMM by advecting a spatially-varying regularizer via\nthe estimated spatio-temporal velocity \ufb01eld. Standard LDDMM is a special case of our model.\nFollowing [23], we introduce (Vi)i=0,...,N\u22121 a \ufb01nite family of reproducing kernel Hilbert spaces\n(RKHS) which are de\ufb01ned by the pre-de\ufb01ned Gaussian kernels K\u03c3i with \u03c30 < . . . < \u03c3N\u22121. We use\na partition of unity wi(x, t), i = 0, . . . , N \u2212 1, on the image domain. As we want the kernel K\u03c3i to\ni=0 wi\u03bdi for \u03bdi \u2208 Vi.\n\nbe active on the region determined by wi we introduce the vector \ufb01eld v =(cid:80)N\u22121\n\nOn this new space of vector \ufb01elds, there exists a natural RKHS structure de\ufb01ned by\n\nwhose kernel is K =(cid:80)N\u22121\n\n(cid:107)v(cid:107)2\n\nL := inf\n\n(cid:107)\u03bdi(cid:107)2\n\nVi\n\n| v =\n\ni=0\n\ni=0\n\nwi\u03bdi\n\n,\n\n(3.1)\n\ni=0 wiK\u03c3iwi. Thus the velocity reads (see suppl. 7.1 for the derivation)\n\n(cid:41)\n\nN\u22121(cid:88)\n\nv = K (cid:63) m def.=\n\nwiK\u03c3i (cid:63) (wim), wi \u2265 0 ,\n\n(3.2)\n\n(cid:40)N\u22121(cid:88)\n\nN\u22121(cid:88)\n\ni=0\n\nwhich can capture multi-scale aspects of deformation [28]. In LDDMM, the weights are constant\nand pre-de\ufb01ned. Here, we allow spatially-dependent weights wi(x). In particular (see formulation\nbelow), we advect them via, v(t, x) thereby making them spatio-temporal, i.e., wi(t, x). In this setup,\nweights only need to be speci\ufb01ed at initial time t = 0. As the Gaussian kernels are \ufb01xed convolutions\ncan still be ef\ufb01ciently computed in the Fourier domain.\nWe prove (see suppl. 7.4 for the proof) that, for suf\ufb01ciently smooth weights wi, the velocity \ufb01eld\nis bounded and its \ufb02ow is a diffeomorphism. Following [23], to assure the smoothness of the\ninitial weights we instead optimize over initial pre-weights, hi(0, x) \u2265 0, s.t. wi(0, x) = G\u03c3 (cid:63)\nhi(0, x), where G\u03c3 is a \ufb01xed Gaussian with a small standard deviation, \u03c3. In addition, we constrain\ni (0, x) to locally sum to one. The optimization problem for our RDMM model then becomes\n\n(cid:80)N\u22121\n\ni=0 h2\n\nv\u2217,{h\u2217\n\ni } = argmin\nv,{hi}\n\n1\n2\n\nsubject to the constraints\n\n(cid:90) 1\n\n0\n\n(cid:107)v(t)(cid:107)2\n\nL dt + Sim(I(1), I1) + Reg({hi(0)}) ,\n\n\u2202tI + (cid:104)\u2207I, v(cid:105) = 0, I(0) = I0; \u2202thi + (cid:104)\u2207hi, v(cid:105) = 0, hi(0) = (hi)0;\n\nN\u22121(cid:88)\n\n\u03bdi = K\u03c3i (cid:63) (wim); v =\n\nwi\u03bdi; wi = G\u03c3 (cid:63) hi .\n\nAs for LDDMM, we can compute the optimality conditions for Eq. (3.3) which we use for shooting.\nTheorem 1 (Image-based RDMM optimality conditions). With the adjoints \u03b3i (for hi) and \u03bb (for I)\n\nand the momentum m := \u03bb\u2207I +(cid:80)N\u22121\n\ni=0 \u03b3i\u2207hi the optimality conditions for (3.3) are:\n\ni=0\n\n\u2202tI + (cid:104)\u2207I, v(cid:105) = 0, I(0) = I0; \u2202t\u03bb + div(\u03bbv) = 0, \u2212 \u03bb(1) +\n\n\u03b4\n\nSim(I(1), I1) = 0;\n\n\u03b4I(1)\n\n\u2202thi + (cid:104)\u2207hi, v(cid:105) = 0, hi(0) = (hi)0;\n\n\u2202t\u03b3i + div(\u03b3iv) = G\u03c3 (cid:63) (m \u00b7 \u03bdi), \u03b3i(0) +\n\n\u03b4\n\n\u03b4hi(0)\n\nReg({hi(0)}) = 0 .\n\nsubject to\n\n\u03bdi = K\u03c3i (cid:63) (wim); v =\n\nN\u22121(cid:88)\n\ni=0\n\n4\n\nwi\u03bdi; wi = G\u03c3 (cid:63) hi .\n\n(3.3)\n\n(3.4)\n\n(3.5)\n\n(3.6)\n\n(3.7)\n\n(3.8)\n\n\f\u2202t\u03d5\u22121 + D\u03d5\u22121v = 0, \u03d5\u22121(0, x) = x ,\n\nN\u22121(cid:88)\n\n(3.9)\n\n(3.10)\n\nTheorem 2 (Momentum-based RDMM optimality conditions). The RDMM optimality conditions of\nThm. (1) can be written entirely w.r.t. the momentum (as de\ufb01ned in Thm (1)). They are:\n\n\u2202tm + div(v)m + DvT (m) + Dm(v) =\n\nG\u03c3 (cid:63) (m \u00b7 \u03bdi)\u2207hi, m(0) = m0 ,\n\nwhere hi(t, x) = hi(0, x) \u25e6 \u03d5(t, x)\u22121 and subject to the constraints of Eq. (3.8) which de\ufb01ne the\nrelationship between the velocity and the momentum.\n\ni=0\n\nFor spatially constant pre-weights, we recover EPDiff from the momentum-based RDMM optimality\nconditions. Instead of advecting hi via \u03d5(t, x)\u22121, we can alternatively advect the pre-weights directly,\nas \u2202thi +(cid:104)\u2207hi, v(cid:105) = 0, hi(0) = (hi)0. For the image-based and the momentum-based formulations,\n\nthe velocity \ufb01eld v is obtained by smoothing the momentum, v =(cid:80)N\u22121\n\ni=0 wiK\u03c3i (cid:63) (wim).\n\nRegularization of the Regularizer\nGiven the standard deviations \u03c30 < . . . < \u03c3N\u22121 of Eq. (3.2), assigning larger weights to the Gaus-\nsians with larger standard deviation will result in smoother (i.e., more regular) and therefore simpler\ntransformations. To encourage choosing simpler transformations we follow [23], where a simple\noptimal mass transport (OMT) penalty on the weights is used. Such an OMT penalty is sensible as the\nnon-negative weights, {wi}, sum to one and can therefore be considered a discrete probability distri-\nbution. The chosen OMT penalty is designed to measure differences from the probability distribution\nassigning all weight to the Gaussian with the largest standard deviation, i.e., wN\u22121 = 1 and all other\nweights being zero. Speci\ufb01cally, the OMT penalty of [23] is\n,\nwhere s is a chosen power. To make the penalty more consistent with penalizing weights for a\nstandard multi-Gaussian regularizer (as our regularizer contains effectively the weights squared)\nwe do not penalize the weights directly, but instead penalize their squares using the same form of\nOMT penalty. Further, as the regularization only affects the initial conditions for the pre-weights, the\nevolution equations for the optimality conditions (i.e., the modi\ufb01ed EPDiff equation) do not change.\nAdditional regularizers, such as total variation terms as proposed in [23], are possible and easy to\nintegrate into our RDMM framework as they only affect initial conditions. For simplicity, we focus\non regularizing via OMT.\nShooting Formulation\nAs energy is conserved (see suppl. 7.3 for the proof) the momentum-based shooting formulation\nbecomes\n\n(cid:12)(cid:12)(cid:12)\u2212s(cid:80)N\u22121\n\n(cid:12)(cid:12)(cid:12)log \u03c3N\u22121\n\n(cid:12)(cid:12)(cid:12)log \u03c3N\u22121\n\n\u03c30\n\ni=0 wi\n\n\u03c3i\n\n(cid:12)(cid:12)(cid:12)s\n\nm(0)\u2217,{hi(0)\u2217} = argmin\nm(0),{hi(0)}\n\n1\n2\n\n(cid:107)v(0)(cid:107)2\n\nL + Sim(I(1), I1) + Reg({hi(0)}) ,\n\n(3.11)\n\nsubject to the evolution equations of Thm. 2. Similarly, the shooting formulation can use the\nimage-based evolution equations of Thm. 1 where optimization would be over \u03bb(0) instead of m(0).\n\n4 Learning Framework\n\nThe parameters for RDMM, i.e., the initial momentum and the initial pre-weights, can be obtained by\nnumerical optimization, either over the momentum and the pre-weights or only over the momentum\nif the pre-weights are prescribed. Just as for the LDDMM model, such a numerical optimization\nis computationally costly. Consequentially, various deep-learning (DL) approaches have been\nproposed to instead predict displacements [4, 6], stationary velocity [29] or momentum \ufb01elds [43, 23].\nSupervised [43, 42] and unsupervised [16, 11, 18, 4, 10] DL registration approaches exist. All\nof them are fast as only a regression solution needs to be evaluated at test time and no further\nnumerical optimization is necessary. Additionally, such DL models bene\ufb01t from learning over an\nentire population instead of relying only on information from given image-pairs.\nMost non-parametric registration approaches are not invariant to af\ufb01ne transformations based on\nthe chosen regularizers. Hence, for such non-parametric methods, af\ufb01ne registration is typically\nperformed \ufb01rst as a pre-registration step to account for large, global displacements or rotations.\nSimilarly, we make use of a two-step learning framework (Fig. 2 (left)) learning af\ufb01ne transformations\n\n5\n\n\fFigure 2: Illustration of the learning framework (left) and its non-parametric registration component\n(right). A multi-step af\ufb01ne-network \ufb01rst predicts the af\ufb01ne transformation map [33] followed by\nan iterable non-parametric registration to estimate the \ufb01nal transformation map. The LDDMM\ncomponent uses one network to generate the initial momentum. RDMM also uses a second network\nto predict the initial regularizer pre-weights. We integrate the RDMM evolution equations at low-\nresolution (based on the predicted initial conditions) to save memory. The \ufb01nal transformation map is\nobtained via upsampling. See suppl. 7.6 for the detailed structure of the LDDMM/RDMM units.\n\nand subsequent non-parametric deformations separately. For the af\ufb01ne part, a multi-step af\ufb01ne\nnetwork is used to predict the af\ufb01ne transformation following [33]. For the non-parametric part, we\nuse two different deep learning approaches, illustrated in the right part of Fig. 2, to predict 1) the\nLDDMM initial momentum, m0, in Eq. (2.5) and 2) the RDMM initial momentum, m0, and the\npre-weights, {hi(0)\u2217}, in Eq. (3.11). Overall, including the af\ufb01ne part, we use two networks for\nLDDMM prediction and three networks for RDMM prediction.\nWe use low-resolution maps and map compositions for the momentum and the pre-weight networks.\nThis reduces computational cost signi\ufb01cantly. The \ufb01nal transformation map is obtained via up-\nsampling, which is reasonable as we assume smooth transformations. We use 3D UNets [9] for\nmomentum and pre-weight prediction. Both the af\ufb01ne and the non-parametric networks can be\niterated to re\ufb01ne the prediction results: i.e. the input source image and the initial map are replaced\nwith the currently warped image and the transformation map respectively for the next iteration.\nDuring training of the non-parametric part, the gradient is \ufb01rst backpropagated through the differen-\ntiable interpolation operator, then through the LDDMM/RDMM unit, followed by the momentum\ngeneration network and the pre-weight network.\nInverse Consistency: For the DL approaches we follow [33] and compute bidirectional (source to\ntarget denoted as st and target to source denoted as ts) registration losses and an additional symmetry\nloss, (cid:107)(\u03d5st)\u22121 \u25e6 (\u03d5ts)\u22121 \u2212 id(cid:107)2\n2, where id refers to the identity map. This encourages symmetric\nconsistency.\n\n5 Experimental Results and Setup\n\nDatasets: To demonstrate the behavior of RDMM, we evaluate the model on three datasets: 1) a\nsynthetic dataset for illustration, 2) a 3D computed tomography dataset (CT) of a lung, and 3) a large\n3D magnetic resonance imaging (MRI) dataset of the knee from the Osteoarthritis Initiative (OAI).\nThe synthetic dataset consists of three types of shapes (rectangles, triangles, and ellipses). There is\none foreground object in each image with two objects inside and at most \ufb01ve objects outside. Each\nsource image object has a counterpart in the target image; the shift, scale, and rotations are random.\nWe generated 40 image pairs of size 2002 for evaluation. Fig. 1 shows example synthetic images.\nThe lung dataset consists of 49 inspiration/expiration image pairs with lung segmentations. Each\nimage is of size 1603. We register from the expiration phase to the inspiration phase for all 49 pairs.\nThe OAI dataset consists of 176 manually labeled MRI from 88 patients (2 longitudinal scans per\npatient) and 22,950 unlabeled MR images from 2,444 patients. Labels are available for femoral and\ntibial cartilage. We divide the patients into training (2,800 pairs), validation (50 pairs) and testing\ngroups (300 pairs), with the same evaluation settings as for the cross-subject experiments in [33].\nDeformation models: Af\ufb01ne registration is performed before each LDDMM/RDMM registration.\n\n6\n\n\fAf\ufb01ne model: We implemented a multi-scale af\ufb01ne model solved via numerical optimization and a\nmulti-step deep neural network to predict the af\ufb01ne transformation parameters.\nFamily of non-parametric models: We implemented both optimization and deep-learning versions of\na family of non-parametric registration methods: a vector-momentum based stationary velocity \ufb01eld\nmodel (vSVF) (v(x), w = const), LDDMM (v(t, x), w = const), and RDMM (v(t, x), w(t, x)).\nWe use the dopri5 solver using the adjoint sensitivity method [7] to integrate the evolution equations in\ntime. For solutions based on numerical optimization, we use a multi-scale strategy with L-BGFS [19]\nas the optimizer. For the deep learning models, we compute solutions for a low-resolution map (factor\nof 0.5) which is then upsampled. We use Adam [17] for optimization.\nImage similarity measure: We use multi-kernel Localized Normalized Cross Correlation (mk-\nLNCC) [33]. mk-LNCC computes localized normalized cross correlation (NCC) with different\nwindow sizes and combines these measures via a weighted sum.\nWeight visualization: To illustrate the behavior of the RDMM model, we visualize the estimated\n\nstandard deviations, i.e.the square root of the local variance \u03c32(x) =(cid:80)N\u22121\n\ni=0 w2\n\ni .\ni (x)\u03c32\n\nEstimation approaches: To illustrate different aspects of our approach we perform three types\nof RDMM registrations: 1) registration with a pre-de\ufb01ned regularizer (Sec. 5.1), 2) registration\nwith simultaneous optimization of the regularizer, via optimization of the initial momentum and\npre-weights (Sec. 5.2), and 3) registration via deep learning predicting the initial momentum and\nregularizer pre-weights (Sec. 5.3). Detailed settings for all approaches are in the suppl. 7.8.\n\n5.1 Registration with a pre-de\ufb01ned regularizer\n\nTo illustrate the base capabilities of our models, we prescribe an initial spatially-varying regularizer\nin the source image space. We show experimental results for pair-wise registration of the synthetic\ndata as well as for the 3D lung volumes.\nFig. 1 shows the registration result for an example synthetic image pair. We use small regularization in\nthe blue area and large regularization in the surrounding area. As expected, most of the deformations\noccur inside the blue area as the regularizer is more permissive there. We also observe that the\nregularizer is indeed advected with the image. For the real lung image data we use a small regularizer\ninside the lung (as speci\ufb01ed by the given lung mask) and a large regularizer in the surrounding tissue.\nFig. 3 shows that most of the deformations are indeed inside the lung area while the deformation\noutside the lung is highly regularized as desired. We evaluate the Dice score between the warped lung\nand the target lung, achieving 95.22% on average (over all inhalation/exhalation pairs). Fig. 3 also\nshows the determinant of the Jacobian of the transformation map J\u03d5\u22121 (x) := |D\u03d5\u22121(x)| (de\ufb01ned in\ntarget space): the lung region shows small values (illustrating expansion) while other region are either\nvolume preserved (close to 1) or compressed (bigger than 1). Overall the deformations are smooth.\n\nFigure 3: RDMM lung registration result with a pre-de\ufb01ned regularizer. Lung images at expiration\n(source) are registered to the corresponding inspiration (target) images. Last column: Inside the\n0 = {0.1, 0.4, 0.5}) and\nlung a regularizer with small standard deviation (\u03c3i = {0.04, 0.06, 0.08} , h2\n0 = {1.0}). Deformations are\noutside the lung with large standard deviation is used (\u03c3i = {0.2}, h2\nlargely inside the lung, the surrounding tissue is well regularized as expected. Columns 1 to 3 show\nresults in image space while columns 4 to 6 refer to the results in label space. The second to last\ncolumn shows the determinant of the Jacobian of the spatial transformation, \u03d5\u22121.\n5.2 Registration with an optimized regularizer\n\nIn contrast to the experiments in Sec. 5.1, we jointly estimate the initial momentum and the initial\nregularizer pre-weights for pairwise registration. We use the synthetic data and the knee MRIs.\nFig. 4 shows that the registration warped every object in the source image to the corresponding\nobject in the target image. The visualization of the initial regularizer shows that low regularization is\n\n7\n\n\fFigure 4: Illustration of the RDMM registration results with an opti-\nmized regularizer on the synthetic dataset. All objects are warped from\nthe source image space to the target image space. The last two columns\nshow the regularizer (\u03c3(x)) at t = 0 and t = 1 respectively.\n\nFigure 5: Average Dice\nscores for all objects.\nLeft to right: SVF, LD-\nDMM and RDMM.\n\nNiftyReg-NMI[25, 20, 30, 21]\n\naf\ufb01ne-NiftyReg\n\nMethod\n\u2014\u2014\u2014\n\naf\ufb01ne-opt\naf\ufb01ne-net\n\u2014\u2014\u2014\u2013\n\nDemons[38, 37]\n\nSyN[2, 1]\n\nNiftyReg-LNCC\n\nvSVF-opt\n\nLDDMM-opt\nRDMM-opt\n\n\u2014\u2014\u2014-\n\nvSVF-net [33]\nLDDMM-net\nRDMM-net\n\nVoxelMorph[10](with aff)\n\nLearning-based Methods \u2014\u2014- \u2014\u2014 \u2014-\n\nOAI Dataset\nDice\n\nFolds Time (s)\nAf\ufb01ne Methods \u2014\u2014\u2014- \u2014\u2014\u2013\n30.43 (12.11)\n34.49 (18.07)\n44.58 (7.74)\n\n45\n8\n\n0.20\n\nOptimization Methods \u2014\u2014- \u2014\u2014\u2014-\n\n0\n0\n0\n\n0\n0\n\n0\n0\n\n0.56\n\n35.19\n\n17.37\n\n3.31\n0.39\n\n0\n\n0.47\n\n114\n1330\n143\n270\n79\n457\n627\n\n0.31\n0.62\n0.85\n1.1\n\n63.47 (9.52)\n65.71 (15.01)\n59.65 (7.62)\n67.92 (5.24)\n67.35 (9.73)\n67.72(8.94)\n68.18(8.36)\n\n66.08 (5.13)\n67.59 (4.47)\n67.63(4.51)\n67.94(4.40)\n\nFigure 6: Comparison of registration methods for cross-subject registrations on the OAI dataset\nbased on Dice scores. -opt and -net refer to optimization- and DL-based methods respectively. For\nall DL methods, we report performance after two-step re\ufb01nement. Folds refers to the absolute value\nof the sum of the determinant of the Jacobian in the folding area (i.e., where the determinant of the\nJacobian is negative); Time refers to the average registration time for a single image pair.\n\nassigned close to object edges making them locally deformable. The visualizations of the regularizer\nat the initial time point (t = 0) and the \ufb01nal time point (t = 1) show that it deforms with the image.\nThat low regularization values are localized is sensible as the OMT regularizer prefers spatially sparse\nsolutions (in the sense of sparsely assigning low levels of regularity). If the desired deformation\nmodel is piecewise constant our RDMM model could readily be combined with a total variation\npenalty as in [23]. Fig. 5 compares average Dice scores for all objects for vSVF, LDDMM and\nRDMM separately. They all achieve high and comparable performance indicating good registration\nquality. But only RDMM provides additional information about local regularity.\nWe further evaluate RDMM on 300 images pairs from the OAI dataset. The optimization methods\nsection of Tab. 6 compares registration performance for different optimization-based algorithms.\nRDMM achieves high performance. While RDMM is in theory diffeomorphic, we observe some\nfoldings, whereas no such foldings appear for LDDMM and SVF. This is likely due to inaccuracies\nwhen discretizing the evolution equations and when discretizing the determinant of the Jacobian.\nFurther, RDMM may locally exhibit stronger deformations than LDDMM or vSVF, especially when\nthe local regularization (via OMT) is small, making it numerically more challenging. Most of the\nfolds appear at the image boundary (and are hence likely due to boundary discretization artifacts) or\ndue to anatomical inconsistency in the source and target images, where large deformations may be\nestimated.\n\n5.3 Registration with a learnt regularizer via deep learning\n\nFinally, we evaluate our learning framework for non-parametric registration approaches on the OAI\ndataset. For vSVF, we follow [33] to predict the momentum. We implement the same approach for\nLDDMM, where the vSVF inference unit is replaced by one for LDDMM (i.e., instead of advecting\nvia a stationary velocity \ufb01eld we integrate EPDiff). Similarly, for RDMM, the inference unit is\nreplaced by the evolution equation for RDMM and we use an additional network to predict the\nregularizer pre-weights. Fig. 6 shows the results. The non-parametric DL models achieve comparable\n\n8\n\n\fFigure 7: Illustration of RDMM registration results on the OAI dataset. \"* s\" and \"* b\" refer to the\nregularizer with \u03c3 in G\u03c3 set to 0.04 and 0.06 respectively; \"learn *\" and \"opt *\" refer to a learnt\nregularizer and an optimized one respectively; \"Jaco\" refers to the absolute value of the determinant\nof the Jacobian; \"init_w\" refers to the initial weight map of the regularizer (as visualized via \u03c3(x)).\nThe \ufb01rst four columns refer to registration results in image space.\n\nperformance to their optimization counterparts, but are much faster, while learning-based RDMM\nsimultaneously predicts the initial regularizer which captures aspects of the knee anatomy. Fig. 7\nshows the determinant of the Jacobian of the transformation map. It is overall smooth and folds\nare much less frequent than for the corresponding optimization approach, because the DL model\npenalizes transformation asymmetry. Fig. 7 also clearly illustrates the bene\ufb01t of training the DL\nmodel based on a large image dataset: compared with the optimization approach (which only works\non individual image pairs), the initial regularizer predicted by the deep network captures anatomically\nmeaningful information much better: the bone (femur and tibia) and the surrounding tissue show\nlarge regularity.\n\n6 Conclusion and Future Work\n\nWe introduced RDMM, a generalization of LDDMM registration which allows for spatially-varying\nregularizers advected with a deforming image. In RDMM, both the estimated velocity \ufb01eld and\nthe estimated regularizer are time- and spatially-varying. We used a variational approach to derive\nshooting equations which generalize EPDiff and allow the parameterization of RDMM using only\nthe initial momentum and regularizer pre-weights. We also prove that diffeomorphic transformation\ncan be obtained for RDMM with suf\ufb01ciently regular regularizers. Experiments with pre-de\ufb01ned,\noptimized, and learnt regularizers show that RDMM is \ufb02exible and its solutions can be estimated\nquickly via deep learning.\nFuture work could focus on numerical aspects and explore different initial constraints, such as total-\nvariation constraints, depending on the desired deformation model. Indeed, a promising avenue of\nresearch consists in learning regularizers which include more physical/mechanical a-priori information\nin the deformation model. For instance, a possible \ufb01rst step in this direction consists in parameterizing\nnon-isotropic kernels to favor deformations in particular directions.\nAcknowledgements Research reported in this work was supported by the National Institutes of\nHealth (NIH) and the National Science Foundation (NSF) under award numbers NSF EECS-1711776\nand NIH 1R01AR072013. The content is solely the responsibility of the authors and does not\nnecessarily represent the of\ufb01cial views of the NIH or the NSF. We would also like to thank Dr. Ra\u00fal\nSan Jos\u00e9 Est\u00e9par for providing the lung data.\n\nReferences\n[1] B. B. Avants, C. L. Epstein, M. Grossman, and J. C. Gee. Symmetric diffeomorphic image registration\nwith cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical\nimage analysis, 12(1):26\u201341, 2008.\n\n[2] B. B. Avants, N. Tustison, and G. Song. Advanced normalization tools (ANTS). Insight j, 2:1\u201335, 2009.\n\n[3] R. Bajcsy and S. Kova\u02c7ci\u02c7c. Multiresolution elastic matching. CVGIP, 46(1):1\u201321, 1989.\n\n[4] G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag, and A. V. Dalca. An unsupervised learning model for\n\ndeformable medical image registration. In CVPR, pages 9252\u20139260, 2018.\n\n9\n\n\f[5] M. F. Beg, M. I. Miller, A. Trouv\u00e9, and L. Younes. Computing large deformation metric mappings via\n\ngeodesic \ufb02ows of diffeomorphisms. IJCV, 61(2):139\u2013157, 2005.\n\n[6] X. Cao, J. Yang, J. Zhang, Q. Wang, P.-T. Yap, and D. Shen. Deformable image registration using a\ncue-aware deep regression network. IEEE Transactions on Biomedical Engineering, 65(9):1900\u20131911,\n2018.\n\n[7] T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In\n\nAdvances in Neural Information Processing Systems, pages 6571\u20136583, 2018.\n\n[8] Z. Chen, H. Jin, Z. Lin, S. Cohen, and Y. Wu. Large displacement optical \ufb02ow from nearest neighbor \ufb01elds.\n\nIn CVPR, pages 2443\u20132450, 2013.\n\n[9] \u00d6. \u00c7i\u00e7ek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger. 3D U-Net: learning dense\nvolumetric segmentation from sparse annotation. In International conference on medical image computing\nand computer-assisted intervention, pages 424\u2013432. Springer, 2016.\n\n[10] A. V. Dalca, G. Balakrishnan, J. Guttag, and M. R. Sabuncu. Unsupervised learning for fast probabilistic\n\ndiffeomorphic registration. MICCAI, 2018.\n\n[11] B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and I. I\u0161gum. End-to-end unsupervised\ndeformable image registration with a convolutional neural network. In DLMIA, pages 204\u2013212. Springer,\n2017.\n\n[12] J. Delon, J. Salomon, and A. Sobolevski. Local matching indicators for transport problems with concave\n\ncosts. SIAM Journal on Discrete Mathematics, 26(2):801\u2013827, 2012.\n\n[13] P. Dupuis, U. Grenander, and M. I. Miller. Variational problems on \ufb02ows of diffeomorphisms for image\n\nmatching. Quarterly of applied mathematics, pages 587\u2013600, 1998.\n\n[14] E. Haber and J. Modersitzki. Image registration with guaranteed displacement regularity. International\n\nJournal of Computer Vision, 71(3):361\u2013372, 2007.\n\n[15] G. L. Hart, C. Zach, and M. Niethammer. An optimal control approach for deformable registration. In\n\nCVPR, pages 9\u201316. IEEE, 2009.\n\n[16] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In NIPS, pages 2017\u20132025,\n\n2015.\n\n[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2014.\n\n[18] H. Li and Y. Fan. Non-rigid image registration using fully convolutional networks with deep self-\n\nsupervision. ISBI, 2018.\n\n[19] D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical\n\nprogramming, 45(1-3):503\u2013528, 1989.\n\n[20] M. Modat, D. M. Cash, P. Daga, G. P. Winston, J. S. Duncan, and S. Ourselin. Global image registration\n\nusing a symmetric block-matching approach. Journal of Medical Imaging, 1(2):024003, 2014.\n\n[21] M. Modat, G. R. Ridgway, Z. A. Taylor, M. Lehmann, J. Barnes, D. J. Hawkes, N. C. Fox, and\nS. Ourselin. Fast free-form deformation using graphics processing units. Computer methods and programs\nin biomedicine, 98(3):278\u2013284, 2010.\n\n[22] J. Modersitzki. Numerical methods for image registration. Oxford University Press on Demand, 2004.\n\n[23] M. Niethammer, R. Kwitt, and F.-X. Vialard. Metric learning for image registration. CVPR, 2019.\n\n[24] J. Nocedal and S. Wright. Numerical optimization. Springer Science & Business Media, 2006.\n\n[25] S. Ourselin, A. Roche, G. Subsol, X. Pennec, and N. Ayache. Reconstructing a 3D structure from serial\n\nhistological sections. Image and vision computing, 19(1-2):25\u201331, 2001.\n\n[26] D. F. Pace, S. R. Aylward, and M. Niethammer. A locally adaptive regularization based on anisotropic\ndiffusion for deformable image registration of sliding organs. IEEE transactions on medical imaging,\n32(11):2114\u20132126, 2013.\n\n[27] L. Risser, F.-X. Vialard, H. Y. Baluwala, and J. A. Schnabel. Piecewise-diffeomorphic image registration:\nApplication to the motion estimation between 3D CT lung images with sliding conditions. Medical image\nanalysis, 17(2):182\u2013193, 2013.\n\n10\n\n\f[28] L. Risser, F.-X. Vialard, R. Wolz, D. D. Holm, and D. Rueckert. Simultaneous \ufb01ne and coarse diffeomorphic\nregistration: application to atrophy measurement in Alzheimer\u2019s disease. In MICCAI, pages 610\u2013617.\nSpringer, 2010.\n\n[29] M.-M. Roh\u00e9, M. Datar, T. Heimann, M. Sermesant, and X. Pennec. SVF-Net: Learning deformable image\n\nregistration using shape matching. In MICCAI, pages 266\u2013274. Springer, 2017.\n\n[30] D. Rueckert, L. I. Sonoda, C. Hayes, D. L. Hill, M. O. Leach, and D. J. Hawkes. Nonrigid registration\n\nusing free-form deformations: application to breast MR images. TMI, 18(8):712\u2013721, 1999.\n\n[31] T. Schmah, L. Risser, and F.-X. Vialard. Left-invariant metrics for diffeomorphic image registration\nwith spatially-varying regularisation. In International Conference on Medical Image Computing and\nComputer-Assisted Intervention, pages 203\u2013210. Springer, 2013.\n\n[32] D. Shen and C. Davatzikos. HAMMER: hierarchical attribute matching mechanism for elastic registration.\n\nTMI, 21(11):1421\u20131439, 2002.\n\n[33] Z. Shen, X. Han, Z. Xu, and M. Niethammer. Networks for joint af\ufb01ne and non-parametric image\n\nregistration. CVPR, 2019.\n\n[34] I. J. Simpson, M. J. Cardoso, M. Modat, D. M. Cash, M. W. Woolrich, J. L. Andersson, J. A. Schn-\nabel, S. Ourselin, A. D. N. Initiative, et al. Probabilistic non-linear registration with spatially adaptive\nregularisation. Medical image analysis, 26(1):203\u2013216, 2015.\n\n[35] N. Singh, J. Hinkle, S. Joshi, and P. T. Fletcher. A vector momenta formulation of diffeomorphisms for\nimproved geodesic regression and atlas construction. In 2013 IEEE 10th International Symposium on\nBiomedical Imaging, pages 1219\u20131222. IEEE, 2013.\n\n[36] R. Stefanescu, X. Pennec, and N. Ayache. Grid powered nonlinear image registration with locally adaptive\n\nregularization. Medical image analysis, 8(3):325\u2013342, 2004.\n\n[37] T. Vercauteren, X. Pennec, A. Perchant, and N. Ayache. Symmetric log-domain diffeomorphic registration:\n\nA demons-based approach. In MICCAI, pages 754\u2013761. Springer, 2008.\n\n[38] T. Vercauteren, X. Pennec, A. Perchant, and N. Ayache. Diffeomorphic demons: Ef\ufb01cient non-parametric\n\nimage registration. NeuroImage, 45(1):S61\u2013S72, 2009.\n\n[39] F.-X. Vialard and L. Risser. Spatially-varying metric learning for diffeomorphic image registration: A\nvariational framework. In International Conference on Medical Image Computing and Computer-Assisted\nIntervention, pages 227\u2013234. Springer, 2014.\n\n[40] F.-X. Vialard, L. Risser, D. Rueckert, and C. J. Cotter. Diffeomorphic 3D image registration via geodesic\nshooting using an ef\ufb01cient adjoint calculation. International Journal of Computer Vision, 97(2):229\u2013241,\n2012.\n\n[41] J. Wulff and M. J. Black. Ef\ufb01cient sparse-to-dense optical \ufb02ow estimation using a learned basis and layers.\n\nIn CVPR, pages 120\u2013130, 2015.\n\n[42] X. Yang, R. Kwitt, and M. Niethammer. Fast predictive image registration. In DLMIA, pages 48\u201357.\n\nSpringer, 2016.\n\n[43] X. Yang, R. Kwitt, M. Styner, and M. Niethammer. Quicksilver: Fast predictive image registration\u2013a deep\n\nlearning approach. NeuroImage, 158:378\u2013396, 2017.\n\n[44] L. Younes, F. Arrate, and M. I. Miller. Evolutions equations in computational anatomy. NeuroImage,\n\n45(1):S40\u2013S50, 2009.\n\n11\n\n\f", "award": [], "sourceid": 642, "authors": [{"given_name": "Zhengyang", "family_name": "Shen", "institution": "University of North Carolina at Chapel Hill"}, {"given_name": "Francois-Xavier", "family_name": "Vialard", "institution": "University Paris-Est"}, {"given_name": "Marc", "family_name": "Niethammer", "institution": "UNC Chapel Hill"}]}