{"title": "Arbicon-Net: Arbitrary Continuous Geometric Transformation Networks for Image Registration", "book": "Advances in Neural Information Processing Systems", "page_first": 3415, "page_last": 3425, "abstract": "This paper concerns the undetermined problem of estimating geometric transformation between image pairs. Recent methods introduce deep neural networks to predict the controlling parameters of hand-crafted geometric transformation models (e.g. thin-plate spline) for image registration and matching. However, the low-dimension parametric models are incapable of estimating a highly complex geometric transform with limited flexibility to model the actual geometric deformation from image pairs. To address this issue, we present an end-to-end trainable deep neural networks, named Arbitrary Continuous Geometric Transformation Networks (Arbicon-Net), to directly predict the dense displacement field for pairwise image alignment. Arbicon-Net is generalized from training data to predict the desired arbitrary continuous geometric transformation in a data-driven manner for unseen new pair of images. Particularly, without imposing penalization terms, the predicted displacement vector function is proven to be spatially continuous and smooth. To verify the performance of Arbicon-Net, we conducted semantic alignment tests over both synthetic and real image dataset with various experimental settings. The results demonstrate that Arbicon-Net outperforms the previous image alignment techniques in identifying the image correspondences.", "full_text": "Arbicon-Net: Arbitrary Continuous Geometric\nTransformation Networks for Image Registration\n\nNYU Multimedia and Visual Computing Lab\n\nNYU Multimedia and Visual Computing Lab\n\nNYU Multimedia and Visual Computing Lab\n\nNYU Multimedia and Visual Computing Lab\n\nJianchun Chen \u2217\n\nNew York University\nBrooklyn, NY 11201\njc7009@nyu.edu\n\nXiang Li\n\nNew York University\nBrooklyn, NY 11201\n\nxl845@nyu.edu\n\nLingjing Wang \u2217\n\nNew York University\nBrooklyn, NY 11201\nlw1474@nyu.edu\n\nYi Fang \u2020\n\nNew York University Abu Dhabi\n\nAbu Dhabi, UAE\nyfang@nyu.edu\n\nAbstract\n\nThis paper concerns the undetermined problem of estimating geometric transfor-\nmation between image pairs. Recent methods introduce deep neural networks\nto predict the controlling parameters of hand-crafted geometric transformation\nmodels (e.g. thin-plate spline) for image registration and matching. However the\nlow-dimension parametric models are incapable of estimating a highly complex\ngeometric transform with limited \ufb02exibility to model the actual geometric deforma-\ntion from image pairs. To address this issue, we present an end-to-end trainable\ndeep neural networks, named Arbitrary Continuous Geometric Transformation\nNetworks (Arbicon-Net), to directly predict the dense displacement \ufb01eld for pair-\nwise image alignment. Arbicon-Net is generalized from training data to predict the\ndesired arbitrary continuous geometric transformation in a data-driven manner for\nunseen new pair of images. Particularly, without imposing penalization terms, the\npredicted displacement vector function is proven to be spatially continuous and\nsmooth. To verify the performance of Arbicon-Net, we conducted semantic align-\nment tests over both synthetic and real image dataset with various experimental\nsettings. The results demonstrate that Arbicon-Net outperforms the previous image\nalignment techniques in identifying the image correspondences.\n\n1\n\nIntroduction\n\nImage registration plays a fundamental role in many computer vision applications such as medical\nimage processing [1], camera pose estimation [2], visual tracking [3]. Fig.1 shows the image\nregistration process, which includes geometric transformation estimation and image warping. To\nformulate the problem of image registration, traditional methods often approach the task in two steps:\n1) they \ufb01rstly compute the hand-crafted image features such as SIFT and HOG [4, 5] to capture\npixel-level descriptions, 2) and then iteratively search the optimal geometric transformation model\nto register a pair of images, driven by minimizing an alignment loss function. The alignment loss\nis usually pre-de\ufb01ned as a certain type of similarity metric (e.g. correlation scores) between two\n\n\u2217Equal contribution to this paper\n\u2020Corresponding author\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Illustration of Arbicon-Net for image alignment.\n\nsets of image feature descriptors. Previous efforts [6, 7, 8] have achieved great success in image\nregistration through the development of a variety of image feature descriptors and optimization\nalgorithms as summarized in [9]. However, they often face challenges posed by various deteriorated\nimage conditions such as 1) the dramatic image appearance variation (i.e. texture, color, lighting\nchanges and so on) between image pairs, and 2) the signi\ufb01cant geometric structural variation between\nimage pairs.\nThe recent success of deep neural network motivates researchers [10, 11, 12, 13] to develop deep\nlearning techniques to combine both two steps into an end-to-end trainable network, which aims to\nlearn a pre-de\ufb01ned geometric model (i.e. af\ufb01ne or thin-plate spline) through the regression process\nsupervised by minimizing the image matching loss. With the generalization from training data, those\nmethods are able to predict real-time image matching that is robust to various deteriorated image\nconditions. However, it is suggested by the authors [14] that pre-de\ufb01ned geometric transformation\nmodels only represent a set of low dimension transformations which prevents these methods from\npredicting complex geometric transformations for high-quality image registration. Moreover, the\ntransformations described by hand-crafted geometric models might not reveal the actual transfor-\nmation required for image alignment, which leads to a sub-optimal estimation of desired geometric\ntransformations.\nSome methods [14, 15, 16] tackle this problem by directly estimating semantic \ufb02ow from pixel-\nlevel features. These methods are more \ufb02exible to transfer the keypoints of images to semantically\ncorrelated positions. However, since the \ufb02ow \ufb01eld is estimated entirely by local features without\nintegrating global motion, local points are unable to move coherently, which consequently generates\ndistorted unrealistic images. In real-world applications (e.g. [1]), these \ufb02ow based methods require\nexplicitly imposed penalization to constrain the smoothness of \ufb02ow \ufb01eld.\nTo address the above mentioned issues, we propose to develop a novel geometric transformation\nnetwork, named arbitrary continuous geometric transformation networks (Arbicon-Net), to directly\npredict the dense displacement \ufb01eld that is not formulated by pre-de\ufb01ned hand-crafted geometric\nmodels. Compared with geometric model based approaches, Arbicon-Net uses deep neural network\nto model geometric transformations to accommodate arbitrary complex transformations required for\nthe registration of image pairs. Compared with semantic \ufb02ow based methods, Arbicon-Net features\nan attractive property, which predicts a smooth displacement \ufb01eld. As shown in Fig.2, we design\nan Arbicon-Net to simultaneously train three major modules, namely front-end geometric feature\nextractor module, transformation descriptor encoder module and displacement \ufb01eld predictor module,\nin an end-to-end fashion. The Arbicon-Net \ufb01rstly extracts dense feature maps from input image\npairs and encodes the discriminative local feature correlation into a transformation descriptor. The\nfollowing predictor module uses the transformation descriptor to decode displacement \ufb01eld for image\nregistration.\nContributions. We have three main contributions in this paper. First, we design a novel Arbicon-Net,\nwhich uses deep neural networks to predict dense displacement \ufb01eld to accommodate the arbitrary\ngeometric transformations according to the actual requirement for image registration. This addresses\nthe critical issue that the actual desired geometric transformation does not match with the one that can\nbe provided by pre-de\ufb01ned geometric model. Second, we prove that the Arbicon-Net is guaranteed to\ngenerate spatially continuous and smooth displacement \ufb01eld without imposing additional penalization\nterm as a smoothness constraint. Finally, we show that our proposed Arbicon-Net achieved superior\nperformance against hand-crafted geometric transformation models with both strong and weak\nsupervision.\n\n2\n\n\fFigure 2: Main Pipeline. Our proposed end-to-end trainable Arbicon-Net has three main compo-\nnents. 1) Geometric Feature Extractor Module; 2) Transformation Descriptor Encoder Module; 3)\nDisplacement Field Predictor Module.\n\n2 Related Works\n\nImage Registration. Image registration is de\ufb01ned as a process to determine a smooth geometric\ntransformation between input image pairs, especially for 2D/3D medical images. Existing methods\nsearch optimal geometric transformation by iteratively minimizing alignment loss, which is typically\nde\ufb01ned by the feature similarity or hierarchically de\ufb01ned intensity pattern. To achieve a high-quality\nimage registration, researchers [17, 1, 9] have explored diverse geometric transformation models,\nimage similarity metrics and searching algorithms.\nNon-learning based Image Correspondence Matching. The classic image correspondence match-\ning pipeline [6, 7, 8] starts by detecting key points via hand-crafted pixel-level feature descriptors\n[4, 5, 18], followed by feature matching strategies to determine the optimal point correspondence\n[19, 5]. Following researches have developed various hand-crafted algorithms [19, 20] to remove\nincorrect matches by searching global transformation or utilizing neighbor information. While these\nmethods are limited in matching speed and matching performance, they have far-reaching impact on\ncomputer vision society by imposing a standard pipeline and introducing geometric transformation\nestimation as a mainstream approach for image matching problem.\nLearning based Image Correspondence Matching. Inspired by the success of deep neural network,\npioneer works [21, 22, 23] propose to use pre-trained convolutional neural networks (CNNs) instead\nof hand-crafted ones to extract discriminative pixel-wise feature descriptors. Following researches\ndevelop learnable feature extraction layer [24, 25] and learnable feature matching layer [26, 27] with\ndifferentiable image alignment loss. Han et al. [28] introduce a fully learnable image correspondence\nmatching strategy over region proposals. However, this method is not in an end-to-end trainable\nfashion.\nMore recently, researchers [10, 11, 12, 13] propose end-to-end trainable network architectures for\nimage correspondence estimation. Speci\ufb01cally, these methods de\ufb01ne a regression network to predict\nthe parameter of speci\ufb01c geometric transformation models (i.e. thin-plate spline, af\ufb01ne). But they are\nlimited by the use of low dimension geometric models and consequently less capable of performing\n\ufb01ne-grained image geometric transformation. Other researchers either recurrently regress pixel-\nlevel \ufb02ow \ufb01eld [15, 16] to approximate \ufb01ne-grained image transformation or determine \ufb02ow \ufb01eld\nby neighbourhood consensus assignment [14]. However, as we stated above, they don\u2019t take the\nsmoothness of displacement \ufb01eld into account.\n\n3 Approach\n\n3.1 Geometric Feature Extractor\n\n3\n\n\fFollowing the common image matching paradigms [11], our Arbicon-Net\nstarts with extracting geometric features from input image pair IA, IB. We\n\ufb01rstly leverage a share-weighted CNN to generate a representative feature\nmap F \u2208 Rh\u00d7w\u00d7c for each input image, where at each location the feature\nvector fij \u2208 Rc represents local semantic information.\nIn order to estimate the geometric transformation of given image pairs, we\nestablish the local feature correlations between two feature maps by using the\nnormalized cosine similarity. For each local descriptor from FA, we compute\nits similarity score with all local descriptors in FB to form a 4-D correlation\ntensor S \u2208 Rh\u00d7w\u00d7h\u00d7w as shown in Fig.3. Each element sijkl \u2208 S is\ncomputed as,\n\n(cid:10)f A\n\n(cid:11)\nij , f B\nkl\nkl||2\nij||2||f B\n\nFigure 3: Feature Cor-\nrelation.\n\n(1)\nwhere (cid:104)\u00b7(cid:105) denotes the inner product of two vectors, the denominator acts a normalization term to\nfurther amplify con\ufb01dent matching and reduce ambiguity matching.\n\nsijkl =\n\n||f A\n\n3.2 Transformation Descriptor Encoder\n\nTo learn a more discriminative feature correlation, we leverage 4-D convolutional neural networks\n(CNNs) to re\ufb01ne correlation tensor S by using neighbor information [14]. The 4-D convolution layers\nintegrate additional neighborhood information compared with regular 2-D CNNs. Since the order\nof input image pairs (IA,IB) or (IB,IA) do not in\ufb02uence the result of local feature correlation, the\nconvolution operation is symmetrically applied, formulated as,\n\nSC = Conv(S) + (Conv(ST ))T\n\n(2)\nijkl = sklij. Moreover, we normalize the learned\n, where the transpose of S is computed according to sT\nhwpq}. This\n4-D correlation tensor s by Eq.3, where \u03c6A\nnormalization encourages the bilateral con\ufb01dence of correlated pairs for from source image and target\nimage.\n\npqhw} and \u03c6B\n\npq = {sc\n\npq = {sc\n\npq11, ..., sc\n\n11pq, ..., sc\n\n\u02c6sijkl =\n\nsc\nijkl\n\nsc\nijkl\n\nmax(\u03c6A\nij)\n\nmax(\u03c6B\nkl)\n\nsc\nijkl\n\n(3)\n\nSince our goal is to \ufb01nd a global transformation, we use a Multi-Layer Perceptron to encode\nlearned 4-D tensor \u02c6S into a transformation descriptor dAB \u2208 Rm that represents the overall image\ncorrespondence information, as shown in Eq.4. For global geometric transformation learning,\nthe image correspondence information describes a geometric transformation that optimally aligns\ncorresponding points on two images.\n\ndAB = M LP ( \u02c6S)\n\n(4)\n\n3.3 Displacement Field Predictor\nIn general, the geometric transformation T for each point x in a point set X \u2282 R2 can be de\ufb01ned as:\n(5)\n\nT (x, v) = x + v(x)\n\n, where v : R2 \u2192 R2 is a \u201cpoint displacement\u201d function.\nThe image registration task can be formulated as a process of\ndetermining the displacement function v. It is necessary for func-\ntion v to be a continuous and smooth function according to the\nMotion Coherent Theory (MCT) [29]. Fortunately, by leveraging\ndeep neural network architecture, we can construct the suitable\ndisplacement function v which satis\ufb01es the continuous and smooth\ncharacteristics.\nAs illustrated in Fig.4, given n 2-D points in the source image\nplane, we duplicate the transformation descriptor dAB for n times.\n\n4\n\nFigure 4: Displacement Field\nPredictor Module.\n\n\fEach point is concatenated with the m-D global descriptor dAB. We further construct a Displacement\nField Predictor network with four successive MLPs to decode the concatenated (m + 2)-D vector into\n2-D displacement vector. We de\ufb01ned this neural network structure in Eq.6 as F(\u00b7) : Rm+2 \u2192 R2,\nformulated as,\n\nv(x) = F([x, dAB])\n\n(6)\n\n,where [\u00b7] indicates concatenation operation.\nFurthermore, we brie\ufb02y prove the continuity and smoothness of our displacement \ufb01eld predictor F to\nbe used as our deep learning-based solution for the displacement function v.\nContinuity. Since both MLP and activation function \u03c3 are continuous, the continuity of Displacement\nField Predictor network can be trivially proven as a composite of continuous functions. Since dAB\nis concatenated to each point in X, this concatenation operation does not change the continuity\nfor displacement function v(\u00b7) = F([\u00b7, dAB]). In contrast, commonly used learning paradigms\n[1, 15, 16], which directly map high dimension feature space to 2-D displacement \ufb01eld, output a set of\ndiscrete displacement vectors, while the displacements of other points need to be further interpolated.\nSmoothness. After choosing a smooth function SoftPlus [30] as the activation function in our Dis-\nplacement Field Predictor network, it becomes trivial to estimate its complexity and smoothness since\nthe displacement function is a composite of a number of smooth functions (MLP and SoftPlus). In\npractice, Regularization Theory (RT) [31] uses the oscillatory behavior of a function to further mea-\nsure the smoothness of displacement function. The oscillatory behavior is measured by Reproducing\nKernel Hilbert Space (RKHS) [31, 32] in Eq.7.\n||v||2Hm =\n\n(cid:90)\n\n(7)\n\nds\n\n|\u02dcv(s)|2\n\u02dcg(s)\n\nRD\n\n, where \u02dcv is the Fourier transform of the displacement function v and \u02dcg is a low-pass \ufb01lter. In other\nwords, a smoother displacement function has considerably less energy in high frequency domain. We\ngenerally express models that regress pixel-level displacement vector, including Arbicon-Net and\nRTNs [16], as a composite function v in Eq.8.\n\nv(x) = F(G(x)) : R2 \u2192 R2\n\n(8)\nFunction G and F denotes the point feature encoding network and point displacement vector regres-\nsion network respectively. Speci\ufb01cally, we have G(x) = [x, dAB] in Arbicon-Net. In contrast, in\nRTNs G generates a high-dimensional feature map by sequential CNNs. Therefore, the G in RTNs\nis generally considered as a sparse and oscillate function, especially when the dimension of feature\nvector (output of function G) is high, which causes the widely known \u201ccurse of dimensionality\u201d\nproblem. In this section, we assume that two models have same regression network F and the input\nof function F are normalized to a same scale.\nAccording to [33], the Fourier transform of the composite function v has essentially maximum\nfrequency uv as\n\nx\n\nuv = uF max\n\n(9)\n, where uF is the maximum frequency of F, which is independent of G. Assume that the outputs of\ndifferent function G are in a same scale, the oscillate function G tends to have a larger maximum value\nof |G(cid:48)(x)| compared with linear function in Arbicon-Net. As a result, our composite function has a\nsmaller uv, which is likely to have lower energy in high frequency domain, which further guarantees\na smoother displacement function.\nBased on our proposed paradigm, we further constrain the smoothness of function F. Fortunately,\ngiven the popularity of deep learning models, the recent research community has been proposing\nregularization strategies, which naturally help our Displacement Field Predictor network to reduce the\nrisk of the oscillatory of displacement function. One simple solution is to design a proper network size.\nIn section 4.3, we provide empirical results to validate the smoothness of our estimated displacement\nfunction compared with non-rigid geometric transformation models.\n\n|G(cid:48)(x)|\n\n3.4 Loss functions\n\nAs shown in the right box of Fig.2, our designed method is designed to learn geometric transformation\nunder either strong supervision or weak supervision.\n\n5\n\n\fFor strongly-supervised loss, we have point correspondence information of x \u2208 X from source\nplane and y \u2208 Y from target plane. Lstrong directly minimizes the pairwise L2 distance between\ncorresponding points in transformed image plane and target image plane, as shown in Eq.10.\n\nN(cid:88)\n\ni=1\n\nLstrong =\n\n1\nN\n\n||T (xi) \u2212 yi||2\n\n2\n\n(10)\n\nLweak = \u2212 (cid:88)\n\ni,j,k,l\n\nFor weakly-supervised loss, we maximize the inner product of corresponding location in transformed\nsource feature map T (FA) and target feature map FB following the paradigm described in [10].\nLet T (IA) and IB to be matched, we have T (IA)ij and I ij\nB to be semantically matched, thus\n\nto be maximum. We implement the loss function described in Eq.11.\n\n(cid:68)T (fA)ij \u00b7 f ij\n\nB\n\n(cid:69)\n\nsijkl1d(T (i,j),(k,l))<t\n\n(11)\n\n, where 1(\u00b7) is the indicator function, d(\u00b7) denotes L1 distance.\nSince our proposed network is end-to-end trainable, the Geometric Transformation Network is\noptimized together with other components. It deserves noting that, since our method learns the\ngeometric transformation from training dataset, it is more robust to train or \ufb01ne-tune our network on\nreal image dataset than transfer a pre-trained network from a one-fold synthesized dataset.\n\n4 Experiments\n\nIn this section, we carried out a set of tests under different experimental settings to validate the\nperformance of our proposed Arbicon-Net for its capability of estimating the geometric transformation\nfor image dense correspondence in semantic alignment.\nIn section 4.1, we describe the implementation details of our Arbicon-Net to be tested in our\nexperiments. In section 4.2, we discuss the details of the experimental dataset preparation, evaluation\nmetric for experimental results and baseline models for experimental performance comparison.\nIn section 4.3, we validate the performance of Arbicon-Net for the estimation of the geometric\ntransformation for real image pairs with weakly supervised training. In section 4.4 and 4.5, we\ndemonstrate the performance of Arbicon-Net for the estimation of parametric and non-parametric\ngeometric transformation respectively, and compare to the state-of-the-art techniques.\n\n4.1\n\nImplementation Details\n\nAs shown in Fig.2, the Arbicon-Net is implemented based on deep neural networks with the following\narchitecture con\ufb01guration. Arbicon-Net starts with the use of ResNet [34] (before conv4-23) with\nweight pre-trained on ImageNet for local feature extraction, then followed by three 4-D convolution\nkernels in Section 3.2 which are of size (3, 3, 3) with channels (10, 10, 1) respectively, and end\nwith four MLPs con\ufb01gured with the size (256, 256, 64, 2). In Arbicon-Net, we set the dimension of\ntranslation descriptor dAB at 256 as shown in Fig.4. For weakly supervised Arbicon-Net (refer to\nSection 4.3), we \ufb01rst train our model on synthesize dataset and \ufb01ne-tune our model on training set\nof PF-PASCAL using loss function in Eq.11. We refer [10] for detailed loss function and training\nsetup. For supervised Arbicon-Net, we use Adam optimizer for training with learning rate 0.001. The\nnetwork is implemented by PyTorch framework and ran on an Nvidia GTX 1080Ti GPU.\n\n4.2 Dataset, Metric and Baseline\n\nDataset. In our experiment, three image datasets, Pascal VOC dataset [35], PF-Pascal dataset [36] and\nProposal Flow dataset [37] are used to prepare both synthesized and real image dataset for the various\ntests. Pascal VOC dataset [35] contains 28,952 images. PF-Pascal contains 1351 semantically aligned\nimage pairs from 20 semantic category of Pascal VOC dataset with a 7:3:3 training/validation/testing\nsplit. Proposal Flow dataset [37] with 900 image pairs from 5 categories. We randomly split the\nProposal Flow dataset into 3 folds for k-fold validation in the test. The image pair from PF-Pascal\nand Proposal Flow datasets is annotated with correspondences that could be used as ground truth for\nimage matching performance evaluation. In order to prepare the synthesized image data, we imposed\n\n6\n\n\fMethods\nHOG+PF-LOM [36]\nSCNet-AG+ [28]\nCNNGeo [11]\nA2Net [13]\nWeakAlign [10]\nWeakAlign-4D\nArbicon-Net\n\nPCK(%)\n\n62.5\n72.2\n71.9\n70.9\n75.8\n76.5\n77.3\n\nTable 1: Quantitative results on PF-\nPascal [36] dataset with weakly super-\nvise training.\n\nFigure 5: Qualitative results on PF-Pascal [36] dataset with\nweakly supervise training.\n\nMethods\nMSE\nCNNGeo-4D 0.0037\n0.0002\nArbicon-Net\n\nTable 2: Quantitative result on\nsynthesized dataset with dif-\nfeomorphic non-linear trans-\nformation.\n\nFigure 6: Qualitative comparison on synthesized dataset with\ndiffeomorphic non-linear transformation.\n\ntwo different types of parametric transformations, Thin plate spline (TPS) and Diffeomorphic non-\nlinear transformation, onto the images in Pascal VOC dataset.\nEvaluation Metric. We use standard evaluation protocol, the average probability of correct keypoint\n(PCK) for the evaluation of image matching in [38]. PCK classi\ufb01es keypoint as correct match if the\ndistance between transformed source keypoint and corresponding target keypoint is within threshold\n\u03b1 = 0.1 of the image size [10, 28]. For the tests on the synthesized dataset, we directly use Mean\nSquare Error between transformed source points and corresponding target points for evaluation, as\nexplained in Eq.10. Additionally, we measure the smoothness of estimated displacement \ufb01eld by the\nthe second order derivatives of displacement \ufb01eld denoted as Esmooth following [39].\nBaseline Models. In the experiments, Arbicon-Net is compared with representative methods in-\ncluding HOG+PF-LOM [36], SCNet [28], CNNGeo [11], A2Net [13], WeakAlign [10]. We do not\nconduct the experiment for the comparison between Arbicon-Net and other semantic \ufb02ow methods\nthat directly identify sparse correspondence point without fully aligning two images. In addition,\nsince the 4-D convolution module (refer to Section 3.2) is not proposed by us, for the fair compari-\nson, we replace the 2-D convolution Regression Network module with 4-D convolution module in\nnetwork structures of WeakAlign and CNNGeo, and obtain two new networks, namely WeakAlign-\n4D and CNNGeo-4D. We compared Arbicon-Net to both WeakAlign-4D and CNNGeo-4D in the\nexperiments.\n\n4.3 Arbicon-Net with Weakly Supervised Training\n\nExperiment Setting: In this test, we conduct experiments to test the performance of Arbicon-\nNet for the estimation of geometric transformation for image matching without using annotated\ncorrespondence for training. In the weakly supervised setting, the Arbicon-Net and baseline models\n(WeakAlign and WeakAlign-4D) are \ufb01rstly pre-trained on the synthesized image dataset by TPS\ntransformation, and then followed by \ufb01ne-tuning with weakly-supervised loss (refer to Eq.11). The\nimage pairs from PF-Pascal with a certain split for are used for the training and testing of the models.\nResults Analysis: Table 1 compares PCK scores for Arbicon-Net and baseline models for the test\nresult on PF-Pascal dataset. The comparison result indicates that Arbicon-Net outperforms all baseline\nmethods. To better illustrate the comparison results, we further show two pair images before and\nafter image registration in Fig.5. The \ufb01rst two columns show the source and target images, the third\ncolumn illustrates the transformed source image by Arbicon-Net and the fourth column illustrates the\ntransformed source image by WeakAlign-4D. As we can see from the Figure, compared to baseline\n\n7\n\n\fFigure 7: (a). Quantitative result on synthesized dataset with different complexity TPS transformation.\n(b). Qualitative result on synthesized dataset with 6 \u00d7 6 controlling points TPS transformation in the\nright side.\n\nMethods\nCNNGeo (aff.)\nCNNGeo (TPS)\nCNNGeo (aff.+TPS)\nArbicon-Net\n\nPCK(%)\n73.5 \u00b10.9\n78.2 \u00b10.7\n78.5 \u00b10.9\n84.3 \u00b10.6\n\nEsmooth\n0.000\n0.013\n0.016\n0.008\n\nTable 3: Quantitative comparisons on Proposal\nFlow [37] dataset (Non-parametric transformation)\nwith strong supervision.\n\nFigure 8: Qualitative comparisons on Proposal\nFlow [37] dataset with strong supervision.\n\nmethod WeakAlign-4D, Arbicon-Net is able to predict a \ufb01ne-grained geometric transformation with\ncoherent \ufb02ow motion and preservation of local geometric structural details.\nIn this test, Arbicon-Net and baseline models are pre-trained with TPS-based synthesized image\ndata, and followed by \ufb01ne-tuning with weakly-supervised loss. A small learning rate is posed to\nprevent model degeneration. In this way, the Arbicon-Net are trained with bias tendency to predict the\nTPS-like geometric transformation for image pair. Therefore, we design the following two additional\ntests for the estimation of parametric and non-parametric transformation with supervised training\nsettings.\n\n4.4 Estimation of Parametric Transformation\n\nExperiment Setting: Parametric transformation means a set of geometric transformation that can\nbe expressed by functions of controlling parameters. In this experiment, we prepare the synthesized\nimage data with two types of parametric transformations: TPS and diffeomorphic non-linear trans-\nformation [17]. Then, we evaluate the performance of Arbicon-Net in estimating those parametric\ntransformation after training and compare to CNNGeo-4D which uses 3 \u00d7 3 controlling points TPS\nas its geometric model. We are particularly interested in studying how Arbicon-Net addresses the\ncritical issue that the actual desired geometric transformation does not match with the one which can\nbe provided by pre-de\ufb01ned geometric model. To this end, we prepared two tests: 1) we synthesize\ndifferent levels of transformed images on Pascal VOC by varying the number of TPS controlling\n(increasing from 4 \u00d7 4 to 6 \u00d7 6) while the CNNGeo-4D remains the same TPS parameter setting,\n2) we implement a diffeomorphic non-linear transformation (a non-TPS type transformation) to\nsynthetic data. For simplicity, we ignore the translation term to only simulate a simple 4 DoF local\naf\ufb01ne transformation around the center of image.\nResults Analysis: The experimental results of the \ufb01rst test are illustrated in Fig.7. As shown in Fig.7,\nCNNGeo-4D (red curve) has consistently about 1.5 times higher MSE value than that of Arbicon-Net\n(blue curve). When testing on synthetic data simulated by 6 \u00d7 6 controlling point TPS, we visualize\nthe estimated transformation in red mesh to compare the performance between Arbicon-Net and\nCNNGeo-4D. As we can see, Arbicon-Net clearly predicts a much more accurate displacement \ufb01eld\nwith more re\ufb01ned and detailed local deformation, in contrast to CNNGeo-4D. The performance\n\n8\n\n\fde\ufb01ciency by CNNGeo-4D highlights that mis-match between actual desired transformation and\nthe one provided by a pre-de\ufb01ned geometric model could dramatically impact on the estimation\nof geometric transformation. The experimental results of the second test are shown in Table 2 and\nFig.6. As we can see from the Table, Arbicon-Net clearly outperforms the CNNGeo-4D with a great\nmargin. In the Fig.6, CNNGeo-4D failed to estimate the underlying geometric transformation while\nour Arbicon-Net can predict a high-quality one. This observation suggests that CNNGeo-4D limits\nits ability to estimate a TPS-based geometric transformation while Arbicon-Net is able to estimate\nan arbitrary geometric transformation for a given image pair, which is more suitable for estimating\nundetermined transformation.\n\n4.5 Estimation of Non-parametric Transformation\n\nExperiment Setting: In this experiment, we investigate the performance of Arbicon-Net in estima-\ntion of non-parametric transformation. We consider the transformation between real image pairs\nin Proposal Flow dataset to be non-parametric transformation, as there is no speci\ufb01c parametric\ngeometric model that can be generalized to describe the transformation for any real image pair. When\ntesting, Arbicon-Net and CNNGeo are \ufb01rstly trained with real image pair supervised with annotated\ncorrespondence labels. The parameters of Arbicon-Net and CNNGeo in this test are trained from\nscratch except for \ufb01xed layers from ResNet-101. We repeat experiment three times. Each time we\nselect one-fold as testing set and the other folds as training set. In this test, we use regular 2D-Conv\nto replace 4D-Conv in Arbicon-Net for fairness.\nResults Analysis: Table 3 lists the PCK scores and Esmooth as the comparison results between\nArbicon-Net and CNNGeo with different geometric model settings. Among these models, our\nArbicon-Net achieves better estimation of geometric transformation based on a higher PCK score.\nThis suggests that our Arbicon-Net is more suitable for the estimation of geometric transformation\nfor image pairs in real applications. In addition, Arbicon-Net produces even smoother displacement\nfunction with smaller Esmooth value compared with non-rigid geometric transformation model\nTPS. Fig.8 shows two pairs of images pre- and post-registration (warping). We can clearly see\nfrom the \ufb01gure that Arbicon-Net is able to smoothly transform the source image to the target one\nwithout introducing signi\ufb01cant image distortion, as shown in third column. In contrast, the CNNGeo\nsigni\ufb01cantly deteriorate the source image after warping. This further validates that Arbicon-Net can\npredict a spatially continuous and smooth displacement \ufb01eld as one of our key contributions in this\npaper.\n\n5 Conclusion\n\nWe present a novel Arbicon-Net for image registration, which directly learns a dense displacement\n\ufb01eld between input image pairs. Compared with hand-crafted geometric models, our network is\nmore capable in modeling arbitrary high dimension transformation function. The network structure\npreserves the predicted displacement function to be spatial continuous and smooth and thus removes\nthe limitation of adding penalization term in training. The experiments in semantic alignment task\ndemonstrate the effectiveness of our approach.\n\n6 Acknowledgement\n\nWe would like to thank the reviewers for their thoughtful comments and efforts towards improving\nour manuscript. This work is partially supported by ADEK Grant (No. AARE-18150).\n\n9\n\n\fReferences\n[1] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. An unsupervised\nlearning model for deformable medical image registration. In Proceedings of the IEEE conference on\ncomputer vision and pattern recognition, pages 9252\u20139260, 2018.\n\n[2] Enliang Zheng and Changchang Wu. Structure from motion using structure-less resection. In Proceedings\n\nof the IEEE International Conference on Computer Vision, pages 2075\u20132083, 2015.\n\n[3] Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. Dtam: Dense tracking and mapping in\n\nreal-time. In 2011 international conference on computer vision, pages 2320\u20132327. IEEE, 2011.\n\n[4] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In international\nConference on computer vision & Pattern Recognition (CVPR\u201905), volume 1, pages 886\u2013893. IEEE\nComputer Society, 2005.\n\n[5] David G Lowe et al. Object recognition from local scale-invariant features. In iccv, volume 99, pages\n\n1150\u20131157, 1999.\n\n[6] Alexander C Berg, Tamara L Berg, and Jitendra Malik. Shape matching and object recognition using low\n\ndistortion correspondences. In CVPR (1), pages 26\u201333. Citeseer, 2005.\n\n[7] Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on\n\npattern analysis and machine intelligence, 28(4):594\u2013611, 2006.\n\n[8] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift \ufb02ow: Dense correspondence across scenes and its\n\napplications. IEEE transactions on pattern analysis and machine intelligence, 33(5):978\u2013994, 2010.\n\n[9] Barbara Zitova and Jan Flusser. Image registration methods: a survey. Image and vision computing,\n\n21(11):977\u20131000, 2003.\n\n[10] Ignacio Rocco, Relja Arandjelovi\u00b4c, and Josef Sivic. End-to-end weakly-supervised semantic alignment.\nIn Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6917\u20136925,\n2018.\n\n[11] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. Convolutional neural network architecture for\ngeometric matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\npages 6148\u20136157, 2017.\n\n[12] Angjoo Kanazawa, David W Jacobs, and Manmohan Chandraker. Warpnet: Weakly supervised matching\nfor single-view reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition, pages 3253\u20133261, 2016.\n\n[13] Paul Hongsuck Seo, Jongmin Lee, Deunsol Jung, Bohyung Han, and Minsu Cho. Attentive semantic\nalignment with offset-aware correlation kernels. In Proceedings of the European Conference on Computer\nVision (ECCV), pages 349\u2013364, 2018.\n\n[14] Ignacio Rocco, Mircea Cimpoi, Relja Arandjelovi\u00b4c, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Neigh-\nbourhood consensus networks. In Advances in Neural Information Processing Systems, pages 1658\u20131669,\n2018.\n\n[15] Sangryul Jeon, Seungryong Kim, Dongbo Min, and Kwanghoon Sohn. Parn: Pyramidal af\ufb01ne regression\nnetworks for dense semantic correspondence. In The European Conference on Computer Vision (ECCV),\nSeptember 2018.\n\n[16] Seungryong Kim, Stephen Lin, SANG RYUL JEON, Dongbo Min, and Kwanghoon Sohn. Recurrent\ntransformer networks for semantic correspondence. In Advances in Neural Information Processing Systems,\npages 6126\u20136136, 2018.\n\n[17] Ramkrishnan Narayanan, Jeffrey A Fessler, Hyunjin Park, and Charles R Meyer. Diffeomorphic nonlinear\ntransformations: A local parametric approach for image registration. In Biennial International Conference\non Information Processing in Medical Imaging, pages 174\u2013185. Springer, 2005.\n\n[18] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features.\n\nconference on computer vision, pages 404\u2013417. Springer, 2006.\n\nIn European\n\n[19] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model \ufb01tting with\napplications to image analysis and automated cartography. Communications of the ACM, 24(6):381\u2013395,\n1981.\n\n10\n\n\f[20] David G Lowe. Distinctive image features from scale-invariant keypoints.\n\ncomputer vision, 60(2):91\u2013110, 2004.\n\nInternational journal of\n\n[21] Tatsunori Taniai, Sudipta N Sinha, and Yoichi Sato. Joint recovery of dense correspondence and cosegmen-\ntation in two images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,\npages 4246\u20134255, 2016.\n\n[22] Nikolai Ufer and Bjorn Ommer. Deep semantic feature matching. In Proceedings of the IEEE Conference\n\non Computer Vision and Pattern Recognition, pages 6914\u20136923, 2017.\n\n[23] Fan Yang, Xin Li, Hong Cheng, Jianping Li, and Leiting Chen. Object-aware dense semantic corre-\nspondence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages\n2777\u20132785, 2017.\n\n[24] Seungryong Kim, Dongbo Min, Stephen Lin, and Kwanghoon Sohn. Dctm: Discrete-continuous transfor-\nmation matching for semantic \ufb02ow. In Proceedings of the IEEE International Conference on Computer\nVision, pages 4529\u20134538, 2017.\n\n[25] Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn.\nFcss: Fully convolutional self-similarity for dense semantic correspondence. In Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition, pages 6560\u20136569, 2017.\n\n[26] Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C Berg. Matchnet: Unifying\nfeature and metric learning for patch-based matching. In Proceedings of the IEEE Conference on Computer\nVision and Pattern Recognition, pages 3279\u20133286, 2015.\n\n[27] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural\nnetworks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages\n4353\u20134361, 2015.\n\n[28] Kai Han, Rafael S Rezende, Bumsub Ham, Kwan-Yee K Wong, Minsu Cho, Cordelia Schmid, and Jean\nPonce. Scnet: Learning semantic correspondence. In Proceedings of the IEEE International Conference\non Computer Vision, pages 1831\u20131840, 2017.\n\n[29] Alan L Yuille and Norberto M Grzywacz. A mathematical analysis of the motion coherence theory.\n\nInternational Journal of Computer Vision, 3(2):155\u2013175, 1989.\n\n[30] Vinod Nair and Geoffrey E Hinton. Recti\ufb01ed linear units improve restricted boltzmann machines. In\nProceedings of the 27th international conference on machine learning (ICML-10), pages 807\u2013814, 2010.\n\n[31] Federico Girosi, Michael Jones, and Tomaso Poggio. Regularization theory and neural networks architec-\n\ntures. Neural computation, 7(2):219\u2013269, 1995.\n\n[32] Zhe Chen and Simon Haykin. On different facets of regularization theory. Neural Computation,\n\n14(12):2791\u20132846, 2002.\n\n[33] Steven Bergner, Torsten Moller, Daniel Weiskopf, and David J Muraki. A spectral analysis of function\nIEEE transactions on\n\ncomposition and its implications for sampling in direct volume visualization.\nvisualization and computer graphics, 12(5):1353\u20131360, 2006.\n\n[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\nIn Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770\u2013778, 2016.\n\n[35] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The\npascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303\u2013338,\n2010.\n\n[36] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal \ufb02ow: Semantic correspondences\nfrom object proposals. IEEE transactions on pattern analysis and machine intelligence, 40(7):1711\u20131725,\n2018.\n\n[37] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal \ufb02ow. In Proceedings of the IEEE\n\nConference on Computer Vision and Pattern Recognition, pages 3475\u20133484, 2016.\n\n[38] Yi Yang and Deva Ramanan. Articulated human detection with \ufb02exible mixtures of parts. IEEE transactions\n\non pattern analysis and machine intelligence, 35(12):2878\u20132890, 2012.\n\n[39] Jean Duchon. Splines minimizing rotation-invariant semi-norms in sobolev spaces. In Constructive theory\n\nof functions of several variables, pages 85\u2013100. Springer, 1977.\n\n11\n\n\f", "award": [], "sourceid": 1893, "authors": [{"given_name": "Jianchun", "family_name": "Chen", "institution": "New York University"}, {"given_name": "Lingjing", "family_name": "Wang", "institution": "New York University"}, {"given_name": "Xiang", "family_name": "Li", "institution": "New York University"}, {"given_name": "Yi", "family_name": "Fang", "institution": "New York University"}]}