## Message Board

Volume 47 Issue 8
Aug.  2022
Turn off MathJax
Article Contents

LIU Weiyu, WAN Yi, ZHANG Yongjun, YAO Yongxiang, LIU Xinyi, SHI Lisong. An Efficient Matching Method of LiDAR Depth Map and Aerial Image Based on Phase Mean Convolution[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1309-1317. doi: 10.13203/j.whugis20210524
 Citation: LIU Weiyu, WAN Yi, ZHANG Yongjun, YAO Yongxiang, LIU Xinyi, SHI Lisong. An Efficient Matching Method of LiDAR Depth Map and Aerial Image Based on Phase Mean Convolution[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1309-1317.

# An Efficient Matching Method of LiDAR Depth Map and Aerial Image Based on Phase Mean Convolution

##### doi: 10.13203/j.whugis20210524
Funds:

Basic Research Strengthening Program of China 173 Program

the National Natural Science Foundation of China 42030102

the National Natural Science Foundation of China 41871368

the National Natural Science Foundation of China 42001406

• Author Bio:

LIU Weiyu, postgraduate, specializes in multi-modal image matching, and bundle adjustment. E-mail: liuwy0225@whu.edu.cn

• Corresponding author: ZHANG Yongjun, PhD, professor. E-mail: zhangyj@whu.edu.cn
• Publish Date: 2022-08-05
•   Objectives  Multi-source image matching is primarily disturbed by nonlinear intensity difference, contrast difference and inconspicuous regional structure features, while the significant differences of texture features result in lack of part structure seriously between light detection and ranging(LiDAR)depth map and aerial image, and this problem causes a mutation in the phase extremum, which further increases the difficulty of matching.  Methods  In this paper, a method of efficient matching of LiDAR depth map and aerial image based on phase mean convolution is proposed. In the image feature matching stage, a histogram of phase mean energy convolution(HPMEC) is established, which extended the phase consistency model in order to solve a mean convolution sequence and phase maximum label map by constructing phase mean energy convolution equation. Then the nearest neighbor matching algorithm was completed the initial match and marginalizing sample consensus plus was used to remove outliers. Based on the thread pool parallel strategy, the images were matched by dividing the overlapping grid. Multiple sets of LiDAR depth map and aerial image with different types of ground coverage are used to as dataset to experiment with position scale orientation-scale invariant feature transform (PSO-SIFT), Log-Gabor histogram descriptor (LGHD), radiation-variation insensitive feature transform (RIFT) and histogram of absolute phase consistency gradients (HAPCG) methods respectively.  Results  The results show that the performance of HPMEC method is superior to the other four methods in the matching of LiDAR depth map and aerial image, the average running time is 13.3 times of PSO-SIFT, 10.9 times of LGHD, 10.4 times of HAPCG and 7.0 times of RIFT, at the same time the average correct matching points are significantly higher than the other four methods, the root mean square error is lightly better than the other four methods within 1.9 pixels.  Conclusions  The proposed HPMEC method could achieve efficient and robust matching between LiDAR depth map and aerial image.
•  [1] Jung J, Sohn G. A Line-Based Progressive Refinement of 3D Rooftop Models Using Airborne LiDAR Data with Single View Imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 149: 157-175 [2] Huang R Y, Zheng S Y, Hu K. Registration of Aerial Optical Images with LiDAR Data Using the Closest Point Principle and Collinearity Equations[J]. Sensors, 2018, 18(6): 1770 [3] Peng S B, Zhang L. Automatic Registration of Optical Images with Airborne LiDAR Point Cloud in Urban Scenes Based on Line-Point Similarity Invariant and Extended Collinearity Equations[J]. Sensors (Basel, Switzerland), 2019, 19(5): 1086 [4] Parmehr E G, Fraser C S, Zhang C S. Automatic Parameter Selection for Intensity-Based Registration of Imagery to LiDAR Data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(12): 7032-7043 [5] 吴军, 饶云, 胡彦君, 等. "针孔"模拟成像下的单航空影像与LiDAR点云配准[J]. 遥感学报, 2016, 20(1): 80-93 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB201601009.htm Wu Jun, Rao Yun, Hu Yanjun, et al. Automatic Registration of Single Aerial Image with LiDAR Data Based on "Pin-Hole" Imaging Simulation and Iterative Computation[J]. Journal of Remote Sensing, 2016, 20(1): 80-93 https://www.cnki.com.cn/Article/CJFDTOTAL-YGXB201601009.htm [6] Chen J, Tian J, Lee N, et al. A Partial Intensity Invariant Feature Descriptor for Multimodal Retinal Image Registration[J]. IEEE Transactions on BioMedical Engineering, 2010, 57(7): 1707-1718 [7] Dellinger F, Delon J, Gousseau Y, et al. SAR-SIFT: A SIFT-Like Algorithm for SAR Images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(1): 453-466 [8] Ma W P, Wen Z L, Wu Y, et al. Remote Sensing Image Registration with Modified SIFT and Enhanced Feature Matching[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(1): 3-7 [9] Aguilera C, Barrera F, Lumbreras F, et al. Multispectral Image Feature Points[J]. Sensors, 2012, 12(9): 12661-12672 [10] Aguilera C A, Sappa A D, Toledo R. LGHD: A Feature Descriptor for Matching Across Non-linear Intensity Variations[C]//IEEE International Conference on Image Processing, Quebec City, Canada, 2015 [11] Ye Y X, Shan J, Bruzzone L, et al. Robust Registration of Multimodal Remote Sensing Images Based on Structural Similarity[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(5): 2941-2958 [12] Li J Y, Hu Q W, Ai M Y. RIFT: Multi-Modal Image Matching Based on Radiation-Variation Insensitive Feature Transform[J]. IEEE Transactions on Image Processing, 2020, 29: 3296-3310 [13] 姚永祥, 张永军, 万一, 等. 顾及各向异性加权力矩与绝对相位方向的异源影像匹配[J]. 武汉大学学报·信息科学版, 2021, 46(11): 1727-1736 Yao Yongxiang, Zhang Yongjun, Wan Yi, et al. Heterologous Images Matching Considering Anisotropic Weighted Moment and Absolute Phase Orientation[J]. Geomatics and Information Science of Wuhan University, 2021, 46(11): 1727-1736 [14] Yang Z Q, Dan T T, Yang Y. Multi-Temporal Remote Sensing Image Registration Using Deep Convolutional Features[J]. IEEE Access, 2018, 6: 38544-38555 [15] 南轲, 齐华, 叶沅鑫. 深度卷积特征表达的多模态遥感影像模板匹配方法[J]. 测绘学报, 2019, 48(6): 727-736 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201906008.htm Ke Nan, Qi Hua, Ye Yuanxin. A Template Matching Method of Multimodal Remote Sensing Images Based on Deep Convolutional Feature Representation[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 727-736 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB201906008.htm [16] Yu K, Zheng X, Fang B, et al. Multimodal Urban Remote Sensing Image Registration via Roadcross Triangular Feature[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 4441-4451 [17] 王广帅, 万一, 张永军. 交叉点结构特征约束的机载LiDAR点云与多视角航空影像配准[J]. 地球信息科学学报, 2020, 22(9): 1868-1877 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202009012.htm Wang Guangshuai, Wan Yi, Zhang Yongjun. Registration of Airborne LiDAR Data and Multi-view Aerial Images Constrained by Junction Structure Features[J]. Journal of GeoInformation Science, 2020, 22(9): 1868-1877 https://www.cnki.com.cn/Article/CJFDTOTAL-DQXX202009012.htm [18] Katz S, Tal A. On the Visibility of Point Clouds[C]// IEEE International Conference on Computer Vision, Santiago, Chile, 2015 [19] 张永军, 张祖勋, 龚健雅. 天空地多源遥感数据的广义摄影测量学[J]. 测绘学报, 2021, 50(1): 1-11 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202101001.htm Zhang Yongjun, Zhang Zuxun, Gong Jianya. Generalized Photogrammetry of Spaceborne, Airborne and Terrestrial Multi-source Remote Sensing Datasets[J]. Acta Geodaetica et Cartographica Sinica, 2021, 50(1): 1-11 https://www.cnki.com.cn/Article/CJFDTOTAL-CHXB202101001.htm [20] Shi J B. Good Features to Track[C]//IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 1994 [21] Fischer S, Šroubek F, Perrinet L, et al. Self-Invertible 2D Log-Gabor Wavelets[J]. International Journal of Computer Vision, 2007, 75(2): 231-246 http://staff.utia.cas.cz/sroubekf/papers/gabor_07.pdf [22] Dalal N, Triggs B. Histograms of Oriented Gradients for Human Detection[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 2005 [23] Baráth D, Noskova J, Ivashechkin M, et al. MAGSAC++, a Fast, Reliable and Accurate Robust Estimator[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020 [24] Shoshany B. A C++17 Thread Pool for High-Performance Scientific Computing[J]. arXiv, DOI: 10.5281/zenodo.4742687 [25] 赵明衍, 戴晨光, 狄亚南, 等. 一种POS数据辅助多视角倾斜航空影像匹配方法[J]. 测绘科学技术学报, 2016, 33(4): 431-435 Zhao Mingyan, Dai Chenguang, Di Yanan, et al. A POS Supported Matching Method for Multi-View Oblique Aerial Images[J]. Journal of Geomatics Science and Technology, 2016, 33(4): 431-435
###### 通讯作者: 陈斌, bchen63@163.com
• 1.

沈阳化工大学材料科学与工程学院 沈阳 110142

Figures(6)  / Tables(1)

## An Efficient Matching Method of LiDAR Depth Map and Aerial Image Based on Phase Mean Convolution

##### doi: 10.13203/j.whugis20210524
###### 1. School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China2. Wuhan Branch of Guangzhou Zhongwang Longteng Software Co. Ltd, Wuhan 430074, China
Funds:

Basic Research Strengthening Program of China 173 Program

the National Natural Science Foundation of China 42030102

the National Natural Science Foundation of China 41871368

the National Natural Science Foundation of China 42001406

• Author Bio:

• ###### Corresponding author:ZHANG Yongjun, PhD, professor. E-mail: zhangyj@whu.edu.cn

Abstract:   Objectives  Multi-source image matching is primarily disturbed by nonlinear intensity difference, contrast difference and inconspicuous regional structure features, while the significant differences of texture features result in lack of part structure seriously between light detection and ranging(LiDAR)depth map and aerial image, and this problem causes a mutation in the phase extremum, which further increases the difficulty of matching.  Methods  In this paper, a method of efficient matching of LiDAR depth map and aerial image based on phase mean convolution is proposed. In the image feature matching stage, a histogram of phase mean energy convolution(HPMEC) is established, which extended the phase consistency model in order to solve a mean convolution sequence and phase maximum label map by constructing phase mean energy convolution equation. Then the nearest neighbor matching algorithm was completed the initial match and marginalizing sample consensus plus was used to remove outliers. Based on the thread pool parallel strategy, the images were matched by dividing the overlapping grid. Multiple sets of LiDAR depth map and aerial image with different types of ground coverage are used to as dataset to experiment with position scale orientation-scale invariant feature transform (PSO-SIFT), Log-Gabor histogram descriptor (LGHD), radiation-variation insensitive feature transform (RIFT) and histogram of absolute phase consistency gradients (HAPCG) methods respectively.  Results  The results show that the performance of HPMEC method is superior to the other four methods in the matching of LiDAR depth map and aerial image, the average running time is 13.3 times of PSO-SIFT, 10.9 times of LGHD, 10.4 times of HAPCG and 7.0 times of RIFT, at the same time the average correct matching points are significantly higher than the other four methods, the root mean square error is lightly better than the other four methods within 1.9 pixels.  Conclusions  The proposed HPMEC method could achieve efficient and robust matching between LiDAR depth map and aerial image.

LIU Weiyu, WAN Yi, ZHANG Yongjun, YAO Yongxiang, LIU Xinyi, SHI Lisong. An Efficient Matching Method of LiDAR Depth Map and Aerial Image Based on Phase Mean Convolution[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1309-1317. doi: 10.13203/j.whugis20210524
 Citation: LIU Weiyu, WAN Yi, ZHANG Yongjun, YAO Yongxiang, LIU Xinyi, SHI Lisong. An Efficient Matching Method of LiDAR Depth Map and Aerial Image Based on Phase Mean Convolution[J]. Geomatics and Information Science of Wuhan University, 2022, 47(8): 1309-1317.
• 机载激光雷达（light detection and ranging，LiDAR）点云与航空影像数据的融合使用是城市三维重建的主要技术手段[1]，而数据配准是联合两种数据的关键。但LiDAR数据与航空影像数据在成像原理、结构成分和地理参考都存在较大差异，导致两者之间的空间配准依然面临问题。

常见的点云和影像配准方法主要有以下3种：（1）影像基于多视几何原理生成摄影测量点云和LiDAR点云配准[2]，但该方法对摄影测量点云质量与影像初始定位精度要求较高，容易陷入局部最优解；（2）利用点云与影像分别提取特征进行匹配[3]，但该方法要求点云与影像场景中具有较为丰富的结构信息；（3）将点云转成深度图像，再与影像进行匹配[4]，该方法将三维与二维匹配问题转成了二维匹配问题，但转换关系复杂，匹配难度大。LiDAR深度图与航空影像属于多模态匹配问题，现有的匹配方法大致分为基于相似性测度的方法和基于影像特征的方法两种。以互信息为代表的相似性测度匹配方法[5]可以较好地评价点云强度图与航空影像间的相似关系，该类方法通过定义相似性指标，不断迭代优化光学影像外方位元素，使两类影像间的相似性达到最大，从而完成两者的匹配。然而这类方法存在计算复杂度高、影像姿态信息精度要求较高和容易陷入局部最优解等问题。基于影像梯度特征的匹配方法，如局部强度不变描述、合成孔径雷达-尺度不变特征转换和位置尺度方向-尺度不变特征转换（position scale orientation- scale invariant feature transform，PSO-SIFT）[6-8]都对梯度方向描述符进行改进，以提高多模态影像匹配的能力，但影像梯度特征对非线性强度差异非常敏感，导致算法匹配性能不稳定。

有学者通过提取影像的形状和结构特征实现多模态匹配，受边缘直方图描述符[9]的启发，Log-Gabor直方图描述（Log-Gabor histogram descriptor，LGHD）[10]使用多方向多尺度的Log-Gabor滤波器代替空间滤波器提取影像特征。相位一致性直方图[11]利用Log-Gabor奇对称滤波器计算的相位振幅和方向进行描述符构建，但相位一致性方向直方图（histogram of orientated phase congruency，HOPC）需要精确的地理参考。辐射变化强度特征转换（radiation-variation insensitive feature transform，RIFT）[12]利用相位一致性信息计算最大索引图提取描述子，但是RIFT不支持尺度差异下的多模态影像匹配。绝对相位一致性梯度直方图（histogram of absolute phase consistency gradients，HAPCG）[13]通过扩展相位一致性模型，构造绝对相位梯度方向来替代梯度特征完成描述子提取，但HAPCG无法应对影像间的大旋转问题。上述方法在不同程度上提升了多模态影像的匹配能力，但LiDAR深度图与航空影像间存在的非线性强度差异、结构缺失和大尺寸影像匹配等问题仍面临困难。近年来将深度学习应用于多模态匹配领域的研究得到快速发展[14-16]，该类方法大多使用卷积神经网络训练特征描述子，从而获取高层次特征来完成匹配。利用深度学习进行多模态影像匹配的研究，由于样本采集难度较大，真实场景复杂多变，导致训练有效模型并投入工程应用中较为困难，因此在多模态匹配中，其泛化能力仍有待提高。

上述方法主要是为了解决多模态影像间存在的非线性强度差异与对比度差异导致的匹配困难问题，没有充分考虑到LiDAR深度图中由于部分结构特征缺失造成的相位极值突变对匹配结果的影响，同时没有充分发掘大尺寸影像在匹配效率方面的潜力。因此，本文对相位一致性模型进行了扩展，利用奇偶对称滤波器生成多尺度多方向能量振幅，对每个方向的多尺度能量进行均匀卷积，并且结合方向梯度直方图描述框架提出了一种相位均匀能量卷积直方图（histogram of phase mean energy convolution，HPMEC）方法，能够很好地解决影像间非线性强度差异、对比度差异和结构缺失问题。由于经过中心投影生成的LiDAR深度图与航空影像间存在位移差异，同时为了大幅度提高影像匹配效率，本文通过划分重叠格网将每个格网对应的两种影像的图像块单元组成匹配任务队列，使用多线程并行处理任务队列中的匹配任务来提高匹配效率。

Reference (25)

/